X-ray screening reports

At an airport checkpoint, a human screener looks at an X-ray image of each bag and decides in a few seconds whether to clear it or open it. Object detectors already help. A detector is a vision model that draws a box around each item it recognises and labels it, such as “knife” or “laptop”, with a confidence between 0 and 1. The screener still has to turn those boxes into a judgement: which items are actually prohibited, how sure the detector was, and whether anything needs a closer look.

I worked on a step that writes that judgement out as a short screening report. A large language model writes it, and language models can state things fluently that are not true. Two questions follow, and each gets its own experiment.

1. 1

   What should a report be written from?

   The X-ray image itself, or the detector’s structured output.

   Experiment one
2. 2

   How do we know each line is true?

   A screener who can’t trust individual lines will ignore the report, or trust it too much.

   Experiment two

The detector comes from other parts of the project. For every item it reports a detection: an ID, a primary label, a confidence, a box, and sometimes lower-scoring alternative labels. For example:

obj_001  knife    0.88
obj_002  laptop   0.79   second guess: power bank 0.18

• Report writer. It takes the detections, and optionally the image, and writes a four-part report: items found, suspicious items, evidence for those judgements, and reasons for manual review. Before the model runs, ordinary code works out the simple facts: how many items of each kind, where each box sits in a 3×3 grid over the image, and how pairs of boxes relate (left of, above, overlapping, inside). The model writes from these facts instead of doing the geometry itself. An item counts as suspicious only if its primary label is on a fixed list of 20 prohibited categories, such as knives, guns, lighters, batteries and power banks.
• Claim verification. Each line in the suspicious-items and evidence sections is a claim. A second model scores every claim before the screener sees the report.

The reports are written in Chinese. The examples on this page are translated.

Setup

I built 135 questions from STCray, a public dataset of baggage X-ray images with threat items annotated by boxes. They cover nine types, 15 each: which items are present, whether a given item is present, how many, where an item is, how two items sit relative to each other, which is furthest in some direction, simple geometry such as the largest box, orientation, and occlusion (whether one item covers another). Every question uses a different image.

The same multimodal model, Qwen3.8-27B, answered every question twice:

The “detections” here were the dataset’s own annotated boxes, standing in for a detector. That makes this a test of a perfect detector, a limit covered in Section 6.

Result

A second model, Kimi K3, also scored 32.6% from the image alone. Working from the image never reached 33%. Adding the image on top of the detections didn’t help either: 81%, against 82% without it.

The gap is largest on exactly what a screening report has to get right:

Neither condition answered the occlusion questions. They need visual information the boxes don’t carry, and the model couldn’t read it from the image either.

The first decision makes the second question easier. If a report is written from the detections, every claim in it should be traceable to them. So a claim can be checked against the detections and computed facts alone, as text, without the image.

This also fixes what a verifier’s score means. It measures whether a claim agrees with the detector’s evidence, not whether it is true of the bag. If the detector is wrong and the report repeats it faithfully, the claim should still score high.

Setup

I built a benchmark of 200 simulated reports with 921 claims. Forty reports are fully correct. Each of the other 160 contains exactly one planted error, drawn from 20 error types that cover the ways a report can disagree with its evidence. Three examples, all for the bag above:

• Promoted second guess

  obj_002: power bank (0.18)

  Listed as a suspicious item. The detector’s answer was “laptop”, which is not prohibited. Power bank was only a low-scoring alternative.

• Invented object

  obj_007: knife

  No object with that ID exists.

• Detail beyond the evidence

  obj_001 has a serrated blade about 20 cm long.

  Nothing in the detections supports this, and a text-only check cannot confirm it, so it counts as unsupported.

Other error types include a wrong confidence or position, a reversed spatial relation, a wrong count, a wrong prohibited call, and a claim that cites no evidence at all. Wherever possible, labels were checked automatically against the detections.

Each candidate received the same evidence text the report writer saw, plus one claim at a time, and returned a score from 0 to 1. I compared them by AUROC: the probability that a randomly chosen correct claim scores higher than a randomly chosen wrong one. 1.0 is a perfect ranking, and 0.5 is chance.

Result

Errors caught

94%

Flagging every claim below 0.5

Correct claims flagged

6%

At the same threshold

Time per report

0.2–0.3 s

Scoring every claim

On suspicious-item claims, Jev’s ranking is perfect. The open-source models are clearly behind, and each needs its own flagging threshold. Jev has two consistent weak spots:

• Claims that cite no evidence but aren’t false, such as “lighters are prohibited”, usually score high.
• Correct claims about spatial relations, such as “obj_001 is left of obj_002”, score lower than other correct claims.

Its scores also vary by up to 0.2 between identical requests, so the flagging threshold needs a margin.

One report from the benchmark, for the bag used in the examples above, with Jev’s actual scores. Its single planted error is the promoted second guess.

Every correct line scores above 0.9. The one wrong line scores 0.02 and is flagged, so the screener knows not to act on it. The knife, the real finding, keeps its high score. The items-found and manual-review sections aren’t scored: they summarise the bag and prompt a check, rather than making claims about specific items.

What the benchmarks don’t cover yet

• Real detector errors. Experiment one used dataset annotations, which means a perfect detector. A real detector misses and mislabels items, and by design the verifier can’t catch a report that faithfully repeats a wrong detection. The comparison should be repeated with a real detector’s output.
• New contraband types. The pipeline knows only the 20 prohibited categories and what the detector can label. A new kind of contraband is invisible to the detector, so the report never mentions it and the verifier has nothing to check.
• Freely worded reports. Benchmark claims follow fixed conventions so their labels can be checked automatically, and each report has one error. Real output is worded more freely and can have several, so 0.988 is closer to an upper bound than a field result.
• Visual-only questions. Occlusion was unanswerable in both conditions. Questions like that stay a reason for manual review.
• Real screener behaviour. Nothing here measures what screeners do with the scores. At the 0.5 threshold, about a quarter of fully correct reports still show at least one flag. Whether that trains screeners to ignore flags, or high scores lead them to over-trust, is untested.
• Size. 135 questions and 200 reports are pilot-scale. In experiment one, a single question moves the score by almost one percentage point.

Next: add screener disagreements to the benchmark

The most useful new test cases are those where a screener disagreed with the system: a flag they overrode, a high-scoring line they rejected, or an item they found that the report never mentioned. Each becomes a labelled claim in the benchmark. That would test the verifier on the mistakes that actually reach people, not only the ones I thought to plant.