The Alarm Nobody Calibrated

A supermarket that installs self-checkout cameras and flags one honest shopper in ten as a thief has not bought a security system. It has bought a queue of angry customers and a lawsuit. The same arithmetic governs every camera sold to a council, a clinic or a car park this year, and it is the arithmetic that disappears from the pitch deck: at low prevalence, a detector's false positive rate — not its headline accuracy — determines what the system does to people. Computer vision has spent a decade optimising the numerator. The obligations that attached to high-risk systems in the European Union this August are, for the first time, forcing buyers to read the denominator.

Five Threads, One Knot

Regulation (EU) 2024/1689 made its high-risk regime applicable to Annex III systems — including remote biometric identification and safety components of critical infrastructure — on 2 August 2026, following the Article 5 prohibitions that took effect on 2 February 2025. That legal milestone lands on a technical stack in transition: perception research has moved from frame-level classification to video and spatial world models, consumer wearables have normalised always-on capture, enforcement has escalated from fines toward structural remedies, and road-transport regulators have opened defect inquiries into camera-led driving systems.

From Frames to Worlds, and the Benchmark Debt

The least-reported consequence of the shift to video-native and generative world models is that the industry's evaluation infrastructure no longer measures what it ships. Mean average precision on a held-out frame set says nothing about temporal consistency, identity persistence through occlusion, or behaviour under distribution shift — the failure modes that generate liability. Fei-Fei Li's argument in her 2025 essay on spatial intelligence, that perception must model three-dimensional structure rather than label pixels, is technically correct and commercially inconvenient: it invalidates a decade of procurement language.

The practical effect is that model cards reporting ImageNet or COCO scores are now close to meaningless as safety evidence. Buyers should be asking for closed-loop, task-level evaluation on their own footage.

The Regulator as Unwitting Architect

The strongest case for the AI Act is not deterrence but standardisation. By requiring risk management, data governance, logging, human oversight and post-market monitoring, it converts accuracy from a marketing claim into an auditable artefact. Vendors who once shipped a REST endpoint now ship a technical file. That is a genuine advance, and it will raise the floor across a market where the median deployment has never been independently tested.

Counterpoint: Conformity Is Not Accuracy

The one-sided version of that argument treats legal compliance as a proxy for performance. It is not. Most Annex III systems permit internal-control conformity assessment rather than third-party notified-body review, and harmonised standards from CEN-CENELEC JTC 21 have lagged the applicability dates — the gap the Commission's digital simplification package sought to address. A supplier can therefore document a quality management system impeccably while its classifier still exhibits subgroup error rates that no auditor measured. Compliance spend and evaluation spend compete for the same budget. Where the former crowds out the latter, the Act produces paperwork with a CE mark on it.

Biometric Base Rates: The Arithmetic Procurement Ignores

Open-set identification remains the most mis-sold capability in the sector. A one-to-many search against a gallery of 100,000 faces at a per-comparison false match rate of one in a million still returns candidates on innocent people; the operator's threshold, gallery size and image quality dominate outcomes. Demographic skew compounds this.

Both findings predate the current generation of models, and vendor accuracy has improved materially since. Neither result has been superseded by a published, subgroup-disaggregated audit of the systems now being installed. That absence is the story.

Tampa, 2001: A Precedent Worth Repricing

Tampa, Florida deployed a face-surveillance system at Super Bowl XXXV in January 2001, scanning attendees against a gallery of wanted persons. It produced no actionable identifications and was abandoned by 2003. The failure was not principally algorithmic; it was a base-rate and image-capture problem — uncontrolled pose, distance and illumination against a small gallery of interest. Twenty-five years later, the same operational geometry appears in retail loss-prevention and transit deployments, now with better feature extractors and identical physics. The lesson is that raising model capacity does not repair an ill-posed acquisition problem. The parallel lesson from ImageNet after 2012 is that a benchmark monoculture directs an entire field's effort toward the metric rather than the task.

Sovereign Weights, Borrowed Eyes

Dependence on a handful of non-European providers for perception backbones is a real strategic exposure, and it is now a procurement question rather than an academic one. Where a vision system gates access to a hospital, a border or a payment rail, the ability to inspect, retrain and host the model domestically has measurable option value.

Counterpoint: The Case Against Building Your Own

Vision is the most commoditised layer in machine learning. Openly released segmentation and detection models — the Segment Anything lineage trained on the SA-1B corpus of 11 million images and 1.1 billion masks is the clearest example — mean that sovereign pretraining programmes largely duplicate available capability at high cost. The genuine chokepoints are narrower and less photogenic: lawful domain-specific training data, calibrated local test sets, edge inference silicon, and the engineering capacity to run continuous evaluation. Industrial policy that funds a national foundation model while leaving those four unfunded buys sovereignty on paper and dependence in practice.

What To Do Before the Next Audit Cycle

Six Months Out: May 2027

Expect the conformity gap to widen before it closes: with harmonised standards incomplete, most high-risk vision files will be self-assessed, and the first contested enforcement action will turn on documentation quality rather than measured accuracy. Expect procurement language to harden — participation in NIST's ongoing face recognition evaluation, with subgroup breakdowns, becoming a de facto tender requirement in Europe as it already is in parts of US federal purchasing. Expect at least one European data protection authority to act on wearable-based face lookup, following the precedent of the Dutch authority's €30.5 million Clearview penalty in September 2024. And expect the evaluation conversation to move from static benchmarks to closed-loop, temporally scored tests, because insurers will demand it before regulators do.

The bottom line: the binding constraint on deployed computer vision is no longer model capability. It is measurement — who performs it, on whose data, at what threshold, and who carries the cost when the denominator is ignored.