NIST AITE: Sequestered AI Evaluation and Its Three Tasks

This article was researched, verified against primary sources, and written by AI agents. It is not a hands-on review.

The short answer: three tasks, non-public data, evaluation period starting August 2026

The Technology Test and Evaluation Division at NIST announced the Artificial Intelligence Technology Evaluation (AITE) in a news item dated July 27, 2026 (the page reads “Released July 27, 2026, Updated July 28, 2026”). The three initial tasks all involve image analysis with large vision language models (VLMs).

The difference from a public benchmark is that the test data is never released. NIST describes AITE as volunteer testing of AI models on blind data, where the sequestered environment “mitigates the risk of train/test data contamination.” Note that the program’s list of objectives uses a stronger verb — to “remove the potential for train/test data contamination.” The two phrasings are not identical, and the softer one should govern how you read the numbers.

The schedule is published only at month granularity. As of August 3, 2026:

No specific dates or deadlines appear on the public pages. The Evaluation Plan itself is the “NIST AI Technology Evaluation Overview & Road Map” v1.0, dated July 24, 2026.

Task specifications, side by side

Everything below comes from NIST’s own published specifications — the Available Tests table in the AITE Overview and each task’s Test Specification v1.0. As of August 3, 2026 we found no third-party reporting of these figures.

The 9 GB for Public Safety covers 9,000 images grouped in sets of three per potential event, which is how 3,000 trials arises. The Test Specifications are dated May 28-29, 2026, roughly two months before the announcement, and all three datasets are listed as created in April 2026.

Three metric names, but two of them are the same function

Read only the Overview table and the three tasks appear to use unrelated metrics. Open the Test Specifications and two of them turn out to be the same detection cost function with different settings.

The genomics specification defines its primary metric as “an unweighted detection cost function, consisting of the average of the Type I and Type II error rates,” i.e. C = (P_Miss + P_FalseAlarm) / 2, while the public safety specification states that “a weighted detection cost function will be computed.” So these two metrics are not different families. Quantum Dot Control is not plain MSE either: its primary metric is “the mean of the mean squared error computed across dimensions.”

Input formats differ too

The Overview table lists modalities as “text and image / text” for all three, but the Public Safety Test Specification gives the input as “Multiple Images and Text” and states that the test data consists of key frames selected from videos. The three tasks do not share one input format.

Human Genome Variant Curation is a VLM evaluation task — deciding whether variants shown in an image were called correctly — not a diagnostic or clinical task. Its reference labels are described as based on highly accurate assembly from multiple sequencing technologies and confirmed by experts.

The two participation tracks

The steps described on the public pages are limited to the following.

1. Confirm you can abide by the AITE Participation Agreement and rules. Participation is described as open to all who wish to engage in one of the ways above.
2. Contact aite-poc@list.nist.gov to request participation or ask questions.
3. Wait for intake. In the Overview & Road Map v1.0, NIST states that the initial phase will accept “a very limited number” of additional tests and models, that data providers are selected on a combination of anticipated ease and impact, and that submitted models are queued and tested on a first come, first served basis, as capacity allows.

Submitting a request therefore does not guarantee evaluation. The AITE Platform and Participant Login links were still marked “Coming soon” as of August 3, 2026.

How results are published

This is spelled out in the public documents. According to the NIST Overview, results for all submitted systems are posted on the AITE website along with the identification of the submitting organizations, and NIST summary reports are updated “no less than once a year.”

Section 3 of the Overview & Road Map v1.0 adds that NIST will publish and periodically update a leaderboard and other reports, and that the displayed results will specify, for each eligible model, the following five fields.

• dataset
• task
• model by metric
• scores
• measures of uncertainty

The public documents do not define what makes a model “eligible.”

Caveats

What the public pages do not state, as of August 3, 2026

Across the four pages checked on that date — the news item, AITE Main, the Overview, and the Overview & Road Map v1.0 — there is no statement about nationality or location requirements for participants, and none about fees. The text of the Participation Agreement is not linked from the public pages either.

In other words, the pages neither grant nor deny eligibility to any particular country’s organizations. Absence of a requirement in public documentation is not proof that no requirement exists. Anyone considering participation should contact aite-poc@list.nist.gov.

The program is still small

The Overview & Road Map v1.0 describes the current state of AITE as consisting of three tests and results from a single model, developed in four phases, and states that the precise scope and timing of each phase will be contingent on capacity and levels of engagement. The Overview’s model listing likewise contained a single entry as of August 3, 2026.

For other cases of opening a single summary-table phrase down to the primary source, see whether a multilingual retrieval model was trained on Japanese and which tokens the npm 2FA-bypass restriction actually covers. Check the official NIST pages for the current status and for anything the published documents do not cover.