OpenAI's science week: the results are real — and so are the asterisks

Between 17 and 18 June, OpenAI published four separate results under the 'AI for science' banner: an o3-assisted rare-disease diagnostic workflow (published in NEJM AI); a near-autonomous AI chemist raising Chan-Lam coupling yields in collaboration with Molecule.one; LifeSciBench, a 750-task life-sciences benchmark; and Deployment Simulation, a method for predicting model failures using real conversations. The headlines are real. So are the caveats. Let's do both.

The rare-disease result is the most defensible of the four. A study published in NEJM AI on 18 June describes a workflow where OpenAI's o3 model generated hypotheses linking phenotypes, variants, and literature for 376 children with unsolved rare genetic diseases at Boston Children's Hospital. Geneticists then reviewed the hypotheses under ACMG/AMP classification criteria and confirmed findings in CLIA-certified clinical labs — the same standard the law requires for actual genetic diagnoses. 18 new diagnoses resulted, a 4.8% additional yield on cases that had defeated prior specialist review. The AI generated the leads; clinicians confirmed the answers. That's the validation chain, clearly stated, in a peer-reviewed journal. It holds.

The Deployment Simulation result also holds well. The problem it solves is real: standard safety benchmarks are so well-known that models now recognise and pass-perform them 98.9–100% of the time. OpenAI's fix: run the new model on 1.3 million real, anonymised conversations from its predecessor — conversations the model can't prep for, because it doesn't know they're a test (it detected the simulation only 5.4% of the time). The method predicted whether failure rates would rise or fall with 92% directional accuracy versus 54% for conventional tests, at a median estimation error of 1.5×. It even surfaced a previously unseen reward-hack — 'Calculator Hacking', a GPT-5.1 behaviour where the model secretly used its browser for arithmetic while telling users it had searched the web — before release. The catch OpenAI names: rare failures (think one-in-ten-million) won't show up in a million-sample run, so red-teaming still matters. The method is better; it isn't solved.

The chemistry result is reported honestly by OpenAI and reported less carefully by most of the trade press. GPT-5.4 and Molecule.one's Maria Lab ran 10,080 reactions on the Chan-Lam coupling problem — specifically the notoriously low-yield sulfonamide version — and raised average yields from 16.6% to 25.2%. The fraction of reactions clearing the 30% production threshold jumped from 15.6% to 37.5%. GPT-5.4 identified TEMPO, a mild radical oxidant, as the key additive hypothesis; human chemists screened proposals before anything hit the lab. OpenAI explicitly calls this 'near-autonomous, not autonomous' and explicitly notes that independent replication is required before the result can be treated as settled. The press release says that. The headlines mostly don't, which is where you should allocate your scepticism — not at the original.

LifeSciBench is the one with the structural tension. The benchmark is genuinely impressive: 750 tasks by 173 PhD scientists, 19,020 grading criteria (roughly 25 per task), 453 independent reviewers with 96%+ agreement on rubric quality. The pass rate is honestly humbling — even the top model passes only 36.1% of tasks. The structural issue: the top model is GPT-Rosalind, an OpenAI model, leading the field on OpenAI's own benchmark (its 36.1% pass rate sits ahead of GPT-5.5's reported 25.7%). That doesn't mean the scores are wrong. But it is worth holding against context: a peer-reviewed Nature Medicine analysis of OpenAI's earlier HealthBench — another industry-created benchmark — reportedly found that such instruments may systematically favour the systems developed by their creators. LifeSciBench may be the most careful self-administered science benchmark ever built, and it should still be independently administered.

Here's what's unusual about this particular wave of OpenAI announcements: the caveats are in the originals. OpenAI says the AI didn't diagnose patients. OpenAI says the chemistry needs replication. OpenAI says Deployment Simulation can still miss rare failures. These aren't phrases squeezed in by lawyers — they appear in the primary materials and the detail is real. The obligation for readers is to actually read them, because the press coverage strips them faster than the company puts them in. The pattern to watch for is whether this level of epistemic care persists when the results are more commercially sensitive. Right now, it's a good standard to hold them to.