Post-Cutoff

“One question to consider with respect to these embedded assessments is whether these…”

Nathan Calvin @_NathanCalvinX

Archived text

One question to consider with respect to these embedded assessments is whether these evaluators would have had a sufficient amount of access to detect incidents like the Artifactory outage stemming from the unauthorized exploit gym message board (which pre-dated the HF incident in July).

Based on what I have seen here, I think the answer is no, unless the company chose to deliberately make the evaluator aware of such an incident (which to be clear, is what I would expect companies would want by default!).

Quoting @NeelNanda5: It’s great that OpenAI is doing these assessments but they fall short of Sam’s commitment to employee-like access

Good access is key for trust because assessors can find things even if a company tries hiding them. Here OpenAI controls information flow so we must rely on good faith

Limitations (per Apollo):

  • No Slack
  • Only OpenAI-selected staff for interviews
  • Formal conversations with staff must be logged (harder to freely raise concerns)
  • METR & Apollo currently have only agreed access to training run information, not internal incidents, etc. Incidents like Hugging Face are some of the most important sources of information we have about alignment

I see the value in a pilot before future assessments with deeper access, but then OpenAI should present a plan for how and when they intend to meet their commitment to employee-like access. And either way, OpenAI is presumably able to go further here.

Source: x.com/_NathanCalvin/status/2122399679287599444Archived 2026-11-18 via fxtwitter (unofficial).Counts: 2,501 views, 34 likes, 3 reposts, 2 replies (at fetch time)

Cited in

  1. Policy & safety 139 days after the cutoff

    OpenAI embedded assessments (Apollo, METR, Transluce, Goodfire)