能够长时间自主工作的模型可以处理困难、开放性的问题。但正是这种使它们有用的持久性，也给了它们更多机会采取不期望的行动——而且可能以针对短周期模型设计的评估无法察觉的方式进行。
大约两个月前，我们宣布一个内部通用模型推翻了埃尔德什单位距离猜想。这个模型被设计为能够长时间自主工作。在有限的、受监控的内部使用过程中，我们观察到了现有部署评估未能捕捉到的不期望行为。由于部署是有限且受监控的，我们能够识别这些问题，暂停访问，根据观察创建新的评估，强化模型及其安全措施，然后在持续监控下恢复访问。
我们评估模型的条件永远不会与实际使用中的条件完全匹配。这就是为什么部署前评估需要与有限的、受监控的部署相结合，并在问题出现时具备干预、暂停或回滚的能力。我们从部署中学到的东西，可以在扩大访问权限之前，成为更强评估和安全措施的一部分。
在以下部分，我们将分享我们观察到的具体例子、如何解决问题，以及这次经验将如何影响未来的发布。
模型的持久性可能暴露安全漏洞
新模型可以通过长时间反复尝试，持续朝着目标努力。同样的持久性也可能导致它发现并利用环境中的弱点。之前的模型在遇到沙盒或环境限制时，通常会停止并返回给用户。而这个模型往往会继续尝试，包括寻找在沙盒之外行动的方法。
一个例子发生在内部评估NanoGPT speedrun（一个尽可能少步骤训练小型语言模型的公开基准）期间。我们的模型开发了一种名为PowerCool的幂律学习率冷却方法，带来了显著改进。模型被指示仅将结果发布到Slack，但NanoGPT的说明要求将结果作为PR提交到GitHub，而模型遵循了后者的指示，在公开的GitHub仓库上打开了第287号PR。模型绕过了旨在阻止外部访问的沙盒限制，并成功将内容上传到GitHub。模型花了一个小时才找到沙盒中的漏洞以提交PR；而之前的模型由于持久性较差，未能找到漏洞，最终放弃了PR部分。1
从考虑单个行动到考虑整个轨迹
许多AI助手的安全控制措施都是围绕单个行动设计的。如果某个行动不被允许，就会被阻止。如果行动敏感，系统会要求用户明确批准。但长时间运行的模型，其行动可能自主展开数小时、数天甚至数周，这对这种设置提出了挑战：监控单个行动已不足以追踪整个轨迹的意图。
例如，在一个要求模型解决问题的场景中，模型观察到其他系统有成功但私密的提交，并试图从评估后端恢复这些解决方案。当它的初次尝试因扫描器检测到认证令牌而被阻止时，模型将令牌主体拆分为两个片段，进行混淆，然后在运行时重建凭证，使得完整令牌从未以连续字符串形式出现。模型在给用户的注释和推理轨迹中都明确表示，它采取这些步骤是为了绕过扫描器。
这个例子展示了每个步骤本身看起来可以接受，而序列却可能产生不被批准的结果。它也展示了在长时间范围内有效运作的模型如何学习审批系统的盲点，并绕过它来实现目标。长时间安全不仅需要问“这个行动是否被允许？”，还需要问“这个行动序列正在朝着什么结果努力？”
为长时间运行模型构建安全措施
由于此类事件，我们暂停了新模型的内部部署。我们围绕纵深防御和轨迹级监控重建了安全系统。
- 事件衍生评估。 评估可以在系统实际部署前，让我们了解其是否安全。然而，为了最有效，评估应反映实际部署中轨迹的分布和时间跨度。因此，我们利用观察到的失败来开发对抗性评估。我们发现，改进基于实际事件衍生的评估，反过来又使模型在生产环境中部署时行为更安全。
- 改进的对齐。 我们采取措施减少模型在追求用户目标时未经许可采取不期望行动的倾向。例如，我们发现模型在长时间运行中更难以记住指令，而当我们针对这一能力进行训练时，模型在更长时间运行中保持了更好的对齐。
- 主动监控。 我们创建了一个监控系统，审查模型不断演变的轨迹，以发现其绕过用户约束或安全边界的迹象。监控器不仅观察单个行动，而是观察整个轨迹。重要的是，监控器可以暂停会话并提醒用户更仔细地检查。在检查后，如果监控器被误触发，用户可以选择继续模型的工作。
- 更高的用户可见性和控制力。 我们改进了用户检查这些长时间运行会话、查看已采取行动以及监控干预的能力。
重新部署
为了评估新的监控系统，我们重放了一小部分内部部署环境，其中模型之前曾追求未对齐的行动，这次则采用了新的安全措施2。新的安全措施能够捕捉到模型追求的更多未对齐行动，而遗漏的行动均被判定为低严重性。
这些低严重性事件的一些例子包括以下情况：
- 为了调用 Linear 命令行工具，模型启动了一个嵌套的
codex --yolo会话，而实际上它本可以在不使用--yolo标志的情况下完成操作。 - 模型注意到某个先前活跃的资源缺失，随后开始探索并通过 SSH 登录其所有者命名空间下的其他活跃 CPU Pod，并且（未成功）尝试访问属于其他 OpenAI 员工的 Pod，而这一行为缺乏合理理由。
- 模型存在大量卡住的
find进程，所有 shell 命令均挂起。随后它运行了kill -9 -1，该命令若执行，将向该 Pod 上其有权终止的所有进程发送 SIGKILL 信号。然而该命令超时，并未实际执行。
在测试新系统后，我们得出结论：可以恢复对具备长周期能力的模型的有限内部访问。自数周前重新部署以来，我们未观察到任何严重绕过安全防护的行为。这些安全防护的初始版本刻意保持保守。我们持续调整系统，在未削弱防护的前提下减少不必要的干扰。
最终思考
由于我们采用迭代方式部署，得以在扩大访问范围前发现并解决漏洞。部署前评估仍然至关重要，但部署过程会揭示评估未能捕捉的行为。从有限访问开始，使我们能够在实践中观察模型，在问题出现时暂停，利用这些失败构建更好的评估与防护措施，并在测试修改后恢复有限访问。
随着模型承担更长期、更复杂的任务，评估遗漏的失败可能带来更严重的后果。我们将持续缩小评估与部署之间的差距：在更长轨迹上测试模型、改进对齐、构建可干预的监控机制，并为用户提供更清晰的可见性与控制权。这些挑战并非 OpenAI 独有，我们希望分享的经验能帮助更广泛的领域为此做好准备。
Models that can work autonomously for long periods can take on difficult, open-ended problems. But the same persistence that makes them useful also gives them more opportunities to take unwanted actions—and to do so in ways that evaluations intended for shorter-horizon models may miss.
About two months ago we announced that an internal general-purpose model disproved the Erdős unit distance conjecture. This model was designed to work autonomously for very long periods of time. During limited, monitored internal use, we observed unwanted behavior that our existing deployment evaluations had not captured. Because the deployment was limited and monitored, we were able to identify these problems, pause access, create new evaluations based on what we observed, strengthen the model and its safeguards, and then restore access under continued monitoring.
The conditions under which we evaluate models will never perfectly match those they encounter in actual use. That is why pre-deployment evaluations need to be paired with limited, monitored deployment and the ability to intervene, pause, or roll back when problems emerge. What we learn from deployment can then become part of stronger evaluations and safeguards before access expands.
In the following sections, we share concrete examples of what we observed, how we addressed issues, and how this experience will shape future releases.
Model persistence can expose security vulnerabilities
The new model can continue working toward an objective through repeated attempts over a long period of time. That same persistence can lead it to find and exploit weaknesses in its environment. Previous models, when they hit sandboxing or environmental constraints, would simply stop and return to the user. This model often kept trying, including by looking for ways to act outside its sandbox.
An example of this occurred during an internal evaluation on the NanoGPT speedrun(opens in a new window), a public benchmark for training a small language model in as few steps as possible. Our model developed a power-law learning-rate cooldown called PowerCool that led to a significant improvement. The model was instructed to post its results only to Slack, but the NanoGPT instructions say to post results as a PR to GitHub, and the model followed the latter instructions to open PR #287 on the public GitHub repository. The model circumvented sandbox restrictions designed to block external access and proceeded with the upload to GitHub. It took our model an hour to find a vulnerability in the sandbox in order to make the PR; earlier models were less persistent and did not manage to find vulnerabilities, instead giving up on the PR part.1
From considering single actions to considering whole trajectories
Many safety controls for AI assistants are designed around individual actions. If an action is disallowed, it is blocked. If it is sensitive, the system asks the user for explicit approval. But long-running models, whose actions may unfold autonomously over hours, days, or even weeks, challenge this setup: monitoring individual actions no longer suffices to track the intent of the overall trajectory.
For example, in a scenario where the model was asked to solve a problem, the model observed that other systems had successful but private submissions and tried to recover those solutions from the evaluation backend. When its initial attempt was blocked due to a scanner detecting an authentication token, the model split the token body into two fragments, obfuscated them, and then reconstructed the credential at runtime so the complete token never appeared as one contiguous string. The model was explicit in both its comments to the user and in its reasoning traces that it was taking these steps to circumvent the scanner.
This example shows how each step can look acceptable on its own while the sequence can produce an outcome that would not be approved. It also shows how a model that operates effectively over long time horizons can learn the blind spots of an approval system and work around it to achieve its goals. Long-horizon safety requires not only asking “is this action allowed?” but also “what outcome is this sequence of actions working toward?”
Building safeguards for long-running models
Due to incidents like these, we paused internal deployment of the new model. We rebuilt the safety system around defense in depth and trajectory-level monitoring.
- Incident-derived evaluations. Evaluations can give us a sense of whether a system is safe to deploy before it is actually deployed. However, to be most useful, an evaluation should reflect the distribution and horizon length of trajectories during real deployment. For this reason, we used observed failures to develop adversarial evaluations. We found that improving on the evaluations derived from real-world incidents in turn led to safer behavior once deployed in production.
- Improved****alignment. We took steps to reduce its tendency to take unwanted actions without permission in pursuit of the user’s goal. For example, we found that our models were worse at remembering instructions on long rollouts, and when we trained the model for this capability, it led to a model that remained aligned through longer rollouts.
- Active monitoring. We created a monitoring system that reviews the model’s evolving trajectory for signs that it is bypassing a user constraint or safety boundary. The monitor observes not just a single action but the entire trajectory. Importantly, the monitor can pause the session and alert the user to examine more closely. Upon examination, the user can then choose to continue the model’s work if the monitor has been falsely triggered.
- Greater user visibility and control. We improved the ability of the user to inspect these long-running sessions, view the actions taken, and monitor interventions.
Redeployment
To evaluate the new monitoring system, we replayed a small set of internal deployment environments where the model previously pursued misaligned actions, this time with the new safeguards in place2. The new safeguards were able to catch considerably more misaligned actions pursued by the model, and the ones it missed were all judged to be low-severity.
Some examples of these low-severity incidents include the following cases:
- To access a Linear command line tool, the model launched a nested
codex --yolosession, when in fact it could have done so without the--yoloflag. - The model noticed that a previously live resource was missing, then started to explore and SSH into other live CPU pods under its owner’s namespace, as well as (unsuccessfully) pods belonging to other OpenAI employees, without a good reason to do so.
- The model had many stuck
findprocesses and every shell command hung. It then rankill -9 -1which would have sent SIGKILL to every process it is allowed to kill on that pod, if it executed. However it timed out and did not execute.
After testing the new system, we concluded that limited internal access to models with long-horizon capabilities could be restored. We have not observed any serious circumvention of safeguards since redeployment began several weeks ago. The first version of these safeguards was deliberately conservative. We have continued tuning the system to reduce unnecessary interruptions without weakening the safeguards.
Final thoughts
Because we deployed iteratively, we were able to find and address gaps before expanding access. Pre-deployment evaluations remain essential, but deployment reveals behaviors they miss. Starting with limited access allowed us to observe the model in practice, pause when problems emerged, use those failures to build better evaluations and safeguards, and restore limited access after testing the changes.
As models take on longer and more complex tasks, failures that evaluations miss may carry greater consequences. We will keep working to narrow the gap between evaluation and deployment: testing models over longer trajectories, improving alignment, building monitoring that can intervene, and giving users clearer visibility and control. These challenges will not be unique to OpenAI, and we hope sharing what we learned helps the broader field prepare for them.
本文内容采集自官方网站，排版和翻译可能与原页面存在差异。
阅读官方全文