核心转储流行病学：修复一个18年的漏洞

作者：Nathan Bronson，技术团队成员 OpenAI 的模型和代理日益依赖可扩展的数据基础设施，以便在推理时（即模型思考你的问题时）搜索相关数据。其中一些服务用 C++ 编写，这种语言对系统的底层控制让我们能够最大化性能并最小化内存使用。随着规模扩大，这些效率优势变得重要，但 C++ 缺乏内存安全性意味着错误可能导致程序因写入错误或不存在的内存地址而崩溃。

几个月前，我们在 Rockset 服务中观察到一些崩溃，这是 ChatGPT 数据基础设施中一个定制部分，对许多数据插件和搜索对话至关重要。在每次崩溃中，一个正常的 C++ 函数似乎执行完毕，然后返回一个虚假地址，导致内核停止程序，因为指令指针不再指向代码。有时栈帧中的返回地址槽是 NULL。有时栈指针 CPU 寄存器本身似乎偏移了 8 字节，仿佛 %rsp 在正常执行过程中被错误地递减了。在这两种情况下，崩溃都发生在返回时。

这些不是应用程序代码的正常故障模式。仅落在保存的返回地址上的随机写入是可能的，但极不可能。一个不涉及内联汇编、setcontext 或 longjmp（我们都不使用）的、使 %rsp 偏移 8 字节的错误更加奇怪，因为编译后的代码只在函数序言和尾声直接调整该寄存器。我们（或 ChatGPT）能想到的每个假设都有强有力的证据反对它，因此这个错误似乎不可能存在。

我们最初认为的一个问题最终被证明是两个不相关的错误，巧合地同时被发现。首先，一个 Azure 主机上的静默硬件损坏，CPU 无法正确进行数学运算。其次，GNU libunwind 中一个存在 18 年的竞态条件，一个广泛使用的开源库中未被注意的错误。

这篇文章讲述了我们如何像流行病学家一样思考，并构建关于整个崩溃群体的高质量数据集，从而识别并修复看似无法解释的崩溃。

首次调试尝试：仔细检查几个核心转储

首先，让我们深入了解 Rockset。它是一个用于搜索和实时分析的云原生数据系统，我们在 OpenAI 内部用于许多用例，例如同步连接器（Rockset 于 2024 年被 OpenAI 收购）。流式更新用于维护工作区知识库的最新索引，以便 ChatGPT 在回答问题或执行操作时搜索相关信息。

Rockset 的执行层用 C++ 编写。C++ 语言提供对 CPU 的低级访问，有利于性能和效率，但这意味着应用程序错误可能导致无效内存访问和段错误。为了帮助追踪这些问题，我们使用 folly 的致命信号处理程序在崩溃时记录堆栈跟踪，并将相应的核心转储（程序崩溃时的状态快照）上传到 Azure blob 存储以供后续分析。Rockset 的所有查询处理节点都是复制的，这最小化了崩溃对客户端的影响。然而，每个段错误对应一个需要修复的错误，以满足我们的可靠性和质量目标。

我们的初始方法是将这些核心视为传统调试问题：非常仔细地检查几个核心转储，形成假设，并逐一排除。

大多数崩溃发生在名为 DocumentTree::updateDocument 的方法中。在这些崩溃中，似乎 updateDocument 调用了某个未知函数 X，当 X 活动时栈被损坏，然后 X 返回了一个不是可执行代码的地址。在某些情况下，X 刚弹出的帧看起来有效，只是其保存的返回地址是 NULL。在其他情况下，栈指针本身看起来错误，但下一个有效帧似乎仍然是 updateDocument。

我们不知道栈何时被损坏，这留下了巨大的搜索空间。updateDocument 是一个大型方法，经历了大量内联，因此 X 的候选者数量庞大。

这是我们的 C++ 代码中的错误吗？编译器或链接问题？运行时库中的问题？Linux 内核在信号传递或上下文切换方面的错误？还是更罕见的问题？如果是随机写入，为什么我们的 ASAN 暂存环境没有捕获到？

我们尝试使用应用程序级日志来识别所有问题实例，但栈损坏错误很难仅从日志中分类，因为记录的堆栈跟踪本身已损坏或缺失。我们无法构建一个既没有误报也没有漏报的日志查询。我们手动检查了更多核心，并发现了一些额外示例，但这个过程过于劳动密集，无法提供可靠的数据集。

在调查的这个阶段，我们（错误地）排除了硬件错误，因为我们在多个区域和多种硬件类型上看到了崩溃，因此我们仍在寻找纯软件原因。有几天，我们深入研究了单个 %rsp 偏移的崩溃，使用栈和寄存器内容重建崩溃前的历史。这产生了一些可能的线索，但由于我们没有放弃所有错误都有相同原因的初始结论，这并没有让我们摆脱困境。

来自栈的线索

在进入调查的转折点之前，解释我们从核心文件中提取了哪些信息很重要。

Rockset 使用 -fno-omit-frame-pointer 编译，因此活动栈帧始终可以通过 %rsp 访问，调用者形成一个帧指针链表。

在 Linux x86_64 上，AMD64 System V ABI 还在 %rsp 下方保留了 128 字节作为红区。该区域可供用户空间代码使用，重要的是，内核承诺在传递信号时不会破坏它，这是 ABI 契约的一部分。

红区对我们调试返回后崩溃至关重要，因为它保留了返回前的一些信息。当触发 SIGSEGV 时，folly 的致命信号处理程序在崩溃线程的栈上运行。不再活动的栈帧（因为其函数已返回）将被信号处理程序覆盖，除了最后 128 字节。这就是为什么我们可以说“X 刚弹出的栈帧看起来有效，只是返回地址是 NULL”。红区保留了一些非活动帧，有时只是一个非活动帧的尾部。

我们发现了一次栈对齐错误导致的崩溃，其中涉及的所有函数都非常小。这让我们看到，在执行一个相对简单的函数时，%rsp 变得不对齐，并且之后又成功执行了更多调用。程序只有在活动函数最终尝试返回时才崩溃。这些代码路径都没有使用异常、内联汇编、setcontext 或 longjmp，因此如果栈指针确实像核心转储所暗示的那样发生了变化，用户空间代码中就没有合理的错误可以解释这个问题。

这让我们将目光转向了内核。

Rockset 比大多数程序更积极地使用信号。查询执行被分解为许多交换数据的轻量级任务。这对于高效处理高 QPS 工作负载很重要，但它使得每个查询的 CPU 统计变得棘手，因为许多查询的工作被多路复用到同一个线程池中。

我们的解决方案是我们称之为 coarse_thread_cputime_clock 的东西，它近似于 clock_gettime(CLOCK_THREAD_CPUTIME_ID, ...)，成本足够低，可以在每个任务边界进行采样。timer_create API 可用于根据多种时间流逝概念（包括 CPU 时间的累积）来调度周期性信号传递。我们安排一个信号（SIGUSR2）每几毫秒的 CPU 时间传递一次，此时信号处理程序会更新一个线程局部值。尽管许多任务在执行过程中看不到粗略时钟的推进，但将所有增量求和后，可以产生查询实际 CPU 时间的无偏估计。

由于我们如此频繁地传递信号，一个关于上下文切换或信号传递的罕见内核错误似乎是合理的。我们花时间阅读了错误报告、内核源代码以及 Azure 特定的内核补丁。我们尝试了压力测试。但未能找到任何相关的内容。

那时，我们决定退一步，尝试不同的方法。

医生还是流行病学家？

调试这类问题有两种主要方法。

一种是像医生一样行事：专注于一个患者，进行大量测试，并尝试从详细证据中诊断单个病例。

另一种是更像流行病学家：观察整个人群，询问是否存在单个病例无法揭示的模式。这个错误是否在某个特定版本开始出现？它是否与某个硬件 SKU（特定的 CPU 和服务器型号）、某个区域或某个内核版本相关？在看似相同的症状背后，是否隐藏着多个不同的集群？

我们之前主要处于医生模式。关键的转变是决定我们需要收集高质量的人群数据。

清理数据

我们之前尝试自动找到所有问题实例的努力失败了，因为我们试图使用文本搜索日志。核心转储本身包含更多信息，但手动查看它们无法扩展。我们决定投入精力构建一个能够自动分析核心转储的管道。

我们让 ChatGPT 编写了一个脚本，该脚本下载每个核心文件的前缀，提取寄存器，使用日志过滤已知的误报，并自动将崩溃标记为返回空指针、栈对齐错误或其他。然后，我们在过去一年中所有生产 Rockset 核心转储上并行运行该脚本。

这是转折点。

一旦我们有了干净的数据集，相关性立即显现出来。我们之前视为一个奇怪错误的东西，实际上是两个独立的崩溃群体。

返回空指针的核心转储分布在许多集群和地理区域中。它们的频率最近有所增加，但没有明确的开始日期，也没有清晰的基础设施边界。

栈对齐错误的崩溃看起来完全不同。它们都来自一个区域，有明确的开始日期，并且从未发生在运行时间较长的节点上。尽管它们涉及多个 Azure VM（云中托管的虚拟机），但模式看起来像是一台物理机器存在硬件问题，导致恰好落在其上的任何 VM 出现问题。

那一刻，我们意识到自己在心理上混淆了两个错误。因为我们一直在混合两个错误的反例，所以无法找到一个统一的解释。

错误 #1：有问题的宿主机

凭借一份干净的 Kubernetes 节点和时间戳列表，我们能够将栈对齐错误的崩溃追溯到一台物理宿主机，这很容易加入黑名单。

我们无法在受控环境中重现该宿主机上的寄存器损坏，即使经过数周的压力测试。然而，一旦将有问题的宿主机停用，栈对齐错误的崩溃就消失了。

移除有问题的宿主机并不是一个永久解决方案，因为它不能防止同一问题再次发生。但是，我们可以修改软件，以便如果类似问题再次出现，可以轻松检测和处理。我们改进了致命信号处理程序，使其包含寄存器状态，这样我们只需从日志中就能检测到复发（无需核心转储）。我们修改了控制平面，使 VM 通常被重用而不是回收，这在我们基础设施栈的层面上使得检测有问题的节点变得容易得多。我们还更新了我们的操作手册（以及团队的心理模型），以包含这种可能性。

将有问题的宿主机崩溃分离出来后，剩余的返回空指针核心转储变得更容易推理。之前，我们排除了异常展开的可能性，因为我们以为有反例：在肯定不使用异常的代码路径中发生崩溃。但这些反例都来自硬件损坏集群。

一旦我们带着这个想法重新审视剩余的核心转储，我们发现这个结论完全相反：所有崩溃都发生在异常展开过程中。

异常处理是一种动态控制转移

当 C++ 抛出异常时，运行时必须发现哪个 catch 块应该接收它，以及沿途哪些析构函数或清理处理程序应该运行。编译器会发出这些元数据，但实际的匹配是在运行时动态发生的。

异常展开实际上不是由调用 throw 的函数执行的，而是由生成的编译代码调用的辅助函数执行的。这些运行时例程检查栈，获取栈上函数的元数据，动态查找清理处理程序和 catch 块，然后将控制转移到这些位置之一。控制转移包括展开所有中间的栈帧（包括辅助函数的栈帧）。

在操作上，这更接近于 longjmp 或纤程切换，而不是普通的调用和返回。必须恢复被调用者保存的寄存器，以及栈帧寄存器 %rbp 和 %rsp。

我们的二进制文件链接到了两个库，这两个库都包含了执行C++异常展开的函数实现：libgcc和GNU libunwind。动态链接器最终选择了GNU libunwind的定义。这让我们感到惊讶；我们原本以为根据符号版本控制规则，libgcc的实现会胜出；然而，检查运行中的二进制文件后发现并非如此。

推翻最后一个假设

此时，我们的工作假设发生了变化，因为我们放宽了之前认为只有一个bug时所做的另一个假设。

也许我们看到的并不是普通的函数返回NULL。也许我们看到的是展开转移——实际上是一种setcontext风格的寄存器恢复——其中目标指令指针在控制权转移之前已经变成了NULL。换句话说，问题出在展开库提供了错误的数据，而不是栈上的返回地址槽位有误。

这大大缩小了问题范围。要么是GNU libunwind计算了错误的目标状态，要么是它计算了正确的状态，但在应用之前被某些东西破坏了。

我们阅读了GNU libunwind的源代码，发现它在栈上合成了一个ucontext_t结构体，为清理处理器的帧填充了所需的寄存器状态，然后将指向该结构体的指针传递给一个内部汇编例程：_Ux86_64_setcontext。

此时，我们掌握了所有线索。

合成的ucontext_t位于_Ux86_64_setcontext执行期间被该函数展开的某个栈帧中。_Ux86_64_setcontext在修改了%rsp之后，是否还会从该结构体中读取数据？此时该结构体已不再是活动栈的一部分？这会使它容易因信号传递（比如我们频繁使用的SIGUSR2）而被覆盖。

Bug #2：libunwind的bug

答案是肯定的。

以下是我们使用的GNU libunwind版本中_Ux86_64_setcontext的最后六条指令，主要由从内存加载到目标寄存器的mov指令组成：

（%rdi指向栈上分配的ucontext_t，而UC_MCONTEXT_*宏只是展开为存储特定寄存器的固定偏移量。）

第一条指令是竞态窗口的开始。它将%rsp更新为指向活动栈的新底部。一旦发生这种情况，%rdi指向的结构体就不再属于活动栈（或红区），也不再对内核不可访问。

通常这不会造成问题，但如果信号恰好在正确（或错误？）的时刻到达，内核会在%rsp-128处构建信号帧。这可能会覆盖%rdi指向的内存。

如果这发生在下一条指令读取UC_MCONTEXT_GREGS_RIP(%rdi)之前，那么恢复的指令指针就可能被破坏。在我们的崩溃中，它变成了NULL。

这就是bug所在。

为什么核心转储看起来像普通的错误返回

这段汇编代码也解释了让我们困惑的一个观察结果：为什么函数X的前一个栈帧的返回地址槽位中有一个NULL。

setcontext被设计为恢复所有寄存器，包括%rdi，因此在控制转移的最后时刻，它无法使用该寄存器来读取UC_MCONTEXT_GREGS_RIP(%rdi)。相反，它会提前读取该值，将其保存到栈上，恢复更多寄存器，然后使用retq来读取保存的值并转移控制权。

在核心转储中看起来像是“函数返回到了NULL”的情况，实际上是“展开器在栈上合成了一个目标返回地址，但该目标在转移完成之前已被破坏。”我们之前假设返回地址槽位的破坏一定是在原位发生的，因为我们不知道有任何地方会故意将（可破坏的）数据写入返回地址槽位。

单指令竞态窗口

这个bug看似荒谬之处在于竞态窗口非常狭窄。在这种竞态条件下，外部事件（信号）需要在另一个线程执行的两个步骤之间发生。这两个步骤越接近，竞态条件发生的可能性就越小。

在这种情况下，易受攻击的窗口实际上只有一条指令的宽度！信号必须在%rsp被修改之后、下一条指令加载%rip之前传递。在现代超标量乱序CPU上，每个周期可以执行几条这样简单的指令，因此竞态窗口大约为一百皮秒。

当我们发现这个竞态时，第一反应是它一定非常罕见，无法解释观察到的崩溃率。我们在整个集群中每天看到超过十次返回NULL的崩溃。异常清理过程中的单指令竞态真的能解释这一点吗？

我们转向费米估算。如果易受攻击的窗口大约为$10^{-10}$秒，而SIGUSR2每$10^{-2}$秒的CPU时间到达一次，那么每个异常清理处理器或catch块大约有$10^{-8}$的概率输掉竞态。

Rockset将异常作为内部摄入反压机制的一部分。单个过载主机每秒可能抛出大约$10^{4}$次异常。这意味着使用反压的主机平均故障间隔时间为$10^{4}$秒，即每几小时崩溃一次。在集群规模下，这足以解释观察到的崩溃频率。

为什么libunwind的bug现在出现？

GNU libunwind的bug很古老——超过18年，存在于第一个支持C++异常展开的x86_64版本中。

那么为什么它现在才出现？

崩溃率大致与抛出的异常数量和传递的信号数量成正比。它还取决于信号处理器消耗的栈空间。

Rockset在这三个维度上都不同寻常。我们作为正常过载控制的一部分，以高频率抛出异常；由于coarse_thread_cputime_clock，我们异常频繁地传递SIGUSR2；今年早些时候，我们通过添加对timer_getoverrun的调用，使SIGUSR2处理器使用了更多栈空间，以便处理合并的信号。

最后一个变化似乎很重要。如果处理器使用的栈空间足够少，它可能不会触及并覆盖过时的ucontext_t内存。在那个变化之前，我们完全没有观察到这些崩溃。变化之后，崩溃率一直很低，直到我们为某些使用场景增加了负载，这些场景对反压机制造成了压力。

换句话说，libunwind的bug一直存在，但我们的异常率、信号率和处理器栈使用量的乘积直到最近才跨过阈值，使其在操作上变得可见。

这一机制也解释了为何硬件缺陷和 libunwind 缺陷导致的崩溃大多集中在 DocumentTree::updateDocument 方法中。libunwind 引发的崩溃严重偏向该方法，因为在我们抛出异常以施加摄取背压时，该方法始终处于活跃状态。同时，%rsp 未对齐导致的崩溃也高度集中于此，因为出问题的硬件节点属于我们用于批量摄取的 SKU 型号，其 CPU 大部分时间都花在该方法上。

我们立即采取的缓解措施是将 GNU libunwind 切换为 libgcc 的 unwinder。这本身就是一个很好的权衡：libgcc 的实现得益于大量减少锁竞争的工作，这在扩展到大型虚拟机时尤为重要。

我们还向上游提交了一个独立的复现程序和一个修复方案⁠（在新窗口中打开）给 GNU libunwind，并验证了其他 unwinder 不存在类似问题。

群体诊断的力量

这次调试历程让我们深入了解了动态链接、DWARF 展开元数据、Linux 信号传递、System V ABI 以及 C++ 异常机制的具体细节。但最重要的教训比这些都要简单。

最关键的一步并非巧妙的汇编代码阅读或对细节的深刻理解，而是构建高质量的数据集。如果没有这个数据集，我们就会将两种不同的现象混为一谈，试图在混乱中推理出方向。一旦我们获得了准确且完整的群体数据，问题的结构就变得清晰：一类崩溃属于故障主机，另一类则属于 libunwind 中的竞态条件。数据质量提升后，调试也变得更加容易。

对于像 Rockset 这样的基础设施系统，这一点至关重要。这次调查强化了我们对深度监控、自动化调查以及持续改进运维工具的承诺。可靠性不仅仅在于事后修复缺陷，更在于构建数据、工作流程和技能，从而将看似不可能的问题转变为可诊断、可解决的难题。

By Nathan Bronson, Member of Technical Staff OpenAI’s models and agents increasingly rely on scalable data infrastructure in order to search for relevant data at inference time: when the models are thinking about your question. Some of these services are written in C++, whose low-level control of the system lets us maximize performance and minimize memory usage. Those efficiency benefits are important as we scale, but C++’s lack of memory safety means that bugs can cause crashes by writing to incorrect or non-existent memory addresses.

A few months ago we observed some crashes from inside the Rockset service, a bespoke part of our ChatGPT data infrastructure which is key to many data plugins and to searching over conversations. In each of these crashes, a normal C++ function seemed to finish and then return to a bogus address, causing the kernel to stop the program because the instruction pointer no longer pointed at code. Sometimes the return address slot in the stack frame was NULL. Sometimes the stack pointer CPU register itself seemed to be off by 8 bytes, as if %rsp had somehow been decremented in the middle of normal execution. In both cases the crash happened on return.

These are not normal failure modes for application code. A stray write that lands only on a saved return address is possible, but extremely unlikely. A bug that misaligns %rsp by 8 without involving inline assembly, setcontext, or longjmp (none of which we use) is even stranger, because compiled code only adjusts that register directly in the function prologue and epilogue. Every hypothesis we (or ChatGPT) could think of had strong evidence against it, so the bug seemed impossible.

What we assumed was one problem eventually turned out to be two unrelated bugs, coincidentally discovered at the same time. First, silent hardware corruption on one Azure host, where the CPU just didn’t do math correctly. Second, an 18-year-old race condition in GNU libunwind, an unnoticed bug in a widely used open source library.

This post is the story of how we identified and fixed seemingly inexplicable crashes by thinking like an epidemiologist and building a high-quality data set about the entire population of crashes.

First debugging attempt: carefully examining a few core dumps

First, let’s go deeper on Rockset. It’s a cloud-native data system for search and real-time analytics that we use for many internal use cases at OpenAI, such as sync connectors (Rockset was acquired by OpenAI in 2024). Streaming updates are used to maintain an up-to-date index of a workspace’s knowledge base so that ChatGPT can search for relevant information when answering questions or performing actions.

Rockset’s execution layer is written in C++. The C++ language provides low-level access to the CPU, which is good for performance and efficiency, but it means that application bugs can lead to invalid memory accesses and segfaults. To help track these down we use folly’s fatal signal handler to log a stack trace when a crash happens, and we upload the corresponding core dumps (a snapshot of the state of the program when it crashed) to Azure blob storage for later analysis. All of Rockset’s query processing leaves are replicated, which minimizes the client impact of a crash. However, each segfault corresponds to a bug that needs to be fixed to meet our reliability and quality goals.

Our initial approach was to treat these cores like a conventional debugging problem: inspect a few core dumps very closely, form hypotheses, and rule them out one by one.

Most of the crashes occurred in a method called DocumentTree::updateDocument. In these crashes it appeared that updateDocument had called some unknown function X, the stack had become corrupted while X was active, then X had returned to an address that wasn’t executable code. In some cases X’s just-popped frame looked valid except that its saved return address was NULL. In other cases the stack pointer itself looked wrong, but the next valid frame still seemed to be updateDocument.

We didn’t know when the stack was getting corrupted, which left a huge search space. updateDocument is a large method that undergoes a lot of inlining, so the number of candidates for X was overwhelming.

Was this a bug in our C++ code? A compiler or linkage issue? A problem in one of our runtime libraries? A Linux kernel bug around signal delivery or context switching? Something even rarer? If this was a stray write, why wasn’t it caught by our ASAN staging environment?

We tried to use our application-level logs to identify all occurrences of the problem, but stack-corruption bugs are hard to classify from logs alone because the logged stack traces are themselves corrupted or missing. We weren’t able to construct a log query that didn’t have both false positives and false negatives. We manually inspected more cores and found some additional examples, but that process was too labor-intensive to give us a trustworthy data set.

At this stage of the investigation, we (incorrectly) ruled out a hardware bug, because we saw crashes across multiple regions and multiple hardware types, so we were still looking for software-only causes. For a few days, we went super-deep on a single misaligned-%rsp crash, reconstructing the pre-crash history using stack and register contents. This produced some possible clues, but because we didn’t let go of our initial conclusions that all of the bugs had the same cause, this didn’t get us unstuck.

Clues from the stack

Before getting to the turning point of our investigation, it’s important to explain what kind of information we were extracting from the core files.

Rockset is compiled with -fno-omit-frame-pointer, so the active stack frame is always reachable through %rbp, and callers form a linked list of frame pointers.

On Linux x86_64, the AMD64 System V ABI also reserves 128 bytes below %rsp as the red zone. That region is available to userspace code and, importantly, the kernel promises not to clobber it when it delivers a signal, as part of the ABI contract.

The red zone was central to our debugging of a post-return crash, because it preserves some information from before the return. When a SIGSEGV is triggered, folly’s fatal signal handler runs on the crashing thread’s stack. Stack frames that are no longer active (because their function has returned) will get clobbered by the signal handler, except for the last 128 bytes. That’s why we can say things like “X’s just-popped stack frame looked valid, except for a NULL return address.” The red zone preserves some of the inactive frames, or sometimes just the tail of one inactive frame.

We found one misaligned-stack crash in which all of the functions involved were very small. That let us see that %rsp had become misaligned during execution of a relatively simple function, and that more calls had succeeded afterward. The program only crashed when the active function finally tried to return. None of those code paths used exceptions, inline assembly, setcontext, or longjmp, so if the stack pointer truly changed in the way the core suggested, no plausible bug in userspace code explained the issue.

That pushed us toward the kernel.

Rockset uses signals more aggressively than most programs. Query execution is broken into many lightweight tasks that exchange data. This is important for handling high-QPS workloads efficiently, but it makes per-query CPU accounting awkward as work for many queries is multiplexed onto the same thread pool.

Our solution is something we call coarse_thread_cputime_clock, which approximates clock_gettime(CLOCK_THREAD_CPUTIME_ID, ...) cheaply enough to sample at every task boundary. The timer_create API can be used to schedule a periodic signal delivery based on several notions of the passage of time, including the accumulation of CPU time. We schedule a signal (SIGUSR2) to be delivered every few milliseconds of CPU time, at which point the signal handler updates a thread-local value. Even though many tasks don’t see the coarse clock advance while they are executing, summing all of the deltas produces an unbiased estimate of the actual CPU time for a query.

Because we deliver signals so often, a rare kernel bug around context switching or signal delivery seemed plausible. We spent time reading bug reports, kernel source code, and the Azure-specific kernel patches. We tried stress tests. We weren’t able to find anything that seemed related.

At that point we decided to step back and try a different approach.

Doctor or epidemiologist?

There are two broad ways to debug a problem like this.

One is to act like a doctor of sorts: focus on one patient, run lots of tests, and try to diagnose a single case from detailed evidence.

The other is to act more like an epidemiologist: look at the entire population and ask whether there are patterns that a single case cannot reveal. Did the bug start at a specific release? Does it correlate with one hardware SKU (the specific CPU and server model), one region, or one kernel version? Are there multiple distinct clusters hiding inside what looks like one syndrome?

We had mostly been in doctor mode. The key shift was deciding that we needed to gather high-quality population data.

Cleaning the data

Our previous attempts to automatically find all of the instances of the problem failed because we were trying to use text searches over the logs. The core dumps themselves have a lot more information, but looking at them manually didn’t scale. We decided to invest the effort to build a pipeline that could automatically analyze the core dumps.

We had ChatGPT write a script that downloaded a prefix of each core file, extracted the registers, filtered known false positives using the logs, and automatically labeled the crash as return-to-null, misaligned-stack, or other. Then we ran that script in parallel over every production Rockset core dump from the previous year.

This was the turning point.

Once we had a clean data set, correlations appeared immediately. What we had been treating as one weird bug was actually two separate crash populations.

The return-to-null cores were spread across many clusters and geographic regions. Their frequency had increased recently, but there was no crisp start date and no clean infrastructure boundary.

The misaligned-stack crashes looked completely different. They all came from one region, had a clear start date, and never happened on nodes that had been running for a long time. Even though they involved multiple Azure VMs (virtual machines hosted in the cloud), the pattern looked like one physical machine with bad hardware causing problems for whichever VM happened to land on it.

That was the moment we realized we had been mentally conflating two bugs. Because we had been mixing counterexamples from both bugs, we couldn’t find a single coherent explanation.

Bug #1: the bad host

Armed with a clean list of Kubernetes nodes and timestamps, we were able to trace the misaligned-stack crashes back to a single physical host, which was easy to denylist.

We were not able to reproduce the register corruption on that host in a controlled environment, even after several weeks of stress testing. Once the problematic host was taken out of service, however, the misaligned-stack crashes disappeared.

Removing the bad host isn’t a permanent solution, in the sense that it doesn’t prevent a new occurrence of the same problem. We can, however, change the software so that if a similar issue recurs, it is easily detected and handled. We improved our fatal signal handler to include register state so that we can detect recurrence only from the logs (no core dump needed). We changed the control plane so that VMs are usually reused instead of recycled, which makes bad-node detection much easier at our level of the infrastructure stack. We also updated our runbooks (and our team’s mental models) to include this possibility.

With the bad-host crashes separated out, the remaining return-to-null cores became much easier to reason about. Earlier we had ruled out exception unwinding because we thought we had counterexamples: crashes in code paths where exceptions were definitely not used. But those counterexamples were all from the hardware-corruption cluster.

Once we revisited the remaining cores with that in mind we found that this conclusion was exactly backward: the crashes were all happening during exception unwinding.

Exception handling is a dynamic control transfer

When C++ throws an exception, the runtime has to discover which catch block should receive it and which destructors or cleanup handlers should run along the way. The compiler emits this metadata, but the actual matching happens dynamically at runtime.

Exception unwinding is not actually performed by the function that invokes throw, but by helper functions called by the resulting compiled code. Those runtime routines examine the stack, fetch metadata about the functions found on the stack, dynamically look for cleanup handlers and catch blocks, and then transfer control to one of those locations. Transferring control includes unwinding all of the intervening stack frames (including those of the helper functions).

Operationally, this is much closer to a longjmp or a fiber switch than to a normal call and return. Callee save registers must be restored, as well as the stack frame registers %rbp and %rsp.

Our binary links against two libraries that contain implementations of the functions that perform C++ exception unwinding: libgcc and GNU libunwind. GNU libunwind’s definitions were the ones chosen by the dynamic linker. That surprised us; we had expected the libgcc implementation to win because of symbol versioning rules; however, inspecting running binaries showed that wasn’t the case.

Undoing one last assumption

At this point our working hypothesis changed, as we relaxed another assumption that we had made when we thought there was only one bug.

Maybe we were not seeing an ordinary function return to NULL. Maybe we were seeing an unwind transfer—effectively a setcontext-style register restore—where the destination instruction pointer had become NULL before control was transferred. In other words, incorrect data from the unwind library rather than an incorrect return address slot on the stack.

That narrowed the problem dramatically. Either GNU libunwind was computing the wrong destination state, or it was computing the right state and something was corrupting it before it could be applied.

We read the GNU libunwind source and found that it synthesizes a ucontext_t on the stack, fills in the desired register state for the cleanup handler’s frame, and then hands a pointer to that struct to an internal assembly routine: _Ux86_64_setcontext.

At this point we had all of the pieces.

The synthesized ucontext_t lives in one of the stack frames that is unwound by _Ux86_64_setcontext, during that function’s execution. Was _Ux86_64_setcontext reading from the struct after it changed %rsp, at which point the struct was no longer part of the active stack? That would make it vulnerable to being clobbered by a signal delivery, such as our frequent SIGUSR2.

Bug #2: the libunwind bug

The answer was yes.

Here are the last six instructions of _Ux86_64_setcontext in the version of GNU libunwind we were using, which consist mostly of mov instructions that load from memory to a destination register:

(%rdi points at the stack-allocated ucontext_t, and the UC_MCONTEXT_* macros just expand to the fixed offset at which a particular register is stored.)

The first instruction is the beginning of the race window. It updates %rsp to point to the new bottom of the active stack. As soon as this happens, the struct pointed to by %rdi is no longer part of the active stack (or red zone), and it’s no longer off-limits to the kernel.

Usually this doesn’t cause problems, but if a signal arrives at exactly the right (wrong?) moment, the kernel will build the signal frame at %rsp-128. That can overwrite the memory pointed to by %rdi.

If that happens before the next instruction reads UC_MCONTEXT_GREGS_RIP(%rdi), then the restored instruction pointer can be corrupted. In our crashes, it became NULL.

That’s the bug.

Why the cores masked as ordinary bad returns

This assembly also explains one of the observations that confused us: why function X had a NULL in the return address slot of the preceding stack frame.

setcontext was written to restore all registers, including %rdi, so it can’t use that register to read UC_MCONTEXT_GREGS_RIP(%rdi) at the final moment of the control transfer. Instead, it reads the value earlier, saves it to the stack, restores a few more registers, then uses retq to read the saved value and transfer control.

What looked in the cores like “a function returned to NULL” was actually “the unwinder synthesized a target return address on the stack, but that target had been corrupted before the transfer completed.” We had assumed that corruption of the return address slot must happen in-place, because we didn’t know of any places where (corruptible) data was written to the return address slot on purpose.

A single-instruction race window

What makes this bug seem absurd is how narrow this race window is. In this kind of race condition, the external event (the signal) needs to happen in between two steps taken by another thread. The closer those steps are to each other, the less likely the race condition is to happen.

In this case the vulnerable window is literally one instruction wide! A signal must be delivered after %rsp has been changed, but before the next instruction loads %rip. Several simple instructions like this can be run per cycle on a modern super-scalar out-of-order CPU, so the race window is roughly a hundred picoseconds.

When we found this race, our first reaction was that it must be too rare to explain the observed crash rate. We were seeing more than a dozen return-to-null crashes per day across the fleet. Could a one-instruction race during exception cleanup really account for that?

We turned to Fermi estimation. If the vulnerable window is on the order of $1 0^{- 10}$ seconds and SIGUSR2 arrives every $1 0^{- 2}$ seconds of CPU time, then each exception cleanup handler or catch block has a roughly $1 0^{- 8}$ probability of losing the race.

Rockset uses exceptions as part of its internal ingest backpressure mechanism. A single overloaded host can throw on the order of $1 0^{4}$ exceptions per second. That implies the mean time between failures of a host using backpressure is $1 0^{4}$ seconds, or one crash every few hours. At fleet scale, that is more than enough to explain the observed crash frequency.

Why did the libunwind bug appear now?

The GNU libunwind bug is old—more than 18 years old, present in the first x86_64 version that supported C++ exception unwinding.

So why did it show up now?

The crash rate is roughly proportional to how many exceptions are thrown and how many signals are delivered. It’s also dependent on how much stack the signal handler consumes.

Rockset is unusual on all three axes. We throw exceptions at high rates as part of normal overload control; we deliver SIGUSR2 unusually often because of coarse_thread_cputime_clock; and earlier this year we made the SIGUSR2 handler use more stack by adding a call to timer_getoverrun, so we could account for merged signals.

That last change seems to have been important. If the handler uses little enough stack, it may not reach and overwrite the stale ucontext_t memory. Before that change, we do not observe these crashes at all. After the change the rate remained low until we ramped up load for some use cases that stressed the backpressure mechanism.

In other words, the libunwind bug has always been there, but the product of our exception rate, signal rate, and handler stack usage had only recently crossed the threshold where it became operationally visible.

This mechanism also explains the coincidence that both the hardware bug and the libunwind bug crashed mostly inside DocumentTree::updateDocument. Crashes from libunwind were heavily biased toward this method, because it’s always active at the point we throw an exception to apply ingest backpressure. It was also heavily selected for the %rsp-misalignment crashes because the bad hardware node was of a SKU that we use for bulk ingest, which spends the majority of its CPU time in that method.

Our immediate mitigation was to switch from GNU libunwind to libgcc’s unwinder. That was a good trade on its own: libgcc’s implementation has benefited from a lot of work to reduce lock contention, which matters when scaling to large VMs.

We also upstreamed a self-contained reproducer and a fix⁠(opens in a new window) to GNU libunwind, and verified that the other unwinders don’t have a similar issue.

The power of a population-level diagnosis

This debugging journey taught us a lot about the specific details of dynamic linking, DWARF unwind metadata, Linux signal delivery, the System V ABI, and C++ exception machinery. But the main lesson was simpler than any of that.

The most important step was not the clever assembly reading or deep knowledge of the details. It was building a high-quality data set. In the absence of this data set, we were mixing two distinct phenomena into one story and trying to reason our way out of the confusion. Once we had accurate and complete population data, the structure of the problem became obvious: one crash population belonged to a bad host, and the other belonged to a race in libunwind. Once the data got better, the debugging got easier.

For infrastructure systems like Rockset, that matters a lot. This investigation reinforced our commitment to deep instrumentation, automated investigations, and continual improvements in our operational tooling. Reliability is not just about fixing bugs after they happen—it’s about building the data, workflows, and skills that turn impossible problems into diagnosable and solvable ones.