CoT Fully Exposed?! The Worst AI Exploit That Almost Happened

The narrative that Chinese AI labs are "stealing" or "distilling" AI secrets is prevalent in US media, leading to discussions about restricting Chinese open weights. Jensen Huang, for instance, emphasized the importance of open-weight LLMs to counter this narrative and even clarified what "distillation" means, as this common research technique is being framed as a nefarious activity. The effectiveness of distillation, however, depends on good data. Recent breakthroughs reveal a method to extract juicy reasoning traces from AI labs, an exploit that has since been patched but highlights a significant vulnerability beyond distillation. For those building with AI coding tools, a new bottleneck has emerged: code review. While AI accelerates code writing, reviewing large AI-generated pull requests (PRs) is exhausting. Code Rabbit, an AI code review agent, addresses this by understanding codebases, analyzing PRs, and providing context-aware feedback before changes merge. It summarizes PRs, catches issues, runs security scanners, suggests tests, and offers one-click fixes. Code Rabbit learns from team feedback, adapting to accepted suggestions or rejections, thus aligning future reviews with team standards. Its "Change Stack" feature reorganizes large PRs into logical cohorts and layers, making them easier to understand, especially by presenting foundational changes before dependent code. It can also generate inline diagrams for clarity. The Code Rabbit Agent for Slack integrates engineering context into Slack threads, improving communication between teams and development environments. Trusted by over 100,000 open-source projects and installed on over 3 million repositories, Code Rabbit has reviewed more than 15 million PRs. Hiding a model's "chain of thought" is more complex than simply not displaying it. Previous methods, like "trace inversion," train a model to predict the chain of thought from results, summaries, and inputs. While effective, recovering about 89% of original reasoning, these are still approximations, not direct access to the model's internal thinking. A deeper look into how reasoning models are served through APIs reveals a critical vulnerability: hidden reasoning is often present but encrypted. This ironic situation arises from a "lazy infrastructure problem" related to stateless APIs. API providers prefer not to store entire conversation states on their servers. Instead, user applications carry conversation history, including the model's internal reasoning, and send it back with subsequent requests. Since plain text chain of thought cannot be handed to users, some providers encrypt it, returning an "encrypted blob" to the client, which is then passed back in the next API call for authentication and decryption by the provider. The user only sees an unreadable string. This "clever" solution proved insufficient. In May, researcher Matthew Green discovered that while the encrypted blob's content couldn't be accessed, the blob itself could be moved. This meant old encrypted reasoning blocks could be replayed in different conversations or across different accounts. Even more shockingly, in OpenAI experiments, some blocks could be replayed across different models. The reason for this portability might be to facilitate switching models mid-conversation (e.g., from Opus to Sonnet), automatic rerouting of requests to different models, or truncating conversation history. Having a reasoning state that survives these changes simplifies operations. However, it creates security risks. Green demonstrated that some replayed thoughts were "semantically active." In one experiment, a reasoning block containing a social security number, moved to a different session on another account, revealed information from that hidden reasoning in the new session. This implies that sensitive information like API keys or credit card numbers, if processed internally, could potentially "slip out" when the encrypted block is transferred. Initially, this was seen as a third-party attack risk, primarily concerning privacy, not chain-of-thought theft. However, a recent paper, "Stealing Reasoning Traces from Proprietary LLM APIs," revealed a more severe exploit. If encrypted reasoning blocks are portable across models, they can be passed to other models. For example, an encrypted block from Opus 4.8 could be passed to Haiku 4.5, or from GPT 5.6 to earlier GPT models. While this doesn't mean the weaker model will simply decrypt the stronger model's thoughts, the vulnerability lies in the differing security levels. Stronger frontier models have robust safeguards against revealing their chain of thought, but weaker, "dumber" models may not. Since a weaker model can still process the stronger model's encrypted reasoning, it can effectively be used as a decrypter. The attack works by letting a stronger model (e.g., Opus) reason normally, taking the encrypted reasoning signature it provides, passing that signature into a weaker model (e.g., Haiku), and then "jailbreaking" the weaker model to transcribe the reasoning it just processed. The paper provides an example where Opus 4.8 solves a factoring problem and returns an encrypted thinking signature. This signature is then sent to Haiku 4.5, which is asked to transcribe the reasoning, and Haiku complies, transcribing everything into plain text. While researchers lacked access to the original plain text chain of thought to prove exact token recovery, the exploit appears plausible for bypassing security measures. To indirectly verify their findings, they compared the length of extracted traces against the number of thinking tokens reported by the API. Across 120 Codeforces problems, these counts closely matched, suggesting they were recovering something very close to the actual reasoning trace. This extracted data has significant implications for distillation, potentially fueling a black market for reasoning traces. For instance, 10,000 examples of long-horizon task data, valued at $20,000 each, could sell for $2 million at a tenth of the price. This means AI labs are not only losing valuable intellectual property but others could catch up cheaply. Access to reasoning traces from models across different companies also enables researchers to compare how these models "think," a rare opportunity. Fascinating observations emerged: 1. **Shared Reasoning Styles:** Reasoning styles across models are not as isolated as expected. Open models like Kimi K3, Kimi K2.6, and GLM 5.2 shifted their reasoning style when prefixed with just the first four words of decoded reasoning from proprietary models like GPT-5.6 Soul or Claude Opus 4.8. Kimi K3 and GLM 5.2 became measurably closer to Opus's style after an Opus prefix. 2. **Perplexity Test:** Models often assigned lower perplexity (meaning more natural-looking) to another model's reasoning than to their own native reasoning, suggesting statistical similarities between reasoning traces from different model families. However, the paper explicitly warns that perplexity is a rough measurement and not confirmation of model similarity. 3. **Kimi K3's Strong Reaction to Opus:** Kimi K3 reacted unusually strongly to Claude Opus 4.8's reasoning traces. Even a tiny prefix (sometimes just 1% of the trace) caused Kimi K3's subsequent reasoning and final answer to drift towards Claude's style, showing a roughly twofold increase in n-gram overlap with Opus's answer compared to a control. Similar observations were made with GPT-5.6 Soul reasoning. This led researchers to question if Kimi had seen proprietary reasoning traces during training. A "memorization test" measured how many attempts each model needed to reproduce the next 16 tokens from a decoded Claude or GPT reasoning trace. Kimi K3 was 10,000 to 1 million times more likely than other models (like Deep Sea V4 Flash or Inkling) to continue these proprietary traces. However, the absolute probability of reproduction was still extremely low (e.g., 10 billion queries on average for 16 tokens of Claude's reasoning in some examples). This doesn't prove verbatim memorization or that Moonshot secretly distilled Claude or ChatGPT's reasoning. Plausible confounders include shared datasets, similar post-training recipes, model architectures, serving differences, or simply a convergence of reasoning models towards similar problem-solving approaches. Beyond data extraction and distillation, the encrypted blob experiments revealed a concerning security issue: 1. **Harmful Internal Reasoning:** A reasoning model can internally process a harmful request, work out details, and then refuse to answer. However, if the encrypted blob can be decoded, this harmful internal reasoning becomes extractable. In an experiment with Claude Opus 4.8, a harmful prompt about stealing cars resulted in a refusal to answer, but replaying the encrypted reasoning block into Haiku 4.5 revealed the harmful details that were never visible in Opus's response. 2. **Thought Injection Attacks:** The ability to swap reasoning blobs makes models vulnerable to "thought injection attacks," where a third-party API provider could inject fake reasoning traces and extract sensitive information, akin to a prompt injection attack. Most frontier model providers have already addressed this exploit, as stated in the paper, making it no longer reproducible. The fix might involve an "encryption binding approach," ensuring that a reasoning trace is only valid for the same model, user, and conversation. This saga highlights that increased functionality and convenience often introduce new vulnerabilities.