AI Safety Still Needs Great Engineers

Often I meet seasoned software engineers who are interested in pivoting their careers into AI safety, and specifically AI safety research. This impulse is understandable; the fastest AI capability gains over the past couple of years have been in coding and an engineer's daily work is completely unrecognizable from what it was even a year ago. Software engineers in particular are feeling the exponential more than any other profession.An aside on these professional automation fears: it strikes me as likely that if frontier engineering is fully automatable in the near-term, then frontier research might be shortly thereafter (to be clear I think automating either job fully is very hard); both are skills that humans learn over many years, with many verifiable components on the lower end and many taste-based components on the upper end. In my view, the horizon length of the task is likely the more meaningful variable. Even if frontier research ends up being substantially harder to fully automate, the exponential may mean we are talking about a barrier of months, not years. Many of these people are smart, motivated by impact rather than money, and could probably become decent researchers with enough time. But the urgency of the current moment means a pivot from engineering to research is usually misguided. Pivoting now risks spending the dawn of the singularity as a mediocre researcher, even as great engineers are in short supply.Great engineers in this field must still be obsessed with the research: if you don't deeply understand the work, you will not be effective.

After the recent hack of HuggingFace caused by a swarm of rogue agents, it should be clear that safety is no longer an abstract concern. Much has been said already, but there are a couple of details here that deserve more reflection:

• Safety progress is now a blocker for business. Since the HuggingFace incident OpenAI has paused some of their training efforts, and earlier this year Anthropic worked for months before they were ready to deploy Mythos-class models. These are reasonable business decisions: the delays can easily amount to billions in missed revenue, but models that hack are a bigger liability still, especially as governments take notice.
• This is an incident that existing technology could have contained, but implementation complexity likely kept that technology from being deployed where it was needed.The goal should be to make effective guardrails so cheap and efficient that we just run them online all the time as part of the inference stack itself.

My biggest takeaway from these incidents was that in addition to substantial progress in fundamental alignment research, engineering power is needed to deploy safety at scale. Either alone will be insufficient.

Inference-time guardrails make a good case study for these engineering challenges. A naive approach to guardrailing might involve using an LLM-as-a-judge to scan inputs and outputs at inference time; however this quickly becomes prohibitively expensive for long-horizon agents. A more scalable approach is using something like Anthropic's Constitutional Classifiers++, which first monitors with an interpretability signal and then passes only the hardest examples to a more powerful classifier. Anthropic reports this cut monitoring overhead from 23.7% to roughly 1%.

But if this recipe works so well, why has it not been fully propagated across the industry? The answer is that to deploy even this "simple" interpretability technique at frontier scale requires making changes directly to the inference stack to extract the model's internal activations in real time with minimal latency or memory tradeoffs (more on Goodfire's own work to enable efficient frontier interpretability). As open models continue to improve it is likely that this type of guardrailing technology will need to be very rapidly deployed throughout the industry.

This is just one of many engineering problems that immediately come to mind. For instance, some back of the napkin math suggests we are close to, or have recently passed, the point where AIs emit more tokens annually than the collective writing of all of humanity combined;According to Epoch AI's 2024 estimate the amount of high-quality human-generated public text data is somewhere between 100 and 1000 trillion tokens. At I/O this year Google stated that Gemini alone now processes 3.2 quadrillion tokens a month. Even if "processing" overstates new tokens emitted by a couple orders of magnitude, this is just a single provider's monthly allowance. It seems possible that the crossover point is behind us. we will need new scalable methods to audit even a small subset of these transcripts. There are also many trivial ways for agents to escape their sandboxes, and it will require new security paradigms to contain advanced agents.

It is now wartime for safety and alignment, and the techniques coming out of the research community will only matter if we can scale them across the entire AI ecosystem, fast. It'd be a real own-goal to discover a solution to technical alignment and then fail to deploy it. We must hurry to ensure that the best possible systems are in place before the next incident happens.

We need great engineers more urgently than ever. If you're an engineer looking to solve the world's most important problems, consider applying to jobs in AI safety, including at Goodfire.