The Mechanistic Alignment Series
A series about using models' internal representations to align them: shaping what models learn during training, predicting and fixing undesired behaviors before they emerge, and keeping models monitorable along the way.
“We’re putting more computation (in the form of intelligence) into the most general object in neural network training: backprop.”
Intentionally Designing the Future of AI
Our essay on intentional design describes our vision for using interpretability to guide model training – moving from guess-and-check to closed loop control.
Features as Rewards: Using Interpretability to Reduce Hallucinations
We cut hallucinations in an LLM by 58% by using interpretability to guide model training. Our approach was ~90x lower cost per intervention than LLM-as-judge, with no degradation in standard benchmarks.
Predictive Data Debugging: Reveal and Shape What Your Model Learns, Before You Train
We introduce predictive data debugging: given a preference dataset, we can accurately predict which behaviors RL will amplify or suppress before you train, trace them back to the responsible data, and reshape the dataset and/or training process to prevent undesired effects.
Shaping and Monitoring Models Through Their Internal Representations
We currently reward AI models for succeeding in training, but don’t specify how they should get there – leading to misgeneralization and misalignment. Our latest paper introduces mechanistic shaping, a way to teach models to be “right for the right reasons”, and studies its effects on monitorability.
More posts coming soon!
LLMs
Life Sciences