The Nonlinear Library: Alignment Forum
Education
AF - SAEs (usually) Transfer Between Base and Chat Models by Connor Kissane
2024-07-18
Download
Right click and do "save link as"
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SAEs (usually) Transfer Between Base and Chat Models, published by Connor Kissane on July 18, 2024 on The AI Alignment Forum.
This is an interim report sharing preliminary results that we are currently building on. We hope this update will be useful to related research occurring in parallel.
Executive Summary
We train SAEs on base / chat model pairs and find that SAEs trained on the base model transfer surprisingly well to reconstructing chat activations (and vice versa) on Mistral-7B and Qwen 1.5 0.5B.
We also find that they don't transfer on
Gemma v1 2B, and are generally bad at reconstructing
view more
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SAEs (usually) Transfer Between Base and Chat Models, published by Connor Kissane on July 18, 2024 on The AI Alignment Forum.
This is an interim report sharing preliminary results that we are currently building on. We hope this update will be useful to related research occurring in parallel.
Executive Summary
We train SAEs on base / chat model pairs and find that SAEs trained on the base model transfer surprisingly well to reconstructing chat activations (and vice versa) on Mistral-7B and Qwen 1.5 0.5B.
We also find that they don't transfer on
Gemma v1 2B, and are generally bad at reconstructing
More Episodes
Get this podcast on your
phone, FREE
