LW - Guide to SB 1047 by Zvi

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Guide to SB 1047, published by Zvi on August 20, 2024 on LessWrong.
We now likely know the final form of California's SB 1047.
There have been many changes to the bill as it worked its way to this point.
Many changes, including some that were just announced, I see as strict improvements.
Anthropic was behind many of the last set of amendments at the Appropriations Committee. In keeping with their "Support if Amended" letter, there are a few big compromises that weaken the upside protections of the bill somewhat in order to address objections and potential downsides.
The primary goal of this post is to answer the question: What would SB 1047 do?
I offer two versions: Short and long.
The short version summarizes what the bill does, at the cost of being a bit lossy.
The long version is based on a full RTFB: I am reading the entire bill, once again.
In between those two I will summarize the recent changes to the bill, and provide some practical ways to understand what the bill does.
After, I will address various arguments and objections, reasonable and otherwise.
My conclusion: This is by far the best light-touch bill we are ever going to get.
Short Version (tl;dr): What Does SB 1047 Do in Practical Terms?
This section is intentionally simplified, but in practical terms I believe this covers the parts that matter. For full details see later sections.
First, I will echo the One Thing To Know.
If you do not train either a model that requires $100 million or more in compute, or fine tune such an expensive model using $10 million or more in your own additional compute (or operate and rent out a very large computer cluster)?
Then this law does not apply to you, at all.
This cannot later be changed without passing another law.
(There is a tiny exception: Some whistleblower protections still apply. That's it.)
Also the standard required is now reasonable care, the default standard in common law. No one ever has to 'prove' anything, nor need they fully prevent all harms.
With that out of the way, here is what the bill does in practical terms.
IF AND ONLY IF you wish to train a model using $100 million or more in compute (including your fine-tuning costs):
1. You must create a reasonable safety and security plan (SSP) such that your model does not pose an unreasonable risk of causing or materially enabling critical harm: mass casualties or incidents causing $500 million or more in damages.
2. That SSP must explain what you will do, how you will do it, and why. It must have objective evaluation criteria for determining compliance. It must include cybersecurity protocols to prevent the model from being unintentionally stolen.
3. You must publish a redacted copy of your SSP, an assessment of the risk of catastrophic harms from your model, and get a yearly audit.
4. You must adhere to your own SSP and publish the results of your safety tests.
5. You must be able to shut down all copies under your control, if necessary.
6. The quality of your SSP and whether you followed it will be considered in whether you used reasonable care.
7. If you violate these rules, you do not use reasonable care and harm results, the Attorney General can fine you in proportion to training costs, plus damages for the actual harm.
8. If you fail to take reasonable care, injunctive relief can be sought. The quality of your SSP, and whether or not you complied with it, shall be considered when asking whether you acted reasonably.
9. Fine-tunes that spend $10 million or more are the responsibility of the fine-tuner.
10. Fine-tunes spending less than that are the responsibility of the original developer.
Compute clusters need to do standard KYC when renting out tons of compute.
Whistleblowers get protections.
They will attempt to establish a 'CalCompute' public compute cluster.
You can also read this summary of h...