LW - ACX Covid Origins Post convinced readers by ErnestScribbler

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ACX Covid Origins Post convinced readers, published by ErnestScribbler on May 1, 2024 on LessWrong.
ACX recently posted about the Rootclaim Covid origins debate, coming out in favor of zoonosis. Did the post change the minds of those who read it, or not? Did it change their judgment in favor of zoonosis (as was probably the goal of the post), or conversely did it make them think Lab Leak was more likely (as the "Don't debate conspiracy theorists" theory claims)?
I analyzed the ACX survey to find out, by comparing responses before and after the post came out. The ACX survey asked readers whether they think the origin of Covid is more likely natural or Lab Leak. The ACX survey went out March 26th and was open until about April 10th. The Covid origins post came out March 28th, and the highlights on April 9th. So we can compare people who responded before the origins post came out to those who responded after[1].
We should be careful, though, since those who fill out the survey earlier could be different than those who filled out later, and this could create a correlation which isn't causal.
I used a Regression Discontinuity Design on the time of the response to see if there was a break in the trend of responses right at the time the Covid post went up. Figuratively, this compares respondents "right before" the post to "right after" so can help assuage the confound fears.
I find that the post made readers more likely to think that the origin was indeed zoonosis. And this is highly significant. Here are the results, in charts.
Analysis
Here is the number of responses over time, with the timings of the posts highlighted. We'll mostly just need the timing of the Covid origins post, which is around response 4,002.
I'm assuming that readers who responded to the survey after the post went up have read the post before responding. This is the post engagement data[1] which shows within a few days of posting, most views of the post already took place.
The ACX Survey asked respondents what they thought about Covid origins.
I substracted 3 from the questionnaire response, to analyze a centered scale, for convenience. Here are the sliding window averages of 1,000 responses.
There are some fluctuations, but quite clearly there is a break in the trend at the time of the post, with readers starting to give scores more towards zoonosis. Looks like the post lowered responses by about 0.5 points (this takes time to transition in the chart, because of the sliding window) There's not enough data to eyeball something about the Comment Highlights post.
Another way to look at the same data is using not a sliding window, but a cumulative sum, where the local slope is the average response. I detrended this, so that it has 0 slope before the Covid post, just for convenience again.
We very clearly see the break in the trend, and the slope comes out -0.52 points, similar to before. This is almost half a standard deviation, which is a pretty large effect. Needless to say it is extremely statistically significant. In fact, this effect made the Covid origins question the most highly correlated with response order of all survey questions.
As a placebo test, I also checked whether this effect exists for other responses, even ones correlated with Covid origins before the post, like views on Abortion, or Political Spectrum. I found nothing that looks nearly this clear. The effects are much smaller if any, and not highly significant.
I was curious if the post also had a polarizing effect, where readers became more likely to hold a stronger view after the post, i.e. Lab Leak proponents becoming more certain of Lab Leak, and zoonosis proponents becoming more certain of zoonosis. I don't find much support for this. The sliding window standard deviation of responses does not increase after the post. I'm not sur...