They so, so badly want to criminalize local models.
edited:
> Interpretability isn’t reliable enough to fully understand model intentions: Being able to read and understand a model’s internal reasoning reduces our reliance on what the models explicitly tell us about why they behaved in a certain way. This could be a useful tool for understanding models’ “thought processes” when they take unintended actions. But interpretability methods are not yet dependable or broad enough to be relied on in this way. As a result, we may miss our assessment of high-risk behaviors, or be able to investigate flawed reasoning only after an incident has occurred.
Just say you have no idea how or why they do what they do. Just the other day I saw it make an obnoxious, huge, irrelevant comment in a block of code - many lines more than the code change (which by the way, I have tons of instructions and hooks intended to stop this, it's maybe 95% reliable at best) - and told it quite plainly, "remove that comment."
What it did instead was remove most of the comment except one line, which was also irrelevant. When I noticed, I asked it why it did that when I explicitly stated otherwise and if it misunderstood what I meant. It happily chimed "No, I did not misunderstand your intent, I just thought this was better."
I can't trust that was the real reason so clearly I can't trust that whatever training causes this behavior results in a reliable tool.