LessWrong (30+ Karma)

Promoting enmity and bad vibes around AI safety

I've observed some people engaged in activities that I believe are promoting enmity in the course of their efforts to raise awareness about AI risk. To be frank, I think those activities are increasing AI risk, including but not limited to extinction risk. However, that's a stronger claim than I intend to argue here. Rather, I'll just be presenting a simple and harmful causal pathway and some strategies that can be used for mitigating it:

PromotingEnmity → Conflict → Catastrophe (PE→C→C)

(Enmity is not the same as conflict, which can sometimes can be constructive. Parties in conflict can be quite focussed on finding a mutually beneficial solution, even if that solution is difficult to find. By contrast, enemies do not generally pursue positive trade relations with each other. So, enmity is particularly relevant to watch out for when pursuing a positive future.)

Promoting enmity

Suppose groups X and Y are in a tense and dangerous relationship for some reason. If I say "Obviously X Leader and Y Leader hate and want to destroy each other", I'm promoting the hypothesis that they're enemies, and if they believe [...]
---
Outline:
(00:11) Promoting enmity and bad vibes around AI safety
(01:16) Promoting enmity
(02:11) Examples
(03:30) Is anyone actually promoting enmity like this around AI?
(04:56) How can promoting enmity increase AI risk?
(05:19) Can you moderate the promotion of enmity without escalating social violence?
(05:56) Moderation vs tone-policing
(06:48) Closing thoughts
---

First published:

March 9th, 2026


Source:

https://www.lesswrong.com/posts/A3rP5dQJnfARcWSpg/promoting-enmity-and-bad-vibes-around-ai-safety

---

Narrated by TYPE III AUDIO.

I keep running into similar arguments online, where people attack “the other” and use the (correct) observation of badness to claim their side is therefore doing well. There's a temptation to correct this by saying that in a dispute between two sides, one side being bad isn’t causally making the other better, or asserting that badness of the two are not correlated.
This is tempting, but wrong - because they are correlated, in the opposite direction, and that leads to my observation:
Manheim's Law of Positive-Sum Badness: In polarized disputes, evidence that one side is stupid, malicious, or evil increases the probability that the opposing side is too.
As the name points out, the badness isn’t zero-sum: both camps can be stupid, reasoning poorly, being annoying, and/or factually mistaken, and they often are several. The law isn’t saying the two sides are equivalent, and the observed side should get most of the update, but the mechanisms generating dysfunction in one camp tend to reach the other as well.
The underlying claim is Bayesian. Observing evidence E about side A shifts the posterior over both sides’ quality, with a much larger shift for A than for B. But the claim [...]
---
Outline:
(02:55) Correlational Variants
(03:28) Selection and Visibility Bias
(05:28) Coalitional Contamination
(07:05) Environmental Decay
(09:19) Causal Variants
(09:31) Trifecta of Escalatory Failures
(09:45) Contagion and Imitation
(11:46) Backlash and Reactive Deformation
(13:47) Symmetric Arms-Race Dynamics
(15:31) Adversarial and Mediated Causality
(17:23) Incentive Poisoning (or: Adversarial Environmental Decay)
(18:43) Provocation and Trap-Setting
(20:16) When the Law Doesnt Apply
(22:53) Motivated invocation
---

First published:

March 8th, 2026


Source:

https://www.lesswrong.com/posts/HZCEoFRKdzFAKcgFj/the-law-of-positive-sum-badness

---

Narrated by TYPE III AUDIO.

On 3 September 2022, Igor Kiriluk suddenly died (see EA Forum obituary). He was a great communicator and organized the first Moscow EA meetup. He had been active in the transhumanist scene since 2003 and even attracted the founders of KrioRus to transhumanism. He worked as community glue and visited every meetup.
On 5 January 2026, his closest friend Veter (Sergey Kamenev) had a dream that Igor's sideload (a mind-model based on an LLM) was created and that Igor calls him and suggests they celebrate his birthday. This was a signal I had been waiting for –I had wanted for some time to create a mind model of a dead person. I immediately started working.
We got an archive of his private communications, 2,000 pages, and 1,300 pages of scientific publications. His mother decided to collaborate and provided a lot of memories, recordings, and photos. Only two long videos exist. Altogether it is a 4,000-page mindfile = 3M tokens.
I decided on a new approach ­– instead of building a passive chatbot, to program an agent. After some experiments, I ended up giving Claude Code an instruction: roleplay this person as an agent – along with a lot of [...]
---
Outline:
(04:22) Post by Igors sideload
(04:42) System Overview by Claude Code
---

First published:

March 8th, 2026


Source:

https://www.lesswrong.com/posts/3g5pSQTpdw4K98w8M/recreation-of-ea-pioneer-igor-kiriluk

---

Narrated by TYPE III AUDIO.

The context is MIRI's twist on Axelrod's Prisoner's Dilemma tournament.
Axelrod's competitors were programs, facing each other in an iterated Prisoner's Dilemma.
MIRI's tournament is a one-shot Prisoner's Dilemma, but the programs get to read their opponent's code.
Or, rather, a description of the behavior of the code in Gödel-Löb provability logic, which turns out to be enough to determine their behavior in the setup.

One fun result, right in the beginning of the paper, is about a program, FairBot, whose behavior is specified by "I'll cooperate with you if you (provably) cooperate with me".
Despite the appearance of circularity, FairBot cooperates with itself.
The proof involves Löb's theorem, so we call this Löbian cooperation.

Andrew Critch has suggested another way of proving self-cooperation.
Instead of Löb's theorem, we use what he calls "Payor's lemma".
It suggests a different way of defining a FairBot, something more like "If, when I hypothetically cooperate with you, you would cooperate with me, then I really will cooperate."

This post is my attempt to explain why I think this approach is more promising, or at least why I like it more.

When thinking through these kinds of reflection problems [...]
---
Outline:
(02:04) Löbian FairBot
(03:03) Payorian FairBot
(04:02) Kripke frames
(07:15) Proving cooperation with Kripke frames, and CooperateBot
(09:23) One Payorian FairBot will cooperate with another
(10:30) Why this procedure feels right to me
(11:23) The sense in which this is simpler than Löbian cooperation
---

First published:

March 9th, 2026


Source:

https://www.lesswrong.com/posts/LaCP6WyNzX8kiZn3w/payorian-cooperation-is-easy-with-kripke-frames

---

Narrated by TYPE III AUDIO.

The Fifth Fourth Postulate of Decision Theory
In 1820, the Hungarian mathematician Farkas Bolyai wrote a desperate letter to his son János, who had become consumed by the same problem that had haunted his father for decades:
"You must not attempt this approach to parallels. I know this way to the very end. I have traversed this bottomless night, which extinguished all light and joy in my life. I entreat you, leave the science of parallels alone... Learn from my example."
The problem was Euclid's fifth postulate, the parallel postulate, which states (in one of its equivalent formulations) that through any point not on a given line, there is exactly one line parallel to the given one. For over two thousand years, mathematicians had felt that something was off about this postulate. The other four were short, crisp, self-evident: you can draw a straight line between any two points, you can extend a line indefinitely, you can draw a circle with any center and radius, all right angles are equal. The fifth postulate, by contrast, was long, complicated, and felt more like a theorem that ought to be provable from the others than a foundational assumption standing on its [...]
---
Outline:
(00:09) The Fifth Fourth Postulate of Decision Theory
(04:58) A Tale of Two Utilities
(09:49) Independence Is Sufficient but Not Necessary for Avoiding Exploitation
(09:55) The strongest case for independence
(12:31) Sufficiency, not necessity
(14:08) Resolute choice
(15:10) Sophisticated choice
(16:36) Ergodicity economics as a naturally resolute framework
(19:26) The broader landscape
(21:17) Allais and Ellsberg Behavior Is Rational
(21:21) Allais Paradox
(25:40) Ellsberg Paradox
(29:37) How LessWrong Has Engaged with This
(30:05) Armstrongs Expected Utility Without the Independence Axiom (2009)
(32:20) Scott Garrabrants comment (2022) -- Updatelessness and independence
(35:50) Academians VNM Expected Utility Theory: Uses, Abuses, and Interpretation (2010)
(38:37) Fallensteins Why You Must Maximize Expected Utility (2012)
(42:40) Just Give Up on EUT
---

First published:

March 8th, 2026


Source:

https://www.lesswrong.com/posts/MsjWPWjAerDtiQ3Do/on-independence-axiom

---

Narrated by TYPE III AUDIO.