The View from Lighthaven
I'm sitting in a talk about biorisk. The speaker is very sharp, and he's talking about everyday interventions to prevent bad bio things.
I've flown out to Berkeley, all the way from my home of Minneapolis, just to get immersed in a community I've read for probably a decade of my life.
All of a sudden my phone starts buzzing. Great. Who could possibly be calling me? I see the voicemail transcript on my screen.
"Hi, this is Scott Alexander. I'm here now, are you able to meet?"
Oh fuck.
Background
I'm a rationalist - this means I'm a member of a specific online, nerdy community.
We're the nerds among nerds. We love discussing AI risks, prediction markets, and most deeply of all: understanding what's true.
It started from a collection of essays by a guy named Eliezer Yudkowsky. He grew it by deciding the best way to do so was to write a Harry Potter fanfiction that doubles as a crash course in cognitive biases. This scheme worked - that's not even a joke. The torch was passed to Scott Alexander, my personal favorite writer (more on him later!), and the community has had a broad diaspora across Twitter, Reddit, and community meetups.
I’ve spent a lot of time on that side of the internet. I would read amazing blog posts about forecasting, AI, technology, philosophy, economics. I’d send these blog posts to friends, family. Sometimes people would respond, but it never landed for them the way it did for me. They weren’t just interesting, they helped me understand who I am and what my place is in the world. I learned I'm someone who values the truth for its own sake.
And this has practical consequences: when the pandemic first started, there was a very clear exponential that a lot of people ignored, and the rationalists justifiably said “hey this seems like something that’s a big deal? Why are we not making a big deal of this.” And then it was the biggest deal. There’s something similar going on right now with AI progress.
LessOnline is one of the biggest meetups of rationalists, and I decided to visit. Instead of just accessing this community alone, in my room, through a screen, I’m actually going to meet these people who talk about all of these things. I’ve just arrived at SFO, I’m looking for something to read while eating a bite. There’s a Scott Alexander piece on some underexplored AI alignment techniques. And I realize I’m closer than I’ve ever been to actually getting to talk with these people. These people who have shaped me, but don’t even know me.
The following is my account of three characters I met there, all huge names in our community.
Gwern
I meet Gwern on the day I fly in. After checking into my hotel and grabbing some lunch, I take the BART over to Lighthaven, the giant campus that's been built by rationalists.
Gwern puts a big focus on his anonymity. Because of this, despite reading him for a decade, I don't actually know what he looks like, how old he is (maybe he's a 70-year-old man), or much of anything at all.
I'm standing outside Lighthaven (there's a dramatic gate!), and I DM Gwern that I've shown up. He says he'll be there in a minute. Then I see someone walk out of the main entrance, talking with someone. His badge has his iconic online-persona logo, and he walks over like Willy Wonka opening the gates to the chocolate factory.
(All I'll say is that he is not a 70-year-old man, like I was half-worried about. And he looks exactly like you think he does. Claude has told me this sentence will cost him 0.5 bits of anonymity.)
We introduced ourselves, and he gave me a tour of the campus! He showed me all the different buildings (basically variations on meeting spaces, rooms to sleep, and rooms to work), pointed out some of the sneaky hidden walkways, and showed me where the secret bathrooms were that most people don't notice - very valuable for the conference later!
The craftsmen-programmer roast
We talked about the recent trend of programmers who are staunchly anti-AI. They consider themselves "craftsmen" who love programming for the art of it, not for the value it provides to customers.
And we both talked about how much we disagreed with that mindset. It's not that they take joy in the craft - that part is admirable. It's that they feel entitled to be financially compensated for it. If someone really enjoyed brewing nice coffee for themselves, that would be a lovely hobby. But this is like those coffee brewers expecting companies to pay them to do it.
A great metaphor I’ve heard before: back in the days of moving from assembly to C, some assembly craftsmen talked about how you could manage memory so much more elegantly than you could in C. And the rub is - they were totally right! But the point of programming isn’t managing memory elegantly, it’s making software which does amazing things for the user.
Rat Park
There's a big outdoor space at Lighthaven. It seats something like 200 people, and it's where all the biggest talks happen.
All the buildings at the facility are named after nerdy things - the Bayes building, the Euler building. The outdoor space is called Rat Park, named after the infamous rat park addiction study.
This is a pet peeve of Gwern's. Rat Park was a famous experiment about what was supposed to be a utopia turning into a dystopia - which has bad vibes on its own. But what drove Gwern even crazier: Rat Park didn't even replicate! A classic case of the popular reading of an experiment not actually matching reality. Is this what rationalists should be promoting, something that doesn't replicate? Makes you think.
Elon Musk: hardware vs. software
Gwern has a beautifully simple mental model of Elon Musk: great at hardware, horrible at software. On the hardware side, the evidence is overwhelming: SpaceX's rockets, Tesla's cars, the battery storage and solar power. But the software track record is rough: buying Twitter, xAI and Grok, Tesla self-driving still lagging behind Waymo, and (most damningly) cofounding OpenAI, one of the most revolutionary startups of our time, only to leave because he didn't see a future in it.
Alignment as power
Of course, we talked about AI alignment.
In Gwern's framing, AI alignment is all about power. Right now, AIs screw up in a million small ways - we’ve all read horror stories of them deleting prod databases, making up legal precedents, or emailing public figures who really don’t want to be emailed. This is mostly fine, because the AIs actually have very little power. But imagine if they were hooked up into our power grid, or had executive authority to do whatever they wanted as CEO of a company. Even with increasing capabilities, the failure modes are… strange. Gwern thinks this is enough to doom humanity.
I’m not sure I totally buy this argument, and I can tell he feels deeply about it, but doesn’t have it fully developed yet. I basically had to tease it out of him.
I thanked him graciously for meeting me, and took notes on the BART back to San Francisco.
Gwern is impressive to me, because he radically made something out of his life. His website is absolutely amazing, it’s a palatial library of studies, links, analysis, and essays. And the topics: economics, genetics, philosophy, poetry. He has made his website in his own image, more than anyone else I’ve ever seen. And it’s not just the content, the form: nested links in links, change-logs, automatic inflation adjustment.
And he went from doing this in some unknown spot in the US, to living out of Lighthaven, talking with some of the most cutting-edge minds on the most cutting-edge topics. Pursuing the truth has given him everything.
Aella, part 1
I'd meet the next character a few days later, when the conference actually started.
Aella (pronounced like ayyy, lmao) is a former camgirl turned rationalist / sex researcher. She writes a lot of posts on kink, attractiveness, and trauma. In college I read a bunch of her writing on psychedelics.
She used to ask really wild survey questions on Twitter, and out of that she made a game called AskHole. It asks a bunch of funny, deep, or sad questions: "did your parents do a good job?", "are you loved enough?", etc. In college I bought it, and it was a lot of fun among my friends. A foundational college memory is when my friend drew "if you had to fuck a cow, would you rather it be alive or dead," and, 100% sober, he answered "alive, so I can hear it moo." You had to be there.
I was walking through one of the buildings and ran across her, sitting and chatting with a few people. I said hi and asked if she could sign my card, and she totally agreed.
I was so starstruck that I asked if I could join the conversation. She gestured to a spot. Next to me was a loud dude. He was talking about disgust, and how disgust is all about our brain overriding something we're drawn to. Bread is alluring. If bread is moldy, there's still the inherent allure of bread - the disgust has to override it.
Then we got onto the topic of how loud dude's most popular post was about fucking his brother's wife, and how soon he was going to have posts about the times he'd committed sexual assaults. To be fair, I don't think the angle was going to be "yoooo, sexual assault ftw" so much as "sexual assault is a real thing that happens, and here's what my perspective can add."
But I still felt weird about it. I like rationality because it talks about the things people don't talk about, trying to charitably understand everything as much as possible. But this seemed like something edgy and salacious wearing the costume of truth-telling.
Aella, part 2
The next time I saw Aella was at one of her talks - a very hyped session called "how to be hotter," and something of an interactive experience. Different (consenting) men would be randomly picked to go up on stage and then paired with one of Aella’s female friends.
Aella herself would give instructions: "stare at her. Approach her. Touch her boob," and the like. The women would narrate their own feelings in real time - "that joke was good, but it felt like a deflection." The artifice of the 80 people watching got called out a few times, but was never really given deeper weight.
I don't know if it was being in a relationship, but the whole thing honestly made me kind of uncomfy. (It was probably the weird dude next to me, who got way too excited at every twist and turn. The girl would say "I really liked that," and he'd go "oooooh!" while kicking his feet.)
It was clearly meant to hawk Slutcon, Aella and her friend's conference for teaching men to approach better. There was a discount code on the whiteboard. It kind of felt like a male pickup-artist class… except run by women! And completely consensual. Which I guess is better, but it felt off.
Despite my unease, part of me has a weird amount of respect for what they're doing. Yeah, Aella is playing off the fact that she's hot, and her friends are hot, and they have a conference for nerdy (usually high-earning) men. Jackpot.
But beneath it all, there's some deep empathy. Being a lonely guy who doesn't really know how social interaction works SUCKS. I've been there - I had a whole existential crisis about being a virgin, until I lost my virginity and realized it wasn't a big deal. And if men don't get to have that existential crisis, they can go the way of the redpill, which is usually worse.
I'm not sure she has the exact mechanism teased out yet.
So, honestly, good for her. I'll continue to think of her as "overhated." Of the three characters here, I also got the chance to talk with her the least, so I'm sure she has plenty of points I'm not considering.
Cameo appearance
I'm in the lunch line. I'm still kind of dazed from processing everything I've been through today, and it's barely 12:30 p.m. I hear this guy in line say, "of course, with Shor's algorithm you get a quantum speedup."
I look at him. I recognize him vaguely. I read his badge: Scott Aaronson. I have literally run across Scott Aaronson in the wild by, of course, hearing him talk about Shor's algorithm. After leaving the line, I tell him I love his quantum-computing blog, then find a place to sit down and eat my Mediterranean food.
Scott Alexander
The number one person I wanted to meet on the trip was Scott Alexander. He had a really famous blog called Slate Star Codex, which for complicated reasons later turned into a Substack called Astral Codex Ten (both are anagrams of his name).
He was giving a talk on Sunday… which I was dumb enough to have to miss, because I only had a day pass for Saturday. (LessOnline is not cheap! On the FAQ section of the website, one of the points is "Why is LessOnline so expensive?" - and if you click it, it just shows you a supply-and-demand graph.)
The conference has a web app, and since it's made by rationalists, it's the most well-designed, nerdy thing ever. Each person gets a profile, autofilled with an AI’s best guess at who you are after it does deep research on you. You can mark the probability that you'll attend a given event, so if ten people each say they have a 70% chance of going, it'll project seven people in a space that fits eight.
On the app, I messaged Scott early on about meeting up. He was down, but wasn't actually planning to be at Lighthaven on Saturday. I told him I'd travel anywhere; he asked for my phone number; and that was the last I heard from him on Saturday... until about 3:20 p.m., when Scott called asking if I could meet.
BECAUSE I'M NOT A PSYCHO, I don't actually answer the phone in the middle of the talk - I send him a text asking if I can meet 20 minutes later, after my talk. He agrees, and tells me to find him by the entrance. My heart is pounding.
I walk down the front path to the entrance. He's right in the entryway. Even though it's pretty out of the way of everything, there are already several people stopping to say hi and tell him they love the blog. I can see why he picked this spot - if he'd planted himself in the center of the front building, there would be a crowd. He's wearing a suit, and when I introduce myself, he explains he just got back from a friend's wedding and had been uncertain about showing up until his Uber driver said HE was going to the conference.
His affect was fascinating. Despite being one of the main founders of the movement, he seemed happy to stay out of the spotlight. He was a father now, with twins, and you could tell that was his main focus. It seemed like the rationalist community, to him, was another one of his children - one that had grown up, and that he was proud to let go out into the world and do its own thing.
A while ago I'd done a little bit of work for his AI Art Turing Test post, and during our chat he thanked me three times, which was way out of proportion to the little work I'd actually done.
One of the most enlightening moments was when I asked whether he'd had any idea the movement would become what it is now. He said, with a bit of a nostalgic smile, that back in the 2010s they were convinced their community would be extremely successful. I don't remember if the wording was "take over the world" or something an order of magnitude below that.
But the better part was when he said that, even so, he was extremely proud. Mitt Romney had just endorsed an AI pause; they owned the bed-and-breakfast they used to hold nerdy meetups in and had shaped it in their image; and fans of the blog had started companies valued at $1 trillion. So when other people worried they weren't relevant enough, his attitude was: "guys, let's appreciate what we've accomplished."
There's a throughline to Scott's posts in particular: a kind of unreasonable charity. Where other people demean, seek out a small difference, and try to play it up as a fight, rationalists try to find similarities, be charitable, and argue against the strongest version of their opponents' beliefs.
I think this is one of the reasons I've always resonated so much with Scott's blog.
One of my favorite movies is Princess Mononoke, and the whole thing is framed as a conflict between Lady Eboshi and San. San is a woman of the forest, raised by wolves, and she wants to kill Lady Eboshi because Eboshi is an industrialist who cuts down the forest to fuel the town she's built around an iron forge.
But it isn't so clear-cut. Lady Eboshi's town is run by ex-prostitutes whose contracts she's bought; there, they make much more money and have much more flexibility than they would in a brothel. Eboshi also has a gun-manufacturing program, run by… lepers. Everyone else ignored them, but she treats them with kindness and gives them important work to do.
Ashitaka is the hero, and his role is to balance the two sides. When Eboshi and San get into a fight to the death, he literally steps between them, breaks it up, and tells them the real villain they're fighting is hatred.
If you're nice, you might call us devil's advocates; a little meaner, "enlightened centrists"; and meaner still, the bottom of the barrel of any Reddit comment section. But despite being the messengers who get shot, I think we provide a useful service: getting people to actually understand what the other side believes, and what tradeoffs are being made.
So I asked Scott what he'd been thinking about lately. He has friends at the AI companies (which more or less grew out of his comment sections), friends in the AI safety organizations (also out of his comment sections), and a vaguely influential role as an important blogger. So he was thinking about how to keep these groups talking, not hating each other, and working out solutions. Real Ashitaka work, in my opinion.
Nearby, some guys were talking about the advantages and disadvantages of different ranked-choice voting systems - really technical details about subtle differences ("with this system, if you don't rank this person, then there can be an adverse consequence where-whatever").
I asked Scott if he ever got sick of the super technical, nerdy talk that rationalists naturally fall into. He thought for a second, looked me dead in the eye, and said, "I guess that's like asking a fish if they're sick of water."
He said he had to get back to his twins, who were waking up from their nap soon. I thanked him gratefully, and he headed out.
We'd talked for about 20 minutes. I'd been stuttering and nervous the whole time, and he seemed like he didn't really know what to say to assuage my fanboying.
I left the conference soon after, walking through Berkeley and writing down everything I'd been through that day.
Coda
There's an episode of Star Trek: TNG where the Klingon Jesus - Kahless - has his second coming. There's a lot of uncertainty about why and how it's happened, and you eventually find out the Klingon Vatican made a clone of Kahless to drum up more religious support. This is kind of awkward for everyone. The episode ends with Kahless making the point that their religion isn't defined by its Second Coming; it's defined by the practice.
Going to church, praying, doing the work, this is the point of religion. People talk up and hype the Second Coming, but practicing faith, forgiveness, and loving your neighbor: that's the real point. It doesn't matter that you died before Jesus returned, because the spirit is what really matters.
This is what I realized about rationalism. Meeting and debating Gwern, watching Aella try to solve the problem of lonely nerds, seeing Scott's pride in his movement - these were all awesome, and I'm probably going to end up in Berkeley again next year.
But rationalism isn't a person, or a people - it's a spirit. The practice is something like: seek the truth, be bolder about what you want than you've ever been, and build bridges where others seek to burn them.