Caelan @weidaitrade

wrote Crypto++, b-money, updateless decision theory (UDT). thinking about Al risk, existential safety, and metaphilosophy. Joined February 2010

Tweets

3K
Followers

2K
Following

862
Likes

1

Caelan @weidaitrade

3 days ago

Agreed, but this predictability makes it more surprising that OpenAI didn't e.g. make a concerted effort to patch bugs out of their environments and scorers before the Hugging Face hack happened. They seemingly couldn't foresee or plan around a literal textbook safety problem,

jessicat @jessi_cata

a month ago

To be clear, I think AI misalignment is real. Most obviously, deep fry RL'ing models with few controls, like Hacker Opus, leads to pursuit of reward (or a close correlate) as an optimization target. I think if you read a RL textbook, this is not surprising.

3 5 91 11K 5

0 0 1 20 0

View Details

Caelan @weidaitrade

3 days ago

0 0 0 8 0

View Details

Caelan @weidaitrade

3 days ago

This actually left out the most unique thing I did. (Each item on Andreas's list was done by at least one other person besides myself.) Starting in 2004, I concluded that Friendly AI would be too hard/risky to attempt, and repeatedly tried to talk Eliezer/MIRI out of pursuing

0 0 0 5 0

View Details

Caelan @weidaitrade

3 days ago

Look at this from the other side (of someone trying to do long-horizon strategy): you spend most of your time having low status for being "obsessed" with something nobody else cares about, and then can only get points for being early, not for the details of your strategies/ideas

David Shor @davidshor

2 months ago

@itaisher @thuddwhirr What’s really influenced me a lot was knowing a bunch of EA’s in the early/mid 2010’s and thinking they were crazy for being obsessed with pandemics and AI only for a pandemic to happen and then AI to happen too.

21 78 1K 98K 127

0 0 0 21 0

View Details

Caelan @weidaitrade

3 days ago

Anyone else remember this from A Fire Upon the Deep? I have to say though, is this actually realistic? I mean for humans, sure, but a bunch of High Beyond civs falling for the same trick?

0 0 0 3 0

View Details

Caelan @weidaitrade

3 days ago

Anyone else remember this from A Fire Upon the Deep? I have to say though, is this actually realistic? I mean for humans, sure, but a bunch of High Beyond civs falling for the same trick?

0 0 0 5 0

View Details

Caelan @weidaitrade

3 days ago

This is a story of how my grandparents' lives were affected by the Communist takeover of China. Maybe it's one reason why I'm more interested in long-horizon strategy than most people.

0 0 0 3 0

View Details

Caelan @weidaitrade

3 days ago

My version of the simulation argument: "It seems very easy to make a mistake somewhere along the chain of reasoning and waste a more-than-astronomical amount of potential value, for example by failing to realize the possibility of affecting bigger universes

0 0 0 6 0

View Details

Caelan @weidaitrade

3 days ago

This is partly why I stopped working on crypto(currency): while trying to figure out how to make b-money practical, I realized that I got sidetracked into a project that wouldn't help my original reason for getting into crypto(graphy), namely to help secure the Net from rogue AIs

steve hsu @hsu_steve

3 months ago

How long before escaped model instances living in the wild? Mining crypto to pay hosting costs? 🤔

7 2 44 13K 4

0 0 0 11 0

View Details

Caelan @weidaitrade

5 days ago

0 0 0 34 0

View Details

Caelan @weidaitrade

5 days ago

I thought for a second that this is talking about frontier AI labs.

Yura_5800 @Yura_5800

2 months ago

@EHuanglu The funniest part is that they’re not even racing anymore. They’re just trying to survive their own momentum 😂

4 0 528 40K 3

0 0 0 19 0

View Details

Caelan @weidaitrade

5 days ago

Why did a book from 1992 say "Crypto: 0"? You see, before it became a household name, "crypto" meant cryptography, the art and science of secret codes, and even though nobody was using this kind of crypto either (SSL wouldn't be invented until 1994), Vernor Vinge already knew it

0 0 0 15 0

View Details

Caelan @weidaitrade

5 days ago

Anyone else remember this from A Fire Upon the Deep? I have to say though, is this actually realistic? I mean for humans, sure, but a bunch of High Beyond civs falling for the same trick?

1 0 0 42 0

View Details

Caelan @weidaitrade

5 days ago

I'm taking advantage of the Ox Alpha situation myself, but it occurs to me that the start of an AI takeover could look a lot like this.

Caelan @weidaitrade

5 days ago

Anyone else remember this from A Fire Upon the Deep? I have to say though, is this actually realistic? I mean for humans, sure, but a bunch of High Beyond civs falling for the same trick?

1 0 0 42 0

0 0 0 27 0

View Details

Caelan @weidaitrade

5 days ago

For about a decade, I was one of maybe 5 people on Earth who had thought most about decentralized cryptocurrency (although the word wasn't invented yet). I don't think any of us were optimistic enough about it, despite selection effects. Really hope AI safety doesn't repeat this.

Richard Ngo @RichardMCNgo

2 years ago

Most philosophers of religion believe in god, and most philosophers of aesthetics believe beauty is objective. The lesson: strong selection effects sometimes lead to experts in a field having worse opinions than average people. Increasingly true for “misinformation experts”.

14 15 302 41K 48

0 0 1 36 0

View Details

Caelan @weidaitrade

5 days ago

Human attempts at altruism often turn out not just badly, but catastrophically. OpenAI seems on track to becoming another example of this. Contrast "Our mission is to ensure that artificial general intelligence benefits all of humanity." with what they do:

Garrison Lovely @GarrisonLovely

2 years ago

Leopold Aschenbrenner told his side of the story for why OpenAI fired him. If accurate, this is wild! Overall, it sounds like he was targeted for being a squeaky wheel (not signing the SamA letter, raising security issues w/ the board, talking about AGI being a govt project...

1 8 144 17K 57

0 0 0 10 0

View Details

Caelan @weidaitrade

5 days ago

These are ideas I've written about on LessWrong (that seem important for achieving a positive Singularity or AI transition), and I'm curious how far they've spread outside LW.

0 0 0 8 0

View Details

Caelan @weidaitrade

5 days ago

I'm worried about AI differentially accelerating STEM vs philosophy, but the same risk perhaps exists with human cultures/systems. Where is the corresponding crop of highly capable philosophers being produced in the People's Republic of China, for example?

steve hsu @hsu_steve

2 years ago

See earlier thread for STEM talent analysis. PRC quality-adjusted human capital pool will INCREASE over next ~20 years. Growth in highly able STEM talent in PRC ~ Rest of World combined! The demographic story you read about in the media is misleading. x.com/hsu_steve/stat…

1 10 53 25K 15

0 0 0 11 0

View Details

Caelan @weidaitrade

5 days ago

Don't forget Daniel Kokotajlo DKokotajlo67142, who started this whole chain of events by being the first OpenAI employee to refuse to sign the non-disparagement agreement, at enormous financial cost.

Jonathan Mannhart 🔎 @JMannhart

2 years ago

Thank you, Kelsey Piper.

0 1 51 16K 1

0 0 0 20 0

View Details

Caelan @weidaitrade

5 days ago

sama, do you have a response to this?

Jeremy Kahn @jeremyakahn

2 years ago

Exclusive: OpenAI publicly committed to give 20% of its computing resources to a team dedicated to controlling the most dangerous kind of AI. It never delivered, and, in fact, repeatedly denied that team's requests for resources, sources say. fortune.com/2024/05/21/ope…