/explore
|
You are here 
|
www.lesswrong.com | ||
| | | | |
www.greaterwrong.com
|
|
| | | | | A Defense of Open-Minded Updatelessness. You can support my work on Patreon. Iterated counterfactual mugging on a single coinflip begins like a classic counterfactual mugging, with Omega approaching you, explaining the situation, and asking for your money. Let's say you buy the classic UDT idea, so you happily give Omega your money. Next week, Omega appears again, with the same question. However, Omega clarifies that it has used the same coin-flip as last week. This throws you off a little bit, but you see that the math is the same either way; your prior still assigns a 50-50 chance to both outcomes. If you thought it was a good deal last week, you should also think it is a good deal this week. You pay up again. | |
| | | | |
www.greaterwrong.com
|
|
| | | | | (This post contains a lot of math, and assumes some familiarity with the earlier LW posts on decision theory. Most of the ideas are by me and Paul Christiano, building on earlier ideas of Wei Dai.) The September MIRI workshop has just finished. There were discussions of many topics, which will probably be written up by different people. In this post I'd like to describe a certain problem I brought up there and a sequence of ideas that followed. Eventually we found an idea that seems to work, and also it's interesting to tell the story of how each attempted idea led to the next one. | |
| | | | |
www.greaterwrong.com
|
|
| | | | | [Epistemic status: I may have gotten some things wrong so please point out any errors. I also papered over a lot of technical details in favor of presenting my intuitions. A lot of this is me thinking out loud. If you already know a lot about decision theory or want the gory details, I've listed a bunch of resources on decision theory at the bottom of this post.] Rationality is sometimes considered to be "maximizing expected utility". This seems pretty unambiguous but it turns out that what this means is pretty tricky to define, and different decision theories have different definitions of it. In this post, I will summarize causal decision theory, evidential decision theory, and updateless/functional/logical decision theory. | |
| | | | |
www.lesswrong.com
|
|
| | | ...could help us to prevent takeover attempts from more dangerous misaligned AIs created later. | ||
