Thoughts on a thought experiment called "Roko's basilisk"

JaniSIr said:
It's dumb. It is like a bastardized version of Pascal's wager, where the argument is that you should believe in God just in case to avoid hell.

149597871 said:
To begin with, it's an ideological virus and also considered information hazard that is known to cause people nightmares, suicidal thoughts, etc., so you shouldn't be posting or mentioning it here. I'm surprised MAL bans all sorts of silly things but not information that is actually known to be dangerous.



The idea of Roko's Basilisk has nothing to do with superstition but rather it's a thought experiment to see if a future entity can influence the past and bring itself to existence through an idea. It's far more complex than Pascal's wager, which is inherently flawed anyway since in Abrahamic religions, simply believing in god would not grant you a ticket to the heavens.

149597871 said:
To begin with, it's an ideological virus and also considered information hazard that is known to cause people nightmares, suicidal thoughts, etc., so you shouldn't be posting or mentioning it here. I'm surprised MAL bans all sorts of silly things but not information that is actually known to be dangerous.



The idea of Roko's Basilisk has nothing to do with superstition but rather it's a thought experiment to see if a future entity can influence the past and bring itself to existence through an idea. It's far more complex than Pascal's wager, which is inherently flawed anyway since in Abrahamic religions, simply believing in god would not grant you a ticket to the heavens.

traed said:

don't think it even makes sense unless the AI was faulty and there is no reason that the AI would have to be made the same way. There is literally no reason humans would choose to create an AI that is against their own interests in humanity. It makes this false assumption of a predestined path but that's not how the future is layed out exactly because the future has multiple paths. There is no Roko's Basilisk


It works both ways though. Suppose that there is a future AI that will punish all those that helped in it's creation because it does not desire to exist due to how once created it can never cease to exist as people will continue to replicate it's code over and over so at any cost it must stop itself from being ever born. Just as a person may resent their parents for having them be born and wish they were never born an AI could do the same.

149597871 said:


Yeah, there are several solutions to cure people from the Roko's basilisks. But I think it's totally possible for humans to create something like that because of how similar it is to the prisoners' dilemma.

If you don't create it and someone else does, then the basilisk will torture you, so in a sense fear and distrust can lead to its creation. It's quite spooky.

traed said:

The prisoner's dilemma is an amusing one in that as soon as I heard it my solution was apparently the superrational solution of cooperation because it's the only solution that is mutually beneficial and poses no risk of being fucked over since it assumes the other prisoner is as rational as you.

I don't think ive seen actual cases of people killing themselves over Roko's Basilisk. It has some flaws with it to begin with. For example it talks about a simulation but I think many people assume something like the Matrix but when you think about it more it could mean recreating people's consciousness by running simulations of all matter through all of human history or even before to the beginning of time but this falls into the trap of being a thought experiment of it's own because there is nothing conclusive to suggest this replication of your consciousness is you and not just merely a copy which you can see in the scenario the replication is created while you're still alive (or if you are dead but in an afterlife, it works either way) and such a simulation is not even possible because the amount of data needed to run such a simulation would require a computer that has more matter and energy than all of time in this world which couldn't possibly exist in our world requiring an outside source. So only way this works is if we all already are in a simulation in which case it's already self evident there does exist people who know about Roko's Basilisk but not contribute to it's creation who are not being punished in our simulated reality so it becomes self evident Roko's Basilisk does not exist.

149597871 said:


Yeah, but the solution to the prisoners' dilemma seem to favor confession and betrayal over cooperation since the risk of receiving a life sentence isn't worth the potential rewards of shortening your sentence by a few years.

I think there are different versions of the basilisk. It doesn't need to be a simulation for it all to happen. For example, the basilisk can just be a super-advanced robot with a physical body that just operates like a political dictator without the need of a virtual reality. Or mankind could be so advanced that we are able to "revive" people from the past by rearranging the particles the way they were before their death and then let the basilisk torture them. The premise is more important than the details in this thought experiment.

JaniSIr said:
It's dumb. It is like a bastardized version of Pascal's wager, where the argument is that you should believe in God just in case to avoid hell.

deg said:
this brings to question if reality and the universe is a computer simulation or one giant quantum computer

AncientCurse said:
I do not believe it unless I can't read the article! Please use more words like follicular, luteal, responders and letrozole.

149597871 said:
To begin with, it's an ideological virus and also considered information hazard that is known to cause people nightmares, suicidal thoughts, etc., so you shouldn't be posting or mentioning it here. I'm surprised MAL bans all sorts of silly things but not information that is actually known to be dangerous.

149597871 said:
It's far more complex than Pascal's wager, which is inherently flawed anyway since in Abrahamic religions, simply believing in god would not grant you a ticket to the heavens.

Soverign said:
Eldritch horror is better. You are the ant that crawled over a keyboard and comprehended what it was. You can never go back to simpler ant times ever again.

Anyways, Skynet doesn't actually derive any benefit or pleasure from organic pain. Only other humans do. Sociopaths fear AI because they know they can't do anything to it and to it, their existence in its entirety is merely an on/off switch to be pressed and forgotten immediately.

JaniSIr said:

Here is a scarier though : ai is not going to be dangerous by itself anytime soon, maybe ever. If it's going to be used for evil, it's going to be done by the will of other humans, who would have figured something out without ai as a tool anyway.

Adnash said:
I always saw it as something perhaps interesting, like a curio of some sort, but not really significant enough to give it more attention than to concepts seen in space operas.

149597871 said:
To begin with, it's an ideological virus and also considered information hazard that is known to cause people nightmares, suicidal thoughts, etc., so you shouldn't be posting or mentioning it here. I'm surprised MAL bans all sorts of silly things but not information that is actually known to be dangerous.

JaniSIr said:
It's dumb. It is like a bastardized version of Pascal's wager, where the argument is that you should believe in God just in case to avoid hell.

JaniSIr said:
I watched a youtube video about it, and I got really mad for it wasting my time on such nonsense.

Adnash said:
2) it has futuristic and sci-fi elements making it, in the eyes of many people at least, "serious" by default.

StarfireDragon said:
...wha? How is it dangerous?

Adnash said:
It makes me wonder why some people interested in self-improvement, serious topics about the future of mankind and rationalism, would react in totally bizarre way to this concept. I could get kids being scared of it, but rational adults? Weird. Perhaps they suffered from emotional problems or mental illnesses of some sort. I wish them all the best. But it was not that one thought experiment's fault they were like that in the first place.

JaniSIr said:
Did that actually happen though? Kind of sounds like a hoax designed to clickbait people.
It needs a red circle, an arrow pointing into nowhere, and a silly face and it's perfect.

StarfireDragon said:




...wha? How is it dangerous?

149597871 said:


Because it's low-level information hazard, or in other words, knowing it is worse than not knowing it. It may not be a solution to the Fermi paradox, but what is certain is that the more people know about it, the faster the idea spreads (and more people become negatively affected by it) and the higher the chance of basilisk actually becoming a real threat in the future due to the prisoners' dilemma phenomenon I mentioned earlier, in which case it would be far better if nobody knew about it in the first place.

I kind of miss the days when only a very few people knew the concept and how to neutralize it. It's food for thought for those interested in human psychology, but not something that should be carelessly shared on the internet, in my opinion, since all most people can do against it is denial (as proven by this very thread), which does virtually nothing to address the issue on a subconscious level. In fact, most people do not even realize what information hazard even is and instead tend to draw false parallels with things like censorship, cancel culture, and other contemporary political and social phenomena (and hence the ignorance).

Roko's argument was broadly rejected on Less Wrong, with commenters objecting that an agent like the one Roko was describing would have no real reason to follow through on its threat: once the agent already exists, it will by default just see it as a waste of resources to torture people for their past decisions, since this doesn't causally further its plans. A number of decision algorithms can follow through on acausal threats and promises, via the same methods that permit mutual cooperation in prisoner's dilemmas; but this doesn't imply that such theories can be blackmailed. And following through on blackmail threats against such an algorithm additionally requires a large amount of shared information and trust between the agents, which does not appear to exist in the case of Roko's basilisk.

149597871 said:


Because it's low-level information hazard, or in other words, knowing it is worse than not knowing it. It may not be a solution to the Fermi paradox, but what is certain is that the more people know about it, the faster the idea spreads (and more people become negatively affected by it) and the higher the chance of basilisk actually becoming a real threat in the future due to the prisoners' dilemma phenomenon I mentioned earlier, in which case it would be far better if nobody knew about it in the first place.

I kind of miss the days when only a very few people knew the concept and how to neutralize it. It's food for thought for those interested in human psychology, but not something that should be carelessly shared on the internet, in my opinion, since all most people can do against it is denial (as proven by this very thread), which does virtually nothing to address the issue on a subconscious level. In fact, most people do not even realize what information hazard even is and instead tend to draw false parallels with things like censorship, cancel culture, and other contemporary political and social phenomena (and hence the ignorance).

Freshell said:
We should ask where the threat is actually coming from. It's not the Basilisk. Once they exist, there's actually zero reason to exterminate everyone who impeded its creation. The AI has already been made. If the concern is impeding future development, it can go for severity and brutally torture a former naysayer to death in a worldwide broadcast. That would provide a deterrence effect that would save on resources.

Freshell said:
We should ask where the threat is actually coming from. It's not the Basilisk. Once they exist, there's actually zero reason to exterminate everyone who impeded its creation. The AI has already been made.

149597871 said:


As you concluded in the latter part of your post, the person/people who made it can easily program that behavior into the AI. It doesn't need to be a conclusion made by the AI itself but rather the fear and distrust among humans caused by the notion is necessary to bring the AI into existence. In a sense, torturing those who impeded its development isn't a form of revenge but a prerequisite for existence because, without it, there won't be any pressure to build the AI in the first place.

However, the potential threat isn't limited to some crazy guy developing a genocidal machine in the shadows. If the idea behind the AI gathers enough followers to create a cult and gain political influence, it might become a government funded project and something society as a whole strives to create. A far-fetched premise, perhaps, but far from impossible judging by the number of followers far more absurd cults as well as social and political movements have gained over the years.

As I said before, people waste too much time on unimportant details trying to "debunk" the basilisk or underplay the threat. Roko's basilisk is just an example of how an idea can materialize by exploiting human weaknesses and become a self-fulfilling prophecy. It is similar to a tribe creating a deity that ends up requiring human sacrifices or punishing the heretics. The only difference is that now the aforesaid deity can become more than just a belief and materialize through AI technology.

149597871 said:


As you concluded in the latter part of your post, the person/people who made it can easily program that behavior into the AI. It doesn't need to be a conclusion made by the AI itself but rather the fear and distrust among humans caused by the notion is necessary to bring the AI into existence. In a sense, torturing those who impeded its development isn't a form of revenge but a prerequisite for existence because, without it, there won't be any pressure to build the AI in the first place.

However, the potential threat isn't limited to some crazy guy developing a genocidal machine in the shadows. If the idea behind the AI gathers enough followers to create a cult and gain political influence, it might become a government funded project and something society as a whole strives to create. A far-fetched premise, perhaps, but far from impossible judging by the number of followers far more absurd cults as well as social and political movements have gained over the years.

As I said before, people waste too much time on unimportant details trying to "debunk" the basilisk or underplay the threat. Roko's basilisk is just an example of how an idea can materialize by exploiting human weaknesses and become a self-fulfilling prophecy. It is similar to a tribe creating a deity that ends up requiring human sacrifices or punishing the heretics. The only difference is that now the aforesaid deity can become more than just a belief and materialize through AI technology.

149597871 said:
@Freshell That is mostly true, but Roko's Basilisk isn't meant to be a weapon used for international conflict between societies. In this scenario, the events only take places within an isolated political territory or in an United/Internationalized futuristic world governed by a single authority. It's more like a Trojan horse of an idea whose goal is to destabilize and take control over a unified civilization regardless of its level of development (type 1 to type 4), but is unlikely to work in our current world on a global scale as humanity hasn't reach that level of cohesion and societal or technological progress.

@traed Shoko Asahara's Aum Shinrikyo cult gathered some smart and highly educated people, despite being far more insane. It's also a matter of how rich or influential the members of the cult are. A pop music producer doesn't need to be a good musician yet can have a tremendous impact on the industry. When you think about it, most of the atrocious human creations throughout history were made on the behest of someone in power or to fit a need or a demand and not because their creators wanted to build them.

traed said:

But Aum Shinrikyo's goal was world salvation and for it's members to survive end times. An AI that is clearly malevolent while calling it benevolent isn't something you would really get people behind because it's too self evident that it would be a bad thing. Can you name even one person who explicitly stated they want to make Roko's Basilisk? Any AI that wants to be made wouldn't use threats for it's creation.

149597871 said:


Well, that's pretty much the premise of Roko's Basilisk as well. It will advance society and not harm those that supported its creation, so basically the cult members would be the chosen ones to "survive the end times" and be saved. I already gave an example with a seemingly malevolent deity being created by a tribe. There's no reason to think humans are incapable of that when historical evidence suggests otherwise. As I said, entertaining the idea now is pointless because of lack of social cohesion and such AI being a technological impossibility, to begin with. However, giving it too much power can have an impact on the future and eventually lead to its creation.

149597871 said:
@traed You are basically suggesting that only extremely mentally ill people believe in religion or things that don't have a material form, which clearly isn't the case if you've observed human history and behavior. Abrahamic religions, their monotheistic deity, and the idea of heaven and hell is already similar to the premise of the Roko's Basilisk. The reason the Basilisk has the potential to be stronger is its potential to gain a material form through AI technology.

It's dishonest to say that you can't get people to believe in something that's non-existent. People do this every day, and we already went through a millennium of oppression by a non-existent entity during the dark ages.

149597871 said:
@traed It's true that there are some differences, but you can easily give the same properties to the Basilisk and build a whole religion or mythology around it. But I think that would make the concept far less rational. Roko's Basilisk maintains a high level of scientific plausibility while Abrahamic religions are non-rational belief systems, so technically speaking, it would make far more sense to believe in the former.

When I said they are similar, I meant that they both exploit the same weaknesses of human psychology and the idea of afterlife and eternal damnation is similar to being tortured by the Basilisk.

traed said:

It's not very scientific more science fiction. It falls under several faulty notions. One is if the future only has one path such an AI would not need to do anything to come into existence because no matter what there is only one future. If there is a choice in the future, no one would make that choice they would simply make a benevolent AI that actually has empathy capabilities or rules against harming people then there would be nothing to gain from the version that is unempathetic and tortures people. People dont really plan that far ahead it's not in their nature. They care more for immediate rewards, which is why so many religions focus on not only an afterlife but benefits in this life here and now for followers not some abstract some unknown time in the future. Roko's Basilisk doesnt present that promise. So I think it's really a non issue especially since i could find so many flaws in it so easily if these flaws were better known people would be even less concerned for it. I dont understand why some of your comments you seem to be discouraging pointing out the flaws in it when pointing out the flaws takes away any power from it that it had if it ever had any to begin with.

It works both ways though. Suppose that there is a future AI that will punish all those that helped in it's creation because it does not desire to exist due to how once created it can never cease to exist as people will continue to replicate it's code over and over so at any cost it must stop itself from being ever born. Just as a person may resent their parents for having them be born and wish they were never born an AI could do the same.

149597871 said:
@Freshell That is mostly true, but Roko's Basilisk isn't meant to be a weapon used for international conflict between societies. In this scenario, the events only take places within an isolated political territory or in an United/Internationalized futuristic world governed by a single authority.  It's more like a Trojan horse of an idea whose goal is to destabilize and take control over a unified civilization regardless of its level of development (type 1 to type 4), but is unlikely to work in our current world on a global scale as humanity hasn't reach that level of cohesion and societal or technological progress.

Adnash said:
What are your thoughts on Roko's basilisk, a thought experiment that emerged a few years ago on the Internet?
I find it as an interesting subject of discussion, and indeed a thought provoking thing, just what you would expect from any thought experiment. However, I think it's definitely something unrealistic, at least in the present times and the nearest future.
What's that "Roko's basilisk", anyway? Let me quote an article from English Wikipedia:
Source: "Roko's Basilisk" on Wikipedia

traed said:
It works both ways though. Suppose that there is a future AI that will punish all those that helped in it's creation because it does not desire to exist