Stone and Mortar

Let us suppose there was a super-intelligent machine with godlike knowledge and superhuman abilities: would we follow its instructions? This seems possible. But if we did that, then the warnings expressed by Elon Musk, Bill Gates, Steve Wozniak, Stephen Hawking and others would have become true: computers would have taken control of the world. We must be clear that a super-intelligence could also make mistakes, lie, pursue selfish interests or be manipulated. Above all, it could not be compared with the distributed, collective intelligence of the entire population. -- Will Democracy Survive Big Data and Artificial Intelligence?

Intelligence means different ideas in different contexts. When someone uses the word for human, we more or less know what it means. If someone is able to understand the environment and act to maximize their benefits, it looks like an intelligent act for us.

For robots though, this line of reasoning is misleading. When someone speaks about superintelligence, they don't claim an artificial mind will surpass humans in humans' goals. It may or it may not, it's not relevant.

The real threat of superintelligence is not that it will be somebody that we can go and manipulate. We can lie to our digital assistants now, we can tell them we want to drive to a museum but instead our intention to see someone in the bar next to it. The naive robot will kindly show the way for us. How clever we didn't record that we're meeting with someone...

However this kind of interaction will leave to automated interaction as the time passes. Computers will be able to understand what we need, maybe checking our communication with one another, maybe doing quick blood tests, maybe reading signals of our brain... They will know what we are up to before us and lead us to there. It will be much easier that our current modus operandi. No need to talk, to tell, to instruct. The device in our pockets already knows to maximize our happiness (and maybe utility) to the society, we just obey to its instructions and enjoy the life.

Our current society looks like a bunch of rocks, which try to fit together. We are all hitting and pushing into each other, we are all trying to reach to the top by stepping on to others. What superintelligence will provide is a mortar to connect us to the rest. Our current society will look like a band of hunter gatherers when it will arrive.

In such a society there won't be lies, because there won't be any need to lie. Why do lie if your true desires are all opaque to the machine in your pocket and it knows how to reach it the fastest? And why do you need to think a rebellion against such an intelligence?