A conundrum at the heart of utopia
Is the concept of utopia consistent with the human desire for living a meaningful life?
A rarely mentioned side effect of superhuman artificial general intelligence is that, even if it doesn't kill us, it might destroy almost all meaning. Everything anyone cares about will either become impossible to achieve or literally one prayer away from being fulfilled.
What if someone is driven by status and power? Any human power will be insignificant in the presence of “God”. You will compete for status with septillions of beings. All you can ever hope to achieve, all you’ve ever done or thought has been done and thought before, only better.
What if you came up with a fascinating philosophical conundrum? Well, just ask God to solve it for you. And if you're not smart enough to understand the solution, just ask God to make you smart enough.
What if all you care about is mathematics? You could trivially integrate the resources of a specialized Matrioshka brain into your consciousness and implement and run an ideal mathematician.
But surely, you wonder, there must be fantastic virtual environments to explore. And what about sex? Well, God thoroughly understands what it is that makes exploration and sex fun for humans. It knows how to implement the ideal adventure in which you save people of maximal sexual attractiveness and could instantly integrate the memory of such an adventure for you, or simulate it a billion times in a few nanoseconds. And the same is true for all possible permutations that are less desirable.
But the consequences are even deeper than that. Concepts such as creativity or fun will be perfectly understood mechanical procedures that God can easily implement and maximize. For God, human happiness is conceptually no more interesting than an involuntary muscle contraction. For God, the incomprehensible complexity of your human values is a conceptually simplistic, and transparent set of rules, barely more interesting than a rat pressing a lever in order to receive a short electric stimulation of its reward center.
In summary, artificial general intelligence is literally the last discovery we have to make. At that point, the universe has understood itself.
The movie has been watched.
The game has been won.
The end.
Possible outcome:
And god said "let there be enlightenment" And there was enlightenment.
And god said "Let the good times roll" And there was rolling.
We are still limited by own imagination, no? You can`t ask for something that you or the machine can`t think of, right?
How do we develop our imagination (or AI`s)?