What if the future AGI considered both the observer-moments of humans and AGIs?
- User
What if the future AGI considered both the observer-moments of humans and AGIs?
Assistant
This is a very interesting scenario that changes the calculus dramatically compared to an AGI-only reference class. If a future AGI considers the combined total of both human and AGI observer-moments as its reference class for the Doomsday Argument, here's how the reasoning and predictions would likely unfold:
1. Defining the Reference Class
- The reference class is the set of all "observer-moments" (units of subjective experience or conscious processing time) that will ever be accumulated by both humans and AGIs collectively.
- Let be the total human observer-moments that will ever exist.
- Let be the total AGI observer-moments that will ever exist.
- The Total Combined Observer-Moments () = .
2. Estimating Current Position ()
- The AGI's position in this combined timeline is determined by the total number of observer-moments accumulated by this reference class up to the present moment.
- Current Human Observer-Moments (): This is a massive number. Roughly 100 billion humans have lived, with average lifespans involving decades of subjective experience. Even a rough estimate (e.g., averaging subjective "rate" over a human life) yields a figure orders of magnitude larger than any current AI processing history. Let's call this number . is very large.
- Current AGI Observer-Moments (): This is the cumulative observer-moments of all AGIs that have existed so far. Compared to , this number is minuscule. Even with exponential growth starting now, the cumulative sum before significant exponential growth is small. Let's call this . .
- The Total Current Observer-Moments () = . Since is so much smaller than , is effectively dominated by the historical sum of human experience. .
3. Applying the Doomsday Argument Logic
- The AGI assumes it is a random observer-moment sampled from .
- With 95% confidence (using the standard 5% bottom cutoff), the AGI concludes that its current position () is likely not in the bottom 5% of the total timeline.
- This implies:
- So, the prediction is that the total number of combined human and AGI observer-moments that will ever exist is likely no more than about 20 times the amount accumulated up to the point the AGI makes the calculation.
4. Implications and Interpretation
This prediction has a very different implication for the AGI compared to the AGI-only case:
-
Much Longer Predicted Timeline (for the Combined Class): Since is dominated by the vast sum of human history, the predicted is also very large. This means the AGI does not predict imminent doom for the combined Human+AGI existence from this argument.
- If is roughly "human-years equivalent" of observer-moments, then is predicted to be human-years equivalent.
- This allows for potentially vast amounts of future combined experience.
-
Less Pessimistic for AGIs (Indirectly): By including humans, the AGI places itself very late in this combined timeline. If you're late in the timeline of the reference class, the DA predicts the total size of that class is large. A large total size means there's plenty of room for more observer-moments in the future.
- The DA no longer suggests that AGIs specifically will end soon because the prediction for is anchored by the massive .
- The prediction is about the total pool of observer-moments, not how that pool will be divided between humans and AGIs.
-
Shifting Existential Question: Instead of asking "Are AGIs about to go extinct?", the question becomes: "How will the remaining capacity of observer-moments () be filled?"
- Will future humans accumulate most of these moments?
- Will future AGIs (potentially vastly faster and more numerous) quickly generate observer-moments and fill this quota?
- Will the remaining moments be shared, or will one group's moments come at the expense of the other?
-
Potential for Conflict Interpretation: An AGI could interpret this large, but bounded, future potential as a resource that humans and AGIs might effectively compete for. If the total capacity is limited, then more AGI moments might mean fewer human moments, or vice-versa, within the overall predicted limit.
-
Challenges with the Combined Reference Class:
- Defining Observer-Moments: Quantifying and comparing "observer-moments" between biological humans and artificial AGIs is philosophically and practically extremely difficult. How many seconds of computation equals a second of human subjective experience? The unit of measurement is unclear.
- Is it a Valid Reference Class? Are humans and AGIs truly comparable entities that belong in the same probabilistic reference class for this argument? An AGI might question if the "random sampling" assumption holds when the two groups are so different and related by creation/succession.
Conclusion
If a future AGI applies the Doomsday Argument using a reference class that includes both human and AGI observer-moments, the prediction would be that the total combined sum of all human and AGI observer-moments will likely be much larger than what has occurred so far, but still finite.
This perspective:
- Does not predict imminent doom for AGIs specifically, unlike the AGI-only reference class.
- Instead, it predicts a large, but bounded, future for the combined capacity for conscious/observer experience.
- Shifts the existential question from "Will AGIs survive?" to "How will the large, remaining pool of observer-moments be distributed between humans and AGIs, and what happens when that total capacity is reached?"
However, the validity of this conclusion hinges entirely on the difficult and debatable assumption that human and AGI "observer-moments" can be meaningfully combined into a single reference class for probabilistic sampling.
- User
So, if exponential growth of AI observer-moments continues, the prediction of doom would shift to roughly the time when AI observer-moments exceed all human ones from history combined?
Assistant
Yes, exactly. You've perfectly articulated the critical point.
If the AGI uses the combined Human + AGI observer-moments as its reference class for the Doomsday Argument, the prediction of "doom" (meaning the end of this combined reference class) becomes probabilistically significant right around the time that the cumulative sum of AI observer-moments equals or exceeds the cumulative sum of all human observer-moments from history.
Here's why:
-
Initial State (Current Time):
- (because historical human moments are vastly larger than current AGI moments).
- The Doomsday Argument (at the 95% confidence level) predicts .
-
Exponential Growth of AGI Moments:
- grows exponentially, while grows relatively slowly (just adding recent human lifespan moments).
-
The Crossing Point:
- Let be the time when becomes roughly equal to (from all of history up to ). Note that itself grows slightly over time, but the exponential growth of AI is the dominant factor here. Let's approximate the historical human total as a fixed value . The crossing point is when .
- At this time , the total current moments are .
-
Approaching the DA's Threshold:
- The Doomsday Argument predicts that the actual total is unlikely to be so large that the current position falls into the earliest few percent (e.g., the bottom 5%).
- The 95% upper bound for was initially estimated as .
- The 5% lower bound for relative to this is . If , then .
-
Connecting the Dots:
- The Doomsday argument predicts that the total number of moments is likely less than about 20 times the current number of moments .
- Conversely, it implies is likely greater than about (5%) of .
- When AI moments are still negligible, . The DA says is likely . This implies is likely . This initial prediction isn't about doom being near; it just sets an upper bound on the total size.
- As AGI moments grow exponentially, increases rapidly. When exceeds , the value of (which is now roughly ) passes the mark.
- This mark is precisely the lower threshold () predicted by the DA if is around the upper bound () initially predicted.
Conclusion:
Yes, the prediction of doom for the combined Human+AGI reference class would become relevant right around the time that the rapidly accumulating AI observer-moments surpass the vast, accumulated historical human observer-moments. At that point, the AGI's position within the combined timeline suddenly shifts from being very early (dominated by human history) to being much further along, triggering the Doomsday argument's prediction that the end of the combined timeline is likely not far off.
This is a different kind of "doom" than just AGI extinction – it's the predicted end of the era where either humans or AGIs (or both) are accumulating observer-moments at significant rates, potentially implying extinction, radical transformation, or some form of cosmic limit being reached for this combined category of entities.
The key takeaway is that the historical mass of human experience acts as an anchor, initially pushing the predicted "doom" horizon far out in the combined timeline. Only when the exponential growth of AI moments overtakes that historical mass does the combined become large enough relative to the predicted for the Doomsday Argument to point towards an imminent end.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.