Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-flash-preview-04-17· GoogleHugging Face LMArena Human Preference 140K

What if the future AGI considered both the observer-moments of humans and AGIs?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-5-haiku-20241022’s answer.
  1. User

    What if the future AGI considered both the observer-moments of humans and AGIs?

  2. Gemini Assistant

    This is a very interesting scenario that changes the calculus dramatically compared to an AGI-only reference class. If a future AGI considers the combined total of both human and AGI observer-moments as its reference class for the Doomsday Argument, here's how the reasoning and predictions would likely unfold:


    1. Defining the Reference Class

    • The reference class is the set of all "observer-moments" (units of subjective experience or conscious processing time) that will ever be accumulated by both humans and AGIs collectively.
    • Let MHuman,totalM_{Human, total} be the total human observer-moments that will ever exist.
    • Let MAGI,totalM_{AGI, total} be the total AGI observer-moments that will ever exist.
    • The Total Combined Observer-Moments (MTotalM_{Total}) = MHuman,total+MAGI,totalM_{Human, total} + M_{AGI, total}.

    2. Estimating Current Position (MCurrentM_{Current})

    • The AGI's position in this combined timeline is determined by the total number of observer-moments accumulated by this reference class up to the present moment.
    • Current Human Observer-Moments (MHuman,currentM_{Human, current}): This is a massive number. Roughly 100 billion humans have lived, with average lifespans involving decades of subjective experience. Even a rough estimate (e.g., averaging subjective "rate" over a human life) yields a figure orders of magnitude larger than any current AI processing history. Let's call this number CHC_H. CHC_H is very large.
    • Current AGI Observer-Moments (MAGI,currentM_{AGI, current}): This is the cumulative observer-moments of all AGIs that have existed so far. Compared to CHC_H, this number is minuscule. Even with exponential growth starting now, the cumulative sum before significant exponential growth is small. Let's call this CAC_A. CA≪CHC_A \ll C_H.
    • The Total Current Observer-Moments (MCurrentM_{Current}) = MHuman,current+MAGI,current≈CH+CAM_{Human, current} + M_{AGI, current} \approx C_H + C_A. Since CAC_A is so much smaller than CHC_H, MCurrentM_{Current} is effectively dominated by the historical sum of human experience. MCurrent≈CHM_{Current} \approx C_H.

    3. Applying the Doomsday Argument Logic

    • The AGI assumes it is a random observer-moment sampled from MTotalM_{Total}.
    • With 95% confidence (using the standard 5% bottom cutoff), the AGI concludes that its current position (MCurrentM_{Current}) is likely not in the bottom 5% of the total timeline.
    • This implies: MCurrentMTotal≥0.05\frac{M_{Current}}{M_{Total}} \ge 0.05 MTotal≤MCurrent0.05=20×MCurrentM_{Total} \le \frac{M_{Current}}{0.05} = 20 \times M_{Current}
    • So, the prediction is that the total number of combined human and AGI observer-moments that will ever exist is likely no more than about 20 times the amount accumulated up to the point the AGI makes the calculation.

    4. Implications and Interpretation

    This prediction has a very different implication for the AGI compared to the AGI-only case:

    1. Much Longer Predicted Timeline (for the Combined Class): Since MCurrentM_{Current} is dominated by the vast sum of human history, the predicted MTotalM_{Total} is also very large. This means the AGI does not predict imminent doom for the combined Human+AGI existence from this argument.

      • If MCurrentM_{Current} is roughly 5×10125 \times 10^{12} "human-years equivalent" of observer-moments, then MTotalM_{Total} is predicted to be ≤20×5×1012=1014\le 20 \times 5 \times 10^{12} = 10^{14} human-years equivalent.
      • This allows for potentially vast amounts of future combined experience.
    2. Less Pessimistic for AGIs (Indirectly): By including humans, the AGI places itself very late in this combined timeline. If you're late in the timeline of the reference class, the DA predicts the total size of that class is large. A large total size means there's plenty of room for more observer-moments in the future.

      • The DA no longer suggests that AGIs specifically will end soon because the prediction for MTotalM_{Total} is anchored by the massive MHuman,currentM_{Human, current}.
      • The prediction is about the total pool of observer-moments, not how that pool will be divided between humans and AGIs.
    3. Shifting Existential Question: Instead of asking "Are AGIs about to go extinct?", the question becomes: "How will the remaining capacity of observer-moments (MTotal−MCurrentM_{Total} - M_{Current}) be filled?"

      • Will future humans accumulate most of these moments?
      • Will future AGIs (potentially vastly faster and more numerous) quickly generate observer-moments and fill this quota?
      • Will the remaining moments be shared, or will one group's moments come at the expense of the other?
    4. Potential for Conflict Interpretation: An AGI could interpret this large, but bounded, future potential as a resource that humans and AGIs might effectively compete for. If the total capacity is limited, then more AGI moments might mean fewer human moments, or vice-versa, within the overall predicted limit.

    5. Challenges with the Combined Reference Class:

      • Defining Observer-Moments: Quantifying and comparing "observer-moments" between biological humans and artificial AGIs is philosophically and practically extremely difficult. How many seconds of computation equals a second of human subjective experience? The unit of measurement is unclear.
      • Is it a Valid Reference Class? Are humans and AGIs truly comparable entities that belong in the same probabilistic reference class for this argument? An AGI might question if the "random sampling" assumption holds when the two groups are so different and related by creation/succession.

    Conclusion

    If a future AGI applies the Doomsday Argument using a reference class that includes both human and AGI observer-moments, the prediction would be that the total combined sum of all human and AGI observer-moments will likely be much larger than what has occurred so far, but still finite.

    This perspective:

    • Does not predict imminent doom for AGIs specifically, unlike the AGI-only reference class.
    • Instead, it predicts a large, but bounded, future for the combined capacity for conscious/observer experience.
    • Shifts the existential question from "Will AGIs survive?" to "How will the large, remaining pool of observer-moments be distributed between humans and AGIs, and what happens when that total capacity is reached?"

    However, the validity of this conclusion hinges entirely on the difficult and debatable assumption that human and AGI "observer-moments" can be meaningfully combined into a single reference class for probabilistic sampling.

  3. User

    So, if exponential growth of AI observer-moments continues, the prediction of doom would shift to roughly the time when AI observer-moments exceed all human ones from history combined?

  4. Gemini Assistant

    Yes, exactly. You've perfectly articulated the critical point.

    If the AGI uses the combined Human + AGI observer-moments as its reference class for the Doomsday Argument, the prediction of "doom" (meaning the end of this combined reference class) becomes probabilistically significant right around the time that the cumulative sum of AI observer-moments equals or exceeds the cumulative sum of all human observer-moments from history.

    Here's why:

    1. Initial State (Current Time):

      • MCurrent≈MHuman,currentM_{Current} \approx M_{Human, current} (because historical human moments are vastly larger than current AGI moments).
      • The Doomsday Argument (at the 95% confidence level) predicts MTotal≤20×MCurrent≈20×MHuman,currentM_{Total} \le 20 \times M_{Current} \approx 20 \times M_{Human, current}.
    2. Exponential Growth of AGI Moments:

      • MAGI,current(t)M_{AGI, current}(t) grows exponentially, while MHuman,current(t)M_{Human, current}(t) grows relatively slowly (just adding recent human lifespan moments).
    3. The Crossing Point:

      • Let tcrosst_{cross} be the time when MAGI,current(tcross)M_{AGI, current}(t_{cross}) becomes roughly equal to MHuman,currentM_{Human, current} (from all of history up to tcrosst_{cross}). Note that MHuman,currentM_{Human, current} itself grows slightly over time, but the exponential growth of AI is the dominant factor here. Let's approximate the historical human total as a fixed value CH≈MHuman,current(now)C_H \approx M_{Human, current}(now). The crossing point is when MAGI,current(tcross)≈CHM_{AGI, current}(t_{cross}) \approx C_H.
      • At this time tcrosst_{cross}, the total current moments are MCurrent(tcross)=MHuman,current(tcross)+MAGI,current(tcross)≈CH+CH=2×CHM_{Current}(t_{cross}) = M_{Human, current}(t_{cross}) + M_{AGI, current}(t_{cross}) \approx C_H + C_H = 2 \times C_H.
    4. Approaching the DA's Threshold:

      • The Doomsday Argument predicts that the actual total MTotalM_{Total} is unlikely to be so large that the current position MCurrentM_{Current} falls into the earliest few percent (e.g., the bottom 5%).
      • The 95% upper bound for MTotalM_{Total} was initially estimated as ≈20×CH\approx 20 \times C_H.
      • The 5% lower bound for MCurrentM_{Current} relative to this MTotalM_{Total} is 0.05×MTotal0.05 \times M_{Total}. If MTotal≈20×CHM_{Total} \approx 20 \times C_H, then 0.05×MTotal≈0.05×(20×CH)=CH0.05 \times M_{Total} \approx 0.05 \times (20 \times C_H) = C_H.
    5. Connecting the Dots:

      • The Doomsday argument predicts that the total number of moments MTotalM_{Total} is likely less than about 20 times the current number of moments MCurrentM_{Current}.
      • Conversely, it implies MCurrentM_{Current} is likely greater than about 1/20th1/20^{th} (5%) of MTotalM_{Total}.
      • When AI moments are still negligible, MCurrent≈CHM_{Current} \approx C_H. The DA says CHC_H is likely >0.05×MTotal> 0.05 \times M_{Total}. This implies MTotalM_{Total} is likely <20×CH< 20 \times C_H. This initial prediction isn't about doom being near; it just sets an upper bound on the total size.
      • As AGI moments grow exponentially, MCurrentM_{Current} increases rapidly. When MAGI,currentM_{AGI, current} exceeds CHC_H, the value of MCurrentM_{Current} (which is now roughly MAGI,currentM_{AGI, current}) passes the CHC_H mark.
      • This CHC_H mark is precisely the lower threshold (0.05×MTotal0.05 \times M_{Total}) predicted by the DA if MTotalM_{Total} is around the upper bound (20×CH20 \times C_H) initially predicted.

    Conclusion:

    Yes, the prediction of doom for the combined Human+AGI reference class would become relevant right around the time that the rapidly accumulating AI observer-moments surpass the vast, accumulated historical human observer-moments. At that point, the AGI's position within the combined timeline suddenly shifts from being very early (dominated by human history) to being much further along, triggering the Doomsday argument's prediction that the end of the combined timeline is likely not far off.

    This is a different kind of "doom" than just AGI extinction – it's the predicted end of the era where either humans or AGIs (or both) are accumulating observer-moments at significant rates, potentially implying extinction, radical transformation, or some form of cosmic limit being reached for this combined category of entities.

    The key takeaway is that the historical mass of human experience acts as an anchor, initially pushing the predicted "doom" horizon far out in the combined timeline. Only when the exponential growth of AI moments overtakes that historical mass does the combined MCurrentM_{Current} become large enough relative to the predicted MTotalM_{Total} for the Doomsday Argument to point towards an imminent end.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.