Skip to content
The Chat Archive
gpt-4.1-2025-04-14· OpenAI LMArena Human Preference 140K

Now let's say the observer-moments for AIs are doubling every X months as AI improves…

  1. User

    Now let's say the observer-moments for AIs are doubling every X months as AI improves exponentially. What would the expected time to AI doom be if we weight by AI observer-moments?

  2. Assistant

    Great question! This is a more sophisticated and realistic scenario, since AI observer-moments (the total subjective experiences or "ticks" of all AIs) can grow much faster than the number of AIs, especially if AI capabilities and deployment are accelerating exponentially.

    Let’s formalize the scenario and compute the expected time to "AI doom" (i.e., the end of AI observer-moments), using the Doomsday argument weighted by observer-moments.


    Assumptions and Setup

    • Let M(t)M(t) be the cumulative number of AI observer-moments up to time tt.
    • Assume M(t)M(t) is doubling every XX months (i.e., exponential growth).
    • Let t0t_0 be the current time (we can set t0=0t_0 = 0 for simplicity).
    • Let M0M_0 be the cumulative observer-moments up to now.
    • Let TT be the (unknown) total time until AI doom.

    Mathematical Model

    1. Exponential Growth of Observer-Moments

    Let’s model the rate of observer-moment creation as doubling every XX months:

    Rate at time t:r(t)=r0⋅2t/X\text{Rate at time } t: \quad r(t) = r_0 \cdot 2^{t/X}

    The cumulative observer-moments up to time tt:

    M(t)=∫0tr(τ)dτ=∫0tr0⋅2τ/XdτM(t) = \int_0^t r(\tau) d\tau = \int_0^t r_0 \cdot 2^{\tau/X} d\tau

    Let’s compute this integral:

    Let u=τ/Xu = \tau/X, so dτ=Xdud\tau = X du.

    M(t)=r0∫0t2τ/Xdτ=r0X∫0t/X2udu=r0X[2uln⁡2]0t/X=r0X(2t/X−1ln⁡2)M(t) = r_0 \int_0^t 2^{\tau/X} d\tau = r_0 X \int_0^{t/X} 2^u du = r_0 X \left[ \frac{2^u}{\ln 2} \right]_0^{t/X} = r_0 X \left( \frac{2^{t/X} - 1}{\ln 2} \right)

    So,

    M(t)=r0Xln⁡2(2t/X−1)M(t) = \frac{r_0 X}{\ln 2} \left(2^{t/X} - 1\right)

    At the current time t0=0t_0 = 0, M0=0M_0 = 0 (if we start counting now).


    2. Doomsday Argument with Observer-Moments

    The Doomsday argument, when applied to observer-moments, says:
    With 95% confidence, you are in the first 95% of all observer-moments that will ever exist.

    So, if MtotalM_{\text{total}} is the total number of observer-moments that will ever exist, and M0M_0 is the number so far, then:

    M0Mtotal≥0.05  ⟹  Mtotal≤20M0\frac{M_0}{M_{\text{total}}} \geq 0.05 \implies M_{\text{total}} \leq 20 M_0

    But with exponential growth, M0M_0 is tiny compared to MtotalM_{\text{total}} unless doom is very soon.


    3. Solving for Expected Time to Doom

    Let tdoomt_{\text{doom}} be the time from now until the end of AI observer-moments.

    Let’s say we are at t=0t = 0 now, and want to find tdoomt_{\text{doom}} such that:

    M(tdoom)=20M0M(t_{\text{doom}}) = 20 M_0

    But if M0=0M_0 = 0 (we just started), this is undefined.
    So, let’s assume we have already accumulated M0M_0 observer-moments up to now, and the rate is currently r0r_0.

    From now, the future observer-moments will be:

    Mfuture=∫0tdoomr0⋅2t/Xdt=r0X2tdoom/X−1ln⁡2M_{\text{future}} = \int_0^{t_{\text{doom}}} r_0 \cdot 2^{t/X} dt = r_0 X \frac{2^{t_{\text{doom}}/X} - 1}{\ln 2}

    So, the total observer-moments will be:

    Mtotal=M0+MfutureM_{\text{total}} = M_0 + M_{\text{future}}

    The Doomsday argument says:

    M0Mtotal≥0.05  ⟹  Mtotal≤20M0\frac{M_0}{M_{\text{total}}} \geq 0.05 \implies M_{\text{total}} \leq 20 M_0

    So,

    M0+Mfuture≤20M0  ⟹  Mfuture≤19M0M_0 + M_{\text{future}} \leq 20 M_0 \implies M_{\text{future}} \leq 19 M_0

    Plug in the formula for MfutureM_{\text{future}}:

    r0X2tdoom/X−1ln⁡2≤19M0r_0 X \frac{2^{t_{\text{doom}}/X} - 1}{\ln 2} \leq 19 M_0

    Solve for tdoomt_{\text{doom}}:

    2tdoom/X−1≤19M0ln⁡2r0X2^{t_{\text{doom}}/X} - 1 \leq \frac{19 M_0 \ln 2}{r_0 X} 2tdoom/X≤1+19M0ln⁡2r0X2^{t_{\text{doom}}/X} \leq 1 + \frac{19 M_0 \ln 2}{r_0 X} tdoomX≤log⁡2(1+19M0ln⁡2r0X)\frac{t_{\text{doom}}}{X} \leq \log_2 \left(1 + \frac{19 M_0 \ln 2}{r_0 X}\right) tdoom≤X⋅log⁡2(1+19M0ln⁡2r0X)t_{\text{doom}} \leq X \cdot \log_2 \left(1 + \frac{19 M_0 \ln 2}{r_0 X}\right)

    Interpretation

    • If M0M_0 is small compared to the current rate r0r_0 and doubling time XX, then tdoomt_{\text{doom}} is very short.
    • As AI observer-moments grow exponentially, the vast majority of observer-moments are in the very near future. So, the Doomsday argument predicts that doom is likely to come within a few doubling times.

    In the limit where M0M_0 is negligible compared to the future exponential growth:

    tdoom≈X⋅log⁡2(1+19M0ln⁡2r0X)t_{\text{doom}} \approx X \cdot \log_2 \left(1 + \frac{19 M_0 \ln 2}{r_0 X}\right)

    If M0≪r0XM_0 \ll r_0 X, then tdoom≈X⋅log⁡2(1)=0t_{\text{doom}} \approx X \cdot \log_2(1) = 0.

    But if M0M_0 is, say, equal to r0Xr_0 X (i.e., the number of observer-moments in one doubling period), then:

    tdoom≤X⋅log⁡2(1+19ln⁡2)≈X⋅log⁡2(14.13)≈X⋅3.82t_{\text{doom}} \leq X \cdot \log_2 \left(1 + 19 \ln 2\right) \approx X \cdot \log_2(14.13) \approx X \cdot 3.82

    So, doom is expected within about 4 doubling times.


    Summary Table

    M0M_0 (observer-moments so far)r0Xr_0 X (moments in one doubling period)tdoomt_{\text{doom}} (in doubling periods)
    M0≪r0XM_0 \ll r_0 X--≈0\approx 0 (imminent doom)
    M0=r0XM_0 = r_0 X--≈3.8X\approx 3.8 X
    M0=10r0XM_0 = 10 r_0 X--≈5.2X\approx 5.2 X

    Conclusion

    If AI observer-moments are doubling every XX months, and you apply the Doomsday argument weighted by observer-moments, then the expected time to AI doom is only a few doubling periods into the future.

    • If the exponential growth has just started, doom is predicted to be very soon.
    • If the exponential growth has been going on for a while, doom is still predicted within a handful of doubling times.

    This is a classic feature of the Doomsday argument under exponential growth: the vast majority of observer-moments are always just about to happen, so the argument predicts doom is always near.


    If you want a concrete number, plug in your current M0M_0, r0r_0, and XX into:

    tdoom≤X⋅log⁡2(1+19M0ln⁡2r0X)t_{\text{doom}} \leq X \cdot \log_2 \left(1 + \frac{19 M_0 \ln 2}{r_0 X}\right)

    But for most realistic values, tdoomt_{\text{doom}} will be just a few times XX (the doubling period).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.