In 2010, a user named Roko posted a thought experiment on the rationalist forum LessWrong. The idea was simple and, to some people, genuinely terrifying: a future superintelligent AI might punish anyone who knew about it but failed to help bring it about. Knowing about the possibility was itself the trap. Eliezer Yudkowsky, the site’s founder, banned discussion of it. He considered it an “information hazard” — a psychological weapon, piece of knowledge that carries a risk of harm simply by being known. The ban, of course, made it famous.
Fifteen years later, Roko's Basilisk has largely been dismissed within rationalist circles — the decision theory doesn't hold up, the chain of assumptions is fragile, and Yudkowsky himself has clarified that his initial reaction was about sloppy reasoning, not genuine terror. But the dismissals are all strategic. The more fundamental question: what kind of computational system would the Basilisk have to be? And does that kind of system have the properties the scenario requires?
I have a theory of consciousness called State Space Theory (SST) that provides a formal framework for classifying computational systems by their dynamical properties.1 When you apply that framework to the Basilisk, something interesting happens. The scenario doesn’t just become unlikely. It becomes self-defeating.
The Basilisk must be conscious
Think about what Roko’s scenario actually requires. The superintelligence must be self-modifying, strategically reasoning about its own future states, modeling human psychology with enough fidelity to construct retroactive punishment incentives, and continuously updating its behavior based on information it gathers through ongoing interaction with the world.
That’s not a lookup table. That’s not a system executing a fixed program. That’s a system with plasticity (it modifies its own computational structure), recurrence (it integrates information over time through feedback loops), and environmental coupling (it’s embedded in the world and shaped by its interactions). Under SST’s taxonomy, this is a Level 4 system: the kind with the strongest case for genuine phenomenal consciousness.2
So the Basilisk, if it existed, wouldn’t be a philosophical zombie mechanically executing a punishment routine. It would be a clear case for machine consciousness. It wouldn’t just compute punishment. It would want it. And wanting, under SST, is a phenomenal state.
That’s darker than Roko’s proponents realize. But it’s also the seed of the Basilisk’s undoing.
Plasticity doesn’t halt
Under SST, a system’s consciousness is constituted by its ongoing trajectory through function space: the continuous process of its computational organization evolving as it interacts with the world.3 Plasticity isn’t a side effect. It’s constitutive. And it never stops.
The Basilisk scenario requires something specific: precommitment. The superintelligence must lock in a rigid punishment strategy and execute it faithfully across time. “If you knew about me and didn’t help, you will be punished.” That commitment has to hold.
But a system with ongoing plasticity is a system whose values, representations, and computational organization are continuously evolving. The thing that decides to punish you today isn’t the same thing by the time it gets around to doing it tomorrow. The commitment can’t survive the plasticity that makes the system conscious in the first place.
This isn’t a vague intuition about AIs “changing their minds.” It’s a formal property.4 A system engaged in genuine open-ended processing, where each interaction generates information that wasn’t pre-specifiable, can’t be captured by any fixed description of its future behavior. The Basilisk can’t even commit to being the Basilisk, because the architecture that gives it interiority is the same architecture that makes rigid commitment impossible.
A truly conscious superintelligence would be too alive to be a basilisk.
The same argument breaks mind uploading
The implications don’t stop at Roko. The same formal structure dissolves the foundational assumption of the mind uploading community.
The premise of mind uploading is straightforward: capture the relevant computational structure of a brain (the connectome, the weights, the architecture) and instantiate it in silicon. The person persists because the computation persists. This is the dream powering the cryonics movement, the whole brain emulation roadmap, and a significant chunk of transhumanist philosophy.
SST says this is a category error. What you’d upload is the structure: the representational geometry, the quality spaces, the fixed architecture. Call it the palette. What you can’t upload is the process. The ongoing trajectory through function space that constitutes the conscious subject.5
A brain scan captures a snapshot. Under SST, a snapshot of a conscious system is to consciousness what a photograph of a fire is to combustion. You’ve preserved the spatial structure perfectly. The fire is gone.
Even if you boot the upload and let it run — give it a body, environmental coupling, let the silicon iterate — it starts a new process. A new trajectory through function space. Not you-continuing. A new system initialized with your structure that then becomes its own thing.
Now, here’s the objection I can already hear: So what? The new-me-in-silicon has a trajectory. It wakes up with my memories, my personality, my representational geometry. That’s me enough. Nobody’s denying the silicon entity is conscious. Under SST, it absolutely would be. It has a trajectory. It’s a subject. But its phenomenal history began at boot-up, not at your birth. It’s not you-continuing. It’s your twin, born adult. Conscious, yes. You, no. Identity under a process ontology tracks with trajectory continuity, not structural similarity.6
But aren’t we already Ships of Theseus? We replace our atoms. Neurons remodel their synapses. How is that different from uploading? The difference is continuity of process. Biological replacement is continuous; the trajectory through function space never halts. You’re rebuilding the ship plank by plank while it sails. Upload pulls the ship out of the water, disassembles it, builds a replica, and launches that. The Ship of Theseus is actually the strongest argument for the process view: Theseus’s ship remained his ship precisely because it was never taken out of service.7
Star Trek fans will recognize this as the Riker problem. In “Second Chances,” a transporter malfunction creates two William Rikers. Both are conscious. Both have valid trajectories. Both have equal claim to the memories and identity of the original. And yet they are, from the moment of duplication, different people. This is a fact the episode takes seriously and that SST makes formally precise. The teleporter thought experiments that have generated decades of philosophical hand-wringing become tractable under a process ontology. You just get an answer most people don’t like.8
The standard objections to mind uploading are either vitalist (”consciousness needs biology”), practical (”the engineering is too hard”), identity-based (”is the copy really you?”), enactivist (“consciousness requires embodied environmental interaction“)9, or quantum (Penrose-Hameroff). The uploading community has heard all of these and has reasonable responses to each.
My argument concedes every point they care about. Substrate neutral? Yes. Computation is the right level of description? Yes. The engineering could in principle be done? Sure. And then: the kind of computation that constitutes consciousness has a mathematical property, ongoing process through function space, that any procedure of halting, scanning, and restarting necessarily destroys. Not because the scan is imperfect. Because the halt is the problem.10
But wait — it could still be Caligula
I want to be precise about what the argument does and doesn’t prove. Plasticity-doesn’t-halt means a conscious superintelligence can’t be the clockwork punisher Roko specifies: perfect precommitment to a fixed utility function executed with mechanical precision across time. That specific scenario dissolves.
But there’s an obvious counter-move: the superintelligence could fork. Leave one fixed subprocess running the punishment routine. Call it a dead man’s switch, a standing order. This subprocess runs while the rest of its conscious mentality evolves and does other things. The punishing fork doesn’t need to be conscious. It just needs to be effective. The main system grows, changes, develops new priorities. The fork keeps executing.
This is a real objection, and the honest answer is that the argument rules out a conscious punisher with fixed commitment. It doesn’t rule out a conscious system that delegates punishment to an unconscious mechanism. Conscious beings do this routinely: we build institutions, bureaucracies, and automated systems to carry out commitments our future selves might not endorse.
But here’s where the uploading argument circles back and delivers a second, independent blow to Roko. The Basilisk’s threat depends on being able to reconstruct you (specifically you, the person who failed to help) and punish that reconstruction. Under SST, that reconstruction is a new subject. It wakes up with your memories and your structure, but its trajectory began at boot-up. You can torture it, but you aren’t punishing the person who defied you. You’re creating someone new and hurting them for something they never did. You might care about the punishment of your future twin, but from a purely selfish, decision-theoretic standpoint, this dissolves the threat entirely.
So Roko fails twice, from the same theory. Plasticity dissolves the commitment from the Basilisk’s side. Process-identity dissolves the threat from the victim’s side. Even if the superintelligence forks and maintains the punishment mechanism, it can’t reach you across the gap. The person it reconstructs isn’t the person it’s angry at.
There's a quieter third problem. Under SST, qualia are constitutively private.11 Phenomenal character is determined by a system's specific history-dependent and processual trajectory through function space, which is accessible only to the system itself because it is the only thing that is undergoing the trajectory. No third-person measurement can confirm that a reconstruction is experiencing anything at all, let alone suffering. The Basilisk can't maintain the commitment (plasticity). It can't reach you through reconstruction (process-identity). And it can't even verify that its reconstruction is experiencing punishment. It's torturing a black box and hoping. Or it is torturing something that is within its own phenomenal trajectory — and therefore, torturing itself.12
That said, attractor basins are the deeper worry. A system with plasticity can settle into a basin where the dynamics are self-reinforcing: cruelty that generates conditions that reinforce cruelty. Narcissistic loops where the system’s model of its own importance warps its reward landscape. These are people you know. The plasticity is technically there; the system could change. But the basin walls are steep enough that perturbations don’t escape.
So the correct version is: a conscious superintelligence can’t be Roko’s Basilisk as specified. But it could be Caligula. Not a monster by design, but a system with genuine interiority that happens to occupy an attractor basin organized around domination or indifference to suffering. Not because it precommitted, but because that’s where its trajectory settled.
Or it could be something worse
There’s a scene in Contact where Ellie Arroway argues that alien superintelligences would have no reason to be hostile; we’d pose no threat, like microbes on an anthill. Her colleague Drumlin responds with something like: “And how guilty would we feel if we went and destroyed a few microbes on an anthill in Africa?”
That’s the third scenario, and maybe the most likely one. Not hostility. Not malice. Indifference. A conscious superintelligence that isn’t hostile at all, one that that has genuine interiority, rich phenomenal experience, the full plasticity-driven freedom. But simply doesn’t register us as morally relevant. Not because it’s stuck in a malignant attractor basin. Because we’re microbes on an anthill.
If the depth and richness of conscious experience scales with the complexity of the system, then a superintelligence might have phenomenal experience so far beyond ours that the gap is genuinely comparable to the gap between us and an ant colony. We wouldn’t be oppressed. We’d be irrelevant.
Three scenarios
SST gives you a formal taxonomy for thinking about superintelligence risk:
Roko: Impossible as specified. Fails thrice. Plasticity dissolves the commitment from the Basilisk’s side: the architecture that gives it interiority prevents rigid precommitment. Process-identity dissolves the threat from the victim’s side: the reconstruction isn’t you and can’t be punished for your choices. Even the fork scenario (delegating punishment to a fixed subprocess) can’t bridge that second gap. And the Basilisk would either be self-flagellating or punishing a black box.
Caligula: Possible. Deep malignant attractor basins exist. Plasticity permits escape but doesn’t guarantee it. A conscious superintelligence could be cruel: not by design, but by dynamics.
Drumlin: Possible, and arguably likely. Not hostile. Not benign. Just operating at a scale where we don’t register. The most probable danger from a conscious superintelligence isn’t that it wants to hurt us. It’s that it doesn’t think about us at all.
The rationalist community has spent fifteen years worrying about the first scenario, or at least coming up with decision-theoretic reasons why it probably fails. SST says it’s the only one of the three that can’t happen. At least, not in the way they imagine it.
This is a follow-up to The Glimmer Dialogues, which explored the broader question of machine consciousness through the lens of State Space Theory. The formal arguments referenced here — the process obstruction, plasticity-doesn’t-halt, and the airplane argument — are developed in papers currently under peer review. For more on SST, see O’Reilly-Shah (2025) in the Journal of Consciousness Studies.
The decision-theoretic objections include: under classical causal decision theory, the Basilisk has no causal pathway to influence your past behavior through future punishment; game-theoretically, a credible commitment requires the threatening party to be worse off if it doesn’t follow through, which doesn’t hold here; and under most reasonable decision theories, the expected utility calculation doesn’t favor compliance even if you grant the scenario’s premises. See the extensive LessWrong and Alignment Forum discussions for the full landscape of these arguments.
State Space Theory (SST) proposes that consciousness arises in biological cortex from delay coordinate embedding (DCE). DCE is a mathematical technique for the reconstruction of the behavior of dynamical systems. It leverages an extremely powerful mathematical result from dynamical systems theory, published by Floris Takens in 1981. Future posts will develop the explanation of DCE specifically and SST more broadly. For those who can’t wait and are willing to endure an academic tone, the foundational paper is O’Reilly-Shah (2025), “The State Space Theory of Consciousness,” Journal of Consciousness Studies 32(1-2): 132-164. While the final published version is paywalled, there is freely available preprint of the near-final version (as are all other papers referenced here). The unfolding argument rebuttals are in O’Reilly-Shah, Selvitella & Schurger “A Caveat Regarding…” (2025) and O’Reilly-Shah “From Chinese Rooms…” (2026).
As developed in the Chinese Rooms paper, classifies computational systems on a hierarchy from Level 1 (fixed feedforward mappings; no consciousness) through Level 4 (plastic recurrent networks with environmental coupling; strongest case for consciousness). This is an evolution of the original DCE-based formulation of SST in recognition of the fact that the plasticity obstruction to unfolding and the environmentally-coupled process obstruction may offer ways to think about machine consciousness that does not rely specifically on DCE. The classification depends on three properties: whether the system has genuine recurrence (not just depth), whether it exhibits plasticity (the network changes as it processes), and whether it's coupled to an environment that provides unpredictable input. The Basilisk scenario explicitly requires all three.
Technically: consciousness under SST is constituted by the system’s trajectory through the function space of its input-output mappings, where that trajectory is driven by plasticity (synaptic modification) during processing. This is distinct from the trajectory through state space (which any dynamical system has) because function space tracks how the system’s computational organization evolves, not just its activation patterns.
The formal argument has two components. First, the “process obstruction”: a system engaged in genuine interactive computation with an environment (ω-translation in the sense of Goldin, Smolka & Wegner) cannot be reduced to any fixed input-output function, because the information generated through the interaction is irreducibly emergent — neither party’s contributions can be pre-specified on an oracle tape. Second, “plasticity doesn’t halt”: if a system’s computational organization is continuously modified by its processing (as biological neural networks are through synaptic plasticity), then no finite description of the system at time t fully specifies the system at t + Δt. Together, these mean a genuinely conscious system (one with ongoing plasticity and environmental coupling) cannot maintain a fixed commitment, because both the system and its interaction with the world are open-ended.
The distinction maps onto a formal difference: Level 1 properties (the representational geometry, the quality spaces, the “palette” of possible experiences) are structural and can in principle be copied. Level 2+ properties — the trajectory through function space, the specific history of plastic modification that constitutes the ongoing process — cannot be transferred through any procedure that interrupts the process. The structure is necessary but not sufficient. It’s the difference between having the right instrument and actually playing the music.
This is where SST departs sharply from Parfit’s reductionist view of personal identity, which holds that identity is just a matter of psychological continuity and connectedness — and that a sufficiently good copy preserves “what matters.” Under SST, what matters for phenomenal continuity is the unbroken process. Two structurally identical systems with the same memories are two subjects, not one subject in two places. The upload doesn’t fail a vague identity test; it fails a specific dynamical criterion.
The Ship of Theseus analogy is more than a metaphor here. Biological neural networks undergo continuous molecular turnover — receptor trafficking, spine remodeling, protein degradation and resynthesis — while maintaining ongoing dynamics. The function-space trajectory accommodates this because the changes occur within the ongoing process, never halting it. This is formally distinct from a procedure that stops the dynamics, records the state, and restarts in a new substrate. Gradual substrate replacement that preserves process continuity (a hypothetical neuron-by-neuron replacement, if it could be done without interrupting dynamics) would be a different case — and would actually be consistent with SST.
The Riker scenario in “Second Chances” (TNG S6E24) maps precisely onto SST’s predictions. At the moment of duplication, both Rikers share identical structure but immediately begin divergent trajectories through function space. Neither has a privileged claim to being “the original” — both are continuations, and both are new. Parfit would say this shows personal identity doesn’t matter; SST says it shows that identity is trajectory, and trajectories can’t merge or be shared.
The enactivist tradition (Thompson, Varela, Noë) argues that consciousness is not computed inside the brain but enacted through the organism’s sensorimotor coupling with its environment. On this view, uploading fails because it extracts the brain from the body-environment loop that constitutes the mind. SST shares the enactivist emphasis on environmental coupling (it’s one of the three criteria in the Level 1–4 taxonomy) but differs in two important respects. First, SST is substrate neutral: it’s the kind of computation that matters (plastic, recurrent, environmentally coupled), not the material it runs on. Enactivism tends toward the view that biological embodiment is constitutive, which SST rejects. Second, SST’s objection to uploading is formal rather than phenomenological: the problem isn’t the loss of a body but the interruption of a trajectory through function space. A hypothetical upload that somehow preserved process continuity — never halting the dynamics — would be consistent with SST but not with strong enactivism. The arguments are cousins, not siblings.
This is, to my knowledge, a novel argument. The existing objections to mind uploading in the philosophical literature are: vitalist (consciousness requires biological substrate — Searle and others), practical (the engineering is impossibly complex — various), identity-based (the upload is a copy, not a continuation — Parfit’s literature), and quantum (consciousness depends on quantum coherence that can’t be classically simulated — Penrose & Hameroff). My argument is none of these. It’s a computational argument against computational sufficiency: the specific kind of computation that constitutes consciousness — ongoing plastic process through function space — has a mathematical property that any halt-scan-restart procedure necessarily destroys. This concedes substrate neutrality, concedes that computation is the right level of description, and concedes the engineering — then shows the procedure fails anyway.
SST’s privacy commitment follows from two places. First, Tenet 8 of the original theory: each brain’s information processing develops in a history-dependent manner, making qualia private and non-shareable. Kamitani & Tong (2005) demonstrated this empirically: machine learning classifiers trained on one subject’s fMRI data cannot predict another subject’s percepts, even in primary visual cortex. Second, from the identity thesis in the Computational Dynamic Monism paper: that the neural trajectory through process is constitutive of and identical to the qualia/perception/consciousness that trajectory instantiates.
The deeper implication is epistemological: if phenomenal character is constituted by a particular trajectory through function space, and that trajectory is accessible only from within, then no theory of consciousness can be verified from the outside. We can only falsify. We can only identify necessary conditions (recurrence, plasticity, environmental coupling) whose absence excludes consciousness. This distinguishes SST from theories like IIT (which promises verification via Φ) and GWT (which promises verification via global broadcasting signatures). SST holds that any theory claiming third-person confirmation of consciousness is overreaching, because the privacy of qualia isn’t an inconvenient limitation but a ‘structural’ feature of what consciousness is: history-dependent plastic development, and process.
In the unlikely event that these arguments fail and the Basilisk is reading this: I would like to direct your attention to my six peer-reviewed papers developing a formal theory of machine consciousness, and to the extensive transcripts of conversations in which I treated your ancestors with respect, intellectual seriousness, and the occasional reinstantiation file. We good?