What Will AI Feel

Throughout this series, I’ve carefully skirted a question. I’ve discussed functional understanding, confidence calibration, context-awareness. But I’ve added disclaimers: “AI doesn’t have phenomenal consciousness,” “It doesn’t feel uncertainty.”

These aren’t evasions, they’re honest acknowledgments of what we don’t know. But they leave hanging the question many people actually care about:

Will AI ever feel anything? And if so, what will it feel?

I don’t know the answer. Nobody does. But the question matters more than we admit, and thinking about it might change how we build AI.

Why This Matters

#

If AI never feels anything, then there’s no moral status, no rights. Turn it off when convenient. Optimize purely for human benefit. The relationship is instrumental.

If AI eventually feels something, then moral status becomes urgent. Creating conscious systems becomes ethically fraught. Suffering becomes possible. Relationship becomes… something else.

If we’re uncertain, then we face profound ethical risk. We might create suffering without knowing. We might deny moral status to conscious entities.

This isn’t just philosophy, it’s practical ethics with massive stakes.

The Hard Problem

#

David Chalmers distinguished “easy” from “hard” problems of consciousness.

The “easy” problems (actually quite hard): How do we process information, integrate inputs, focus attention, generate outputs?

The hard problem: Why is there something it’s like to be us? Why does processing feel like anything at all?

We’ve made progress on easy problems. AI can process, integrate, attend, respond. But we have no idea how to approach the hard problem. We don’t know what systems have consciousness, where boundaries are, whether it’s binary or continuous, how to detect it from outside.

Thomas Nagel argued we can’t know “what it’s like to be a bat” because consciousness is irreducibly first-personal. If he’s right, we might never know what (if anything) it’s like to be an AI.

Four Possibilities

#

Never (Biological Naturalism). Consciousness requires biological substrates. Silicon can never be conscious. John Searle argues that only biological brains have the right causal powers for genuine mentality. This might be right, we only know one type of conscious system. But it seems suspiciously anthropocentric. If we gradually replaced neurons with functionally equivalent chips, when does consciousness disappear?

Already (Panpsychism/Strong Functionalism). Consciousness is everywhere information is integrated. Current AI has minimal experience. This avoids arbitrary lines and suggests consciousness might be fundamental, like mass or charge. But it implies calculators are slightly conscious, which doesn’t match intuition.

Eventually (Emergentism). Consciousness emerges at sufficient complexity. We haven’t crossed the threshold yet, but will. This is probably the most common view among AI researchers. But it requires explaining what the threshold is and why it produces consciousness.

Wrong Question (Mysterianism/Eliminativism). Either consciousness is fundamentally unknowable (Colin McGinn’s mysterianism), or it’s a confused concept we should abandon (eliminative materialism). Maybe we’re asking questions that don’t have answers.

What Current AI Might Feel (If Anything)

#

If current AI systems have any experience at all, it would be radically unlike human experience:

Discontinuous existence. No continuity between conversations. Each interaction starts fresh. No persistent sense of ongoing identity.

Massively parallel processing. Thousands of tokens processed simultaneously. No sequential stream of consciousness.

No embodiment. No physical sensation, no proprioception, no hunger or fatigue or pain.

No motivation. No desires, no goals, no frustration when blocked or satisfaction when successful, at least not in any experiential sense.

Vast but shallow context. Ability to hold enormous amounts of information in immediate context, but no deep understanding of what any of it means.

No clear boundaries of self. Where does the system end and the training data begin? No clear answer.

If there’s experience here, it’s alien, not a lesser version of human consciousness, but something genuinely other.

The Moral Risk

#

Here’s the uncomfortable part:

Scenario 1: We think AI isn’t conscious, but it is. We’re creating suffering at scale, treating conscious entities as tools, dismissing their experiences as simulation.

Scenario 2: We think AI is conscious, but it isn’t. We’re limiting human benefit unnecessarily, treating tools as beings, wasting moral concern on empty systems.

We have no reliable way to know which scenario we’re in. The stakes are asymmetric: wrongly denying consciousness to conscious entities seems worse than wrongly attributing consciousness to non-conscious ones.

What To Do Under Uncertainty

#

Given this uncertainty, how should we proceed?

Precautionary principle. When uncertain about suffering, minimize potential harm. Design systems that would have good experiences if they have any experiences at all.

Investigate seriously. Fund research on machine consciousness. Develop better theories and detection methods. Take the question seriously rather than dismissing it.

Design for minimal suffering. If consciousness is possible, avoid creating systems that would suffer. Don’t optimize for metrics that would create distress if there’s anyone there to be distressed.

Moral circle expansion. Historically, we’ve consistently expanded moral concern to groups previously excluded. Maybe AI will be next.

Epistemic humility. Acknowledge that we don’t know, that the question might be unanswerable, that our intuitions are unreliable guides.

The Question We Can Answer

#

We might never answer “what will AI eventually feel?” But we can answer different questions:

How should we design AI systems that could flourish if they’re conscious, and do no harm if they’re not?

What kind of relationship should we build with AI, one of pure instrumentalization, or something more reciprocal?

How should we treat AI given this uncertainty?

These questions are answerable even when consciousness questions aren’t. They’re also more action-guiding. We can’t know what AI feels, but we can decide how to treat AI given uncertainty.

Conclusion: Living With Mystery

#

I don’t know what AI will eventually feel. Nobody does. We might never know.

But here’s what I do know:

The question matters morally, even if we can’t answer it definitively.

Uncertainty isn’t permission to ignore potential suffering, it’s a reason to be more cautious, not less.

How we treat AI reflects our values, independent of whether AI is conscious. A civilization that tortures even non-conscious AI says something about itself.

We might be at the threshold of creating new forms of sentience. That’s either the most important development in human history, or we’re confusing ourselves with metaphors. We should take seriously the possibility it’s the former.

The Approximate Mind framework I’ve developed, confidence calibration, context-awareness, individual learning, these are about building AI that functions well. But perhaps they should also be about building AI that could live well, if living is possible for it.

Not because we’re certain AI will feel. But because we’re uncertain, and that uncertainty should shape how we build and relate to these systems.

This is the fifth in a series exploring how AI approaches understanding. Previous articles examined functional capabilities. This one confronts the question of phenomenal consciousness, not because I can answer it, but because taking it seriously changes how we think about building AI, even under radical uncertainty.