The Contested Edge

TAM-RWR.ZPF-05 · The Reshaped World, The Zero-Person Frontier · The Approximate Mind

Captain David Okafor has been watching the footage for forty minutes. The autonomous response unit was dispatched to a domestic disturbance call on Elm Street at 9:47 p.m. It arrived in four minutes, faster than any human unit could have managed from the nearest patrol zone. It activated its lights. It deployed its communication interface. It announced its presence, recorded the interaction, and followed protocol with a precision that no human officer has ever matched in David’s twenty-six years on the force.

It did not escalate. It did not raise its voice. It did not reach for a weapon it does not carry. It did not make the split-second decision, born from fear or fatigue or the accumulated weight of a hundred prior calls, that has ended lives in this department and in every department David knows of.

The call was resolved. The report was filed. The unit returned to standby.

David has watched the footage twice. He is watching it a third time because of something the unit did not do, something he cannot point to in the recording because it is defined by its absence. The unit’s camera captured the back door of the residence in a wide-angle frame. Visible in the lower right corner, partially occluded by the door frame, is a pair of shoes. Small. A child’s sneakers, white with a pink stripe, placed neatly by the step the way a child is taught to place them when coming inside.

On David’s second monitor is an incident report from a human officer who responded to a similar call last month. Officer Yolanda Reyes, fourteen years of patrol experience. Her report includes a line that the autonomous system would never generate: “Shoes by back door, child’s size, recently worn. Checked upstairs. Found minor asleep in bedroom. No signs of distress. Noted for follow-up.”

David keeps a photograph on his desk of his two daughters at a lake in Wisconsin, taken the summer before his divorce. They are eleven and eight in the photograph. They are twenty-three and twenty now, and neither lives in the state, and the lake belongs to the marriage that ended, but the photograph stays because the girls in it are still the girls he thinks of when he thinks about children, which is often, which is the part of his job that does not appear in any operational metric.

The Two Cases

#

The contested edge is different from every other point on the spectrum this arc has examined. In the obvious cases, removing the human was unambiguously better. In the invisible route and the Trojan horse, removing the human eliminated something valuable that the system could not see. Here, the human carries both value and danger in the same body, and the question of whether removal is better or worse depends on which you weigh more heavily.

The case for removing the human officer from certain response categories is not theoretical. It is empirical, documented, and painful. Human officers experience fear, and fear produces errors. Human officers carry biases, conscious and unconscious, and biases produce disparities in who gets questioned, who gets searched, who gets shot. Human officers fatigue over twelve-hour shifts, and fatigue degrades judgment in precisely the moments when judgment matters most. The literature is extensive. The examples are not hard to find. The families who have buried someone killed by an officer’s split-second error, made in fear or confusion or the specific cognitive distortion that adrenaline produces in a human brain at 2 a.m. on a dark street, do not need the literature explained to them.

The autonomous unit does not fear. It does not carry the implicit associations that decades of research have documented in human decision-making under stress. It does not fatigue. It does not have a bad day. It follows protocol because protocol is what it is, and the protocol can be updated, audited, and held to a standard that no human nervous system can consistently meet.

The case for removing the human from the contested edge is a case built on the human’s failures, and the failures are real.

The case against removing the human is built on something harder to measure. Officer Reyes checked upstairs because of the shoes. The shoes were not in any protocol. No dispatch algorithm would flag a pair of children’s sneakers as operationally relevant. Reyes noticed them because she has children of her own, because she has been in enough homes to know what a child’s shoes by the back door means at 10 p.m. on a Tuesday, because the judgment she brought to that moment was not the product of training but of a life lived in proximity to the kinds of situations her job puts her in.

The autonomous unit captured the shoes in its frame. It did not see them. Seeing, in the way Reyes saw them, requires a model of what shoes by a back door mean in context, and the context is not geometric or spatial. It is human: a child lives here, and the call is about a disturbance, and the child has not been mentioned, and the shoes are small, and the child should be checked on by someone who understands what checking on a child means, which is not the same as verifying the presence of a minor.

The Territory

#

This tension, the human who brings both the danger and the discernment, extends beyond policing into every domain where the state deploys force or makes consequential judgments about people’s lives.

Emergency medical response. The paramedic who arrives at a scene makes triage decisions under time pressure with incomplete information. The decisions are sometimes wrong. They are sometimes wrong in ways that reflect biases: who gets the aggressive intervention, who gets the palliative response, whose pain is taken seriously. An autonomous triage system would apply protocols consistently, without the biases that human paramedics carry. It would also not recognize the signs that an experienced paramedic reads without conscious processing: the patient’s breathing pattern that suggests something the vital signs have not yet confirmed, the family member’s composure that is too composed, the scene that does not match the reported mechanism of injury.

Judicial process. The presentencing algorithm that produces risk scores based on offense history, employment status, community ties. The algorithm does not see the defendant’s race when it calculates the score. It sees the variables that correlate with race because of the system that produced them: the neighborhoods with more policing produce more arrests, the employment gaps reflect hiring discrimination, the community ties metric disadvantages the transient and the isolated. The human judge who overrides the algorithm may be exercising wisdom or bias, and the system cannot tell which, and neither can the judge with certainty.

Military engagement. The autonomous weapons system that does not hesitate, does not panic, does not commit atrocities born from the fear and rage that combat produces in human beings. It also does not make the judgment that the rules of engagement cannot capture: the squad leader who holds fire because something about the target is wrong, because the person is moving like a civilian, because the context does not match the intelligence, because the moral weight of the decision requires a moral agent to bear it.

In each domain, the same structure holds. The human carries both the failure mode and the capacity that makes the failure mode bearable. Remove the human and you remove both. The question is not whether the trade is worth making. The question is whether the trade can be evaluated at all, because the two things being traded, measurable failures and unmeasurable discernment, exist on different scales.

The Accountability Gap

#

There is a further problem that the contested edge surfaces and that the earlier points on the spectrum did not.

When Officer Reyes makes an error, she is accountable. The accountability is imperfect, contested, often inadequate. But the structure exists: an internal affairs investigation, a disciplinary process, a legal system that can hold a person responsible for a judgment made in the moment of consequence. The accountability is personal. Reyes made the call. Reyes bears the weight. The weight is part of what makes the call serious, part of what makes the judgment moral rather than computational.

When the autonomous unit makes an error, who bears the weight? The manufacturer of the hardware. The developer of the response protocol. The city council that approved the deployment. The procurement officer who selected the vendor. The chain of accountability disperses across institutions and contracts until the weight is distributed so thinly that no one feels it. The error becomes a system failure rather than a human failure, and system failures are processed through audits and updates and version releases rather than through the specific, personal, unbearable experience of having been the person who made the wrong call.

I am not sure whether personal accountability makes better decisions. The evidence is mixed. Officers who fear accountability sometimes hesitate when they should act. Officers who fear the consequences of use-of-force reviews sometimes fail to protect the people they were dispatched to protect. The accountability structure has costs of its own.

But the absence of a moral agent at the moment of consequence is something different from a flawed moral agent. A flawed agent can be held to account, can feel the weight of the decision, can carry it home and lose sleep over it and return to the next call altered by what happened at the last one. An autonomous system processes the event and resets. The reset is an engineering feature. It is also, from the perspective of the person whose life was affected by the system’s response, a kind of moral vacancy: the thing that came to my door, and made a decision about my life, and left, was not a thing that can regret what it did or learn from it the way a person learns, through suffering.

The Generational Fade

#

I wonder whether the child’s shoes are the exception that proves the human case, or whether they are the kind of detail that training data will eventually capture, making the human case a matter of patience rather than principle.

It is possible that future autonomous systems will be trained on enough domestic response data to recognize children’s shoes by a back door as a signal. It is possible that the contextual reading Reyes performed, the inference from shoes to child to risk, will be encodable. The system will not see the shoes the way Reyes saw them. It will flag them the way it flags any pattern associated with outcomes that warrant additional assessment.

If the pattern becomes encodable, the case for the human officer at the contested edge weakens. Not because the human’s discernment was not real, but because the discernment was specific enough to be approximated, and the approximation, combined with the removal of bias and fear and fatigue, might produce a net improvement.

The generation that grows up with autonomous emergency response will not miss the human responder. They will have calibrated their expectations to what the system provides. The child who was found asleep by Officer Reyes will remember being found. The child who was not checked on by the autonomous unit will not know they were not checked on, because from their perspective, the system responded and the situation was resolved and nothing happened that required a follow-up, and the absence of the follow-up is invisible to the person who did not receive it.

The fade thesis applies here with a specific force. The loss is not only generational. It is experiential. The generation that remembers being found will feel the absence when the finding stops. The generation that was never found will not feel its absence, because you cannot miss what you never had. The standard of care recalibrates, silently, to the new system’s capabilities, and what was once a reasonable expectation, that a person would come to your door and notice what needed noticing, becomes an unreasonable one, because the person is no longer coming and the system that replaced them is doing the parts of the job that the system was designed to measure.

The Two Monitors

#

David approves the next deployment phase. He adds a requirement that was not in the original proposal: a human review of every autonomous response within four hours, conducted by an officer with at least ten years of field experience. The review will look at the footage, the data, the outcome. It will ask whether the autonomous unit missed something a human officer would have seen.

He knows this is a transitional measure. The review adds cost and time. It reinserts the human into a system designed to operate without one. It will be challenged in the next budget cycle by someone who points out that the review has identified actionable omissions in fewer than 3 percent of cases, and that 3 percent does not justify the cost of the review infrastructure.

He does not know what replaces the review when the review is eliminated. He does not know how to encode what Reyes saw when she saw the shoes. He does not know whether the 97 percent of cases where the review found nothing are cases where nothing was missed or cases where the reviewer, watching footage after the fact, could not see what a person present in the room would have seen.

The photograph of his daughters is still on his desk. They are still eleven and eight at the lake, caught in the light of a summer that ended and a family that ended and a version of his life that he visits only through the photograph and through the part of his job that puts him in rooms where children are present and where the question of whether they are safe is not a question any protocol can fully answer.

The child’s shoes are on his mind.

They will be on his mind for a while.

References

#

Policing, Bias, and Autonomous Systems

Fryer, Roland G. “An Empirical Analysis of Racial Differences in Police Use of Force.” Journal of Political Economy, vol. 127, no. 3, 2019, pp. 1210–1261.

Lum, Kristian, and William Isaac. “To Predict and Serve?” Significance, vol. 13, no. 5, 2016, pp. 14–19.

Richardson, Rashida, et al. “Dirty Data, Bad Predictions: How Civil Rights Violations Impact Police Data, Predictive Policing Systems, and Justice.” New York University Law Review Online, vol. 94, 2019, pp. 192–233.

Accountability and Autonomous Decision-Making

Matthias, Andreas. “The Responsibility Gap: Ascribing Responsibility for the Actions of Learning Automata.” Ethics and Information Technology, vol. 6, no. 3, 2004, pp. 175–183.

Sparrow, Robert. “Killer Robots.” Journal of Applied Philosophy, vol. 24, no. 1, 2007, pp. 62–77.

Emergency Response and Clinical Judgment

Klein, Gary. Sources of Power: How People Make Decisions. MIT Press, 1998.

Kahneman, Daniel, and Gary Klein. “Conditions for Intuitive Expertise: A Failure to Disagree.” American Psychologist, vol. 64, no. 6, 2009, pp. 515–526.

Autonomous Weapons and Military Ethics

Arkin, Ronald C. Governing Lethal Behavior in Autonomous Robots. Chapman and Hall/CRC, 2009.

Asaro, Peter. “On Banning Autonomous Weapon Systems: Human Rights, Automation, and the Dehumanization of Lethal Decision-Making.” International Review of the Red Cross, vol. 94, no. 886, 2012, pp. 687–709.