The Assessment Gap

TAM-RWR.ZPF-06 · The Reshaped World, The Zero-Person Frontier · The Approximate Mind

Keiko is building something. Not on contract. On her own time, in the apartment where Haru has claimed the left side of the desk and shows no intention of yielding it. The folder on her laptop, the one she started eight months ago, the one labeled “what the form doesn’t ask,” now contains annotated assessments for eleven deployments across six cities. She has read them all in sequence, twice, looking for the pattern she suspects is there.

She has also acquired a whiteboard. It sits against the wall where a bookshelf used to be, propped on a chair because she has not mounted it and probably will not, because mounting it would suggest a permanence she has not decided this project warrants. On the whiteboard is a grid she has been revising for three weeks. One axis is the spectrum she has observed across her deployments: obvious improvements at one end, unacknowledged losses at the other. The other axis is something she does not have a name for yet.

She has been calling it “relational load.” The term is not quite right. Load implies burden, and what the human was carrying in most of these cases did not feel like a burden to the person carrying it. Delores did not experience her knowledge of Mrs. Chen’s Tuesday moods as a load. Tomás did not experience the notebook as overhead. The term needs to capture the weight of what the human held without implying that the holding was onerous. She has not found the right word. “Relational load” is the placeholder, and placeholders, she has learned in three years of deployment consulting, have a way of becoming permanent because nobody goes back to fix them.

What the Standard Framework Measures

#

The standard deployment assessment, the four-page form that Keiko’s industry uses across municipal and nonprofit clients, was designed by competent people solving a real problem. Municipal decision-makers need to justify autonomous system deployments to elected officials, budget committees, and the public. The justification requires numbers: efficiency gains, cost reductions, service reliability improvements, safety outcomes. The form provides these numbers in a format that a city council presentation can absorb in twelve minutes.

The form measures what the system does. Delivery time. Error rate. Cost per unit of service. Uptime. Coverage expansion. Weather resilience. Each metric is legitimate. Each captures something real about the system’s performance. Taken together, they produce a portrait of a deployment that is working or not working, justifiable or not justifiable, expandable or not expandable.

Keiko has no quarrel with the form. The form does what it was designed to do. Her problem is with what the form was not designed to do, which is measure what the system replaced.

What the Standard Framework Cannot See

#

The relational function that the human carried was invisible to the system before automation. This is the structural problem, and the structural nature of the invisibility is what makes the assessment gap so resistant to correction.

The Meals on Wheels volunteer’s knowledge of Mrs. Chen’s Tuesday moods was not in any database. The pharmacy delivery driver’s notebook was not part of any reporting structure. The school bus driver’s awareness of which children were quiet in a way that meant something was not captured in any incident log until it became an incident. The connective tissue, the informal intelligence, the relational knowledge, all of it existed in a form that no institutional system was designed to record, because the institutional system was designed around the nominal function and the relational function was a byproduct.

Byproducts do not have metrics. They do not have baseline measurements. They do not appear in the “before” column of a before-and-after comparison, which means they cannot appear in the “after” column either, which means the assessment framework cannot show their loss, which means the loss is invisible to the people reviewing the assessment.

The gap is not an error in the framework. It is a property of what the framework was built to see.

This distinction matters because the response to an error is to fix it, and fixing an error implies that the correct framework exists and someone failed to build it. The response to a structural property is different. It requires acknowledging that the framework was built for a purpose, that the purpose was legitimate, and that the purpose did not include measuring what is now being lost, because what is now being lost was not measurable before the transition and is only identifiable through the absence it creates after the transition.

You cannot measure what was never counted. You can only notice that something is missing after it is gone, and by then the deployment has been approved, the expansion has been funded, and the framework that justified the decision is not the framework that would identify what the decision cost.

The Attempt

#

Keiko’s whiteboard is her attempt to build the framework that would identify the cost before the decision is made.

The concept is simple in outline. Before a deployment, the assessment should ask: what is the human doing besides the nominal function? The question is directed at the people closest to the operation: the program directors, the route supervisors, the colleagues who have worked alongside the person being replaced. Not as a survey. As a structured interview, conducted by someone trained to listen for the relational function that the job holder may not be able to articulate because it has never been named.

Sandra could answer this question about Delores. She could describe the cups, the orchid, the three flags. Tomás could answer it about himself, though he would not frame it as a relational function. He would frame it as “I notice things on the route.” Ray could answer it about the children. The knowledge exists. It exists in the people doing the work and in the people supervising the work. It does not exist in any system that a deployment assessment currently consults.

The second question: who depends on the relational function, and how? Mrs. Chen depends on Delores for her only regular human contact on Tuesdays and Thursdays. The nurse practitioner in Truchas depends on Tomás for information she cannot get through formal channels. The child on Ray’s bus whose welfare check prevented an escalation depends on Ray’s attention. The dependency is specific, identifiable, and documentable. It is also, in most cases, unknown to the people conducting the deployment assessment because the dependency was invisible to the system before the assessment began.

The third question: what happens to the people who depend on the relational function when it ends? This is where the assessment becomes difficult, because the honest answer is often “we don’t know,” and “we don’t know” is not a value that fits in the framework’s decision structure. Decision structures are built for values: costs, benefits, risks, probabilities. “We don’t know what happens to Mrs. Chen’s Tuesdays” is not a cost, a benefit, a risk, or a probability. It is an uncertainty of a kind that the framework was not designed to hold.

The Resistance

#

Keiko has shared early versions of her framework with four program managers and two municipal administrators. The responses fall into a pattern she did not anticipate but now recognizes.

The first response is interest. The framework names something the program manager has felt but has not been able to articulate within the existing assessment structure. “We’ve been worried about this,” one said. “We just didn’t know how to put it in the report.” The interest is real. The concern is real. The people running these programs are not indifferent to what the transition removes. They are working within a structure that does not have a place for what they know.

The second response is the practical question: what do we do with the information? If the framework identifies a high relational load at a deployment site, and the deployment is otherwise justified by efficiency gains and cost reduction and coverage expansion, what is the program manager supposed to do? Delay the deployment? Add a companion services contract? Report the relational load to the city council and let them decide? Each option has costs, and the costs come out of the savings the deployment was supposed to produce.

This is where the resistance becomes structural rather than attitudinal. The resistance is not that people do not want to know. The resistance is that knowing creates an obligation the system is not designed to fulfill. A framework that identifies relational load without providing a mechanism for addressing it creates awareness without obligation, and awareness without obligation is a specific burden that organizations learn to avoid, not because they are callous but because unfulfillable obligations erode institutional confidence in the assessment process itself.

If the framework says “this deployment will eliminate the only regular human contact for forty-three homebound recipients” and the deployment proceeds anyway, the framework has produced a record of a harm the institution chose to accept. That record has legal, political, and moral implications that the standard assessment, by not asking the question, avoids entirely. The standard assessment’s silence is not accidental. It is protective.

I wonder whether making the loss visible changes the decision, or whether visibility without obligation is the specific kind of knowledge that systems are best at acquiring and worst at acting on. Whether Keiko’s fifth page, expanded into a framework, adopted by a city, applied to a deployment, would produce a different outcome than the folder on her laptop: a record of what was lost, filed after the loss, read by people who are sympathetic and constrained and who do not know what to do with what they now know.

The Honest Limitation

#

There is something Keiko has come to understand about her framework that she did not understand when she started building it. The framework cannot replace what is lost. Even a perfect assessment, one that identifies every relational function, every dependency, every consequence of removal, cannot undo the structural condition that created the assessment gap in the first place.

The structural condition is this: the relational function was never designed. Nobody built Meals on Wheels to provide human contact. Nobody designed the pharmacy delivery route to be a county’s nervous system. Nobody trained the school bus driver to be a welfare monitor. The relational function emerged because human beings were present in systems over time, and human beings who are present in systems over time start to notice things, and the noticing accumulates into a form of knowledge that no system was designed to capture.

The framework can make the knowledge visible. It can document what the human was doing besides the job description. It can identify who depends on it. It can ask what happens when it ends. But it cannot build the infrastructure that would carry the relational function after the human is removed, because that infrastructure does not exist and has never existed and building it is not the work of a deployment assessment. It is the work of a society that has decided human contact is important enough to design for rather than rely on as a byproduct.

Keiko’s framework is a diagnostic instrument applied to a design failure. It can describe the failure with precision. It cannot fix it.

The Response

#

Keiko sends the framework, revised, to three program managers she trusts. Two respond within a week. They want to pilot it. They have deployments coming and they have concerns they cannot currently document within the standard assessment. The framework gives them a language for concerns that the existing structure treated as intuitions.

The third responds with a question she has not considered.

“If we measure the relational load and the number is high, what are we supposed to do with that information?”

Keiko reads the question on her phone, standing in her kitchen, Haru weaving between her ankles. She does not have an answer. She has the framework. She has the diagnostic. She has the questions that the standard form does not ask. She does not have the thing that would make the answers actionable, which is a theory of what society owes the people at the door when it removes the last person who was coming to see them.

She thinks the question is the most important one anyone has asked her. She writes it on the whiteboard, below the grid, in a handwriting that has not developed the compression of someone writing in a moving truck but has developed the slight urgency of someone writing before the thought escapes.

The framework is a tool for seeing. What to do with what it sees is a different question, and it is not hers to answer alone, and she knows this, and she also knows that if she does not ask it, the people who make the decisions will not ask it either, because the standard form does not have a field for it and adding the field is the work that nobody assigned and that she is doing anyway, in an apartment with a cat and a whiteboard and a folder that is no longer called “what the form doesn’t ask.”

She has renamed it. It is called “the map.”

References

#

Assessment Frameworks and Measurement Gaps

Muller, Jerry Z. The Tyranny of Metrics. Princeton University Press, 2018.

Porter, Theodore M. Trust in Numbers: The Pursuit of Objectivity in Science and Public Life. Princeton University Press, 1995.

Espeland, Wendy Nelson, and Mitchell L. Stevens. “Commensuration as a Social Process.” Annual Review of Sociology, vol. 24, 1998, pp. 313–343.

Structural Invisibility in Service Systems

Scott, James C. Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed. Yale University Press, 1998.

Star, Susan Leigh, and Anselm Strauss. “Layers of Silence, Arenas of Voice: The Ecology of Visible and Invisible Work.” Computer Supported Cooperative Work, vol. 8, no. 1, 1999, pp. 9–30.

Social Infrastructure and Design

Klinenberg, Eric. Palaces for the People: How Social Infrastructure Can Help Fight Inequality, Polarization, and the Decline of Civic Life. Crown, 2018.

Oldenburg, Ray. The Great Good Place: Cafes, Coffee Shops, Bookstores, Bars, Hair Salons, and Other Hangouts at the Heart of a Community. Da Capo Press, 1999.

Institutional Resistance to Difficult Knowledge

Rayner, Steve. “Uncomfortable Knowledge: The Social Construction of Ignorance in Science and Environmental Policy Discourses.” Economy and Society, vol. 41, no. 1, 2012, pp. 107–125.