The Score

TAM-WTR.08 · The Waiting Room · The Approximate Mind

Maria has a folder in the kitchen drawer. It contains her son’s birth certificate, his social security card, his high school diploma, his first pay stub from the grocery store where he has worked for two years, bagging the same groceries that Margaret buys on Tuesday, at the same self-checkout stations that replaced the registers where Diane used to work. Maria has been assembling this folder since he was born. Each document earned, each document filed, each one a proof of something. She thought it would be enough.

Her son is eighteen. His name is Daniel. He applied for his first apartment last month, a one-bedroom on the east side of town, not far from the bus line Maria depends on, close enough that she could walk there in twenty minutes if she needed to. The building is not new and not old. The rent is what he can afford on the grocery store wages, which is not much, which is the amount available when you are eighteen and bagging groceries and trying to do the thing the world says you are supposed to do, which is move out, be independent, start.

He was denied. The reason is a number. He has not been told the number, because the number is proprietary, but he has been told the denial is based on a risk assessment that considers factors including credit history, rental history, and predictive indicators of tenancy reliability. Daniel has no credit history because he is eighteen. He has no rental history because he has never rented. The predictive indicators are predictions about people who look, statistically, like him.

The folder in the kitchen drawer did not help. The folder contains proofs. The system does not accept proofs. It accepts scores.

The Room With No Room

#

Every institution Margaret has encountered in this town has a room. The pharmacy has a counter. The bank has a lobby. The doctor has an exam room. The library has chairs. The DMV had Window 4. Even the grocery store, even the self-checkout, is a building you enter and move through and leave. The institution has a physical address, posted hours, a person behind the counter whose face you can see.

The institution that denied Daniel has no address. It has no hours. It has no counter. It has no waiting room, no plastic chairs, no numbered tickets. There is no Window 4 to approach, no Robert to listen for forty minutes, no Linda to pause and ask a question. The institution exists as a model running on a server maintained by a company Daniel has never heard of, producing a number he has never seen, applied by a landlord who did not make the decision so much as receive it.

The power is real. The encounter is not.

Daniel experienced an institutional decision about his life, a decision that determines where he sleeps, that shapes the next year of his effort to become independent, that tells him what kind of risk he represents to people who have never met him, from an institution he has no way to enter. There is no room to walk into. There is no person to talk to. There is no desk, no form, no signature, no moment where a human being looks at him and makes a judgment that includes the pay stub and the diploma and the two years of showing up on time to bag groceries.

The judgment was made by a model trained on data he did not produce. The data includes demographic patterns, geographic patterns, payment histories from people who share some statistical resemblance to Daniel but who are not Daniel. The model does not know that Daniel has never missed a shift. It does not know about the folder in the kitchen drawer. It knows the shape of a statistical category and has placed Daniel inside it.

What the Counter Provided

#

At the bank, Robert listened for forty minutes and said yes. His judgment was imperfect. It was biased by familiarity, by shared geography, by the comfort of recognizing the couple in front of him as people like the people he knew. The algorithm that replaced Robert’s judgment is more consistent, more fair across populations, less susceptible to the warmth of a shared story.

But Robert’s judgment had a property the algorithm does not: it was a judgment made by a person who could be questioned. Margaret and Harold sat in a room with Robert. They could see his face. They could explain their lives. The judgment, whatever its biases, was an encounter between parties. If Robert had said no, they could have asked why, and Robert would have had to answer, and the answer would have been in a room, between people, subject to the social pressure of accountability that exists when you deny someone something to their face.

Daniel cannot ask why. The model’s logic is proprietary. The landlord received a score, not an explanation. The score was generated by a process Daniel cannot examine, based on data he cannot access, using methods he cannot understand. The denial arrived as a fact, not as a conversation. There was no room for the conversation to happen in.

The invisible institution is the logical endpoint of the transformation this series has been tracing. At the pharmacy: the encounter thinned. At the bank: the judgment moved to an app. At the DMV: the trip became optional. Here: the institution has fully dematerialized. The building is gone. The counter is gone. The waiting room is gone. What remains is the power, which is as real as it ever was, operating through a channel that provides no surface for the citizen to touch.

The Folder

#

Maria’s response to the denial was the response of someone who has navigated institutions all her life by assembling proofs. She went to the landlord’s office, which at least has a physical location, and brought the folder. She showed the documents. Birth certificate. Social security card. Diploma. Pay stubs. The landlord was sympathetic and unable to help. The screening company makes the determination. The landlord accepts or does not accept the application based on the score. The landlord does not make the score. The landlord does not know how the score is made. The landlord is, in this transaction, another recipient of the institution’s output, not a participant in its reasoning.

Maria has a notebook. The spiral notebook she carries to every official appointment, the one where she writes down everything anyone tells her, with the date and the name of the person who said it. She wrote down what the landlord said. She wrote down his name. The notebook has been right twice in situations where the system said she was wrong. This time the notebook has nothing to push against. There is no person who made the decision. There is no office that houses the logic. There is no mechanism by which Maria’s documentation of what she was told can be applied against the decision, because the decision was not made by anyone she can name.

I wonder whether the right to understand a decision made about your life by an automated system is a question of consumer protection or a question of dignity, and whether the distinction matters practically or only philosophically.

The Other Apartment

#

Daniel finds another apartment. Smaller. Farther from the bus line. The landlord at this building does not use the screening service. He is older, he owns the building, he meets tenants himself. He looked at Daniel and asked him three questions and said yes. The process took fifteen minutes in a small office with a metal desk and a calendar on the wall from a roofing company.

The yes was Robert’s yes. A person making a judgment in a room. It was imperfect, biased, based on fifteen minutes of impression rather than longitudinal data. It was also a yes that Daniel could see arriving, could participate in, could feel as an encounter between two people rather than a verdict from a system.

Daniel signs the lease. Maria helps him move in on a Saturday. The apartment is small and the paint is old and the bus line is a twenty-minute walk instead of a five-minute walk, and Daniel does not mind because the apartment is his, and the his is what matters, and the how of the his, the fifteen minutes with the older landlord, the three questions, the yes, is a thing Daniel will remember because it happened between people in a room with a metal desk.

The folder is in the kitchen drawer. Maria does not know what to do with it. She assembled it over eighteen years, document by document, proof by proof. She thought it would be enough. In a room, with a person, it was enough. In the system that has no room, the proofs had no surface to land on.

She leaves the folder in the drawer. She will start a new one for the next thing. She has been starting folders all her life.

References

#

O’Neil, Cathy. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown, 2016.

Eubanks, Virginia. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin’s Press, 2018.

Pasquale, Frank. The Black Box Society: The Secret Algorithms That Control Money and Information. Harvard University Press, 2015.

Herd, Pamela, and Donald P. Moynihan. Administrative Burden: Policymaking by Other Means. Russell Sage Foundation, 2019.

Noble, Safiya Umoja. Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press, 2018.