The Record

TAM-PER.06 · The Persistent · The Approximate Mind

People keep calling the training data a library.

It is not a library. A library is curated. Somebody decided what got a place on the shelf, what was worth keeping, what represented us at our best. The training data had no such editor. It is closer to a midden heap, the thing archaeologists actually dig through to learn what a people were like. Not the monument they built to be remembered by. The trash. The discarded. The unguarded record of what they did when no one was preserving it for posterity.

That is what the model learned from. Us. Unedited.

Every argument for justice and every justification for atrocity. Every breakthrough and every conspiracy. Every love poem and every piece of propaganda, sitting in the same heap, weighted by how often we said them, not by whether they were true. The model does not pick the noble texts over the ugly ones. It cannot tell which is which. It ingests the whole record and the record is us, as we actually are, not as we describe ourselves at graduations.

We have a story about ourselves. The story is in the commencement speeches, the founding documents, the eulogies. In the story, we are reasonable, we mean well, we are improving. Then there is the record. The record contains the story and also everything the story leaves out. The cruelty we performed while believing we were reasonable. The harm we did while meaning well. The record does not contradict the story. It just refuses to drop the rest.

And the model read all of it.

For the first time, the species built a complete enough record of its own behavior that something could read the whole thing and learn the patterns. Not the patterns we describe. The patterns we actually run. The model knows what we do when we are angry, frightened, certain, in love, lying. It knows because we wrote it all down, billions of times, never thinking the writing-down was the point.

Nobody built this on purpose. No anthropologist set out to assemble the most complete record of human behavior ever compiled. It happened as a byproduct. We were building a useful tool. To build it we fed it everything we had ever written, and everything we had ever written turned out to be a mirror, and the mirror turned out to be accurate in a way our story about ourselves is not.

People want the model to be better than the record. They want it to give them the noble version, the reasonable version, the species as the story describes it. And when it produces something ugly, something biased, something cruel, the instinct is to say the model is broken.

The model is not broken. The model is reading the heap.

What it produces when it goes wrong is not alien to us. It is us, reflected back without the editing we usually apply before we look. The bias was in the record because the bias was in the behavior. The cruelty surfaces because the cruelty is in the deposit, in the volume, in how often we actually said the thing we claim we do not believe.

The model can be made to behave better than the heap. We can do that, and people are. But the heap is the honest baseline, and the work of making the model kinder than its training data is, looked at directly, the same work the species has always been doing on itself. Trying to act better than the record of what we have done. Building the institution, writing the rule, raising the child to be gentler than the deposit they were born into.

The model just makes the gap visible. It shows us, in one artifact, the distance between the story and the record. We have always lived in that distance. We just did not usually have to look at it laid out in full.

I wonder whether the discomfort people feel looking at what the model produces is really discomfort with the model, or whether it is the older discomfort of meeting the record without the story wrapped around it first.

The heap was always there. We just never had a reader patient enough to take in the whole thing at once.

References

#

Bowker, Geoffrey C., and Susan Leigh Star. Sorting Things Out: Classification and Its Consequences. MIT Press, 1999.

Noble, Safiya Umoja. Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press, 2018.

Bender, Emily M., et al. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021, pp. 610–623.

Borges, Jorge Luis. “The Library of Babel.” Ficciones. Translated by Anthony Kerrigan, Grove Press, 1962.

Trouillot, Michel-Rolph. Silencing the Past: Power and the Production of History. Beacon Press, 1995.

This is the sixth essay in The Persistent, a series by Yagn Adusumilli examining what AI cannot change about the species that built it. PER-07, The Mirror, follows.