The Record — Summary

People keep calling the training data a library. It is not. A library is curated. Somebody decided what earned a place on the shelf. The training data had no editor. It is closer to a midden heap, the thing archaeologists dig through to learn what a people were like. Not the monument. The trash.

That is what the model learned from. Us, unedited.

Every argument for justice and every justification for atrocity, every love poem and every piece of propaganda, in the same heap, weighted by how often we said them rather than by whether they were true.

We have a story about ourselves. It lives in the commencement speeches and the eulogies. In the story we are reasonable, we mean well, we are improving. The record contains the story and everything the story leaves out. It does not contradict the story. It refuses to drop the rest.

Nobody built this on purpose. It happened as a byproduct of building a useful tool.

When the model produces something ugly, the instinct is to say it is broken. The model is not broken. The model is reading the heap. The bias was in the record because the bias was in the behavior.

The model can be made to behave better than the heap, and people are doing that work. It is the same work the species has always been doing on itself. Raising the child to be gentler than the deposit they were born into.

I wonder whether the discomfort people feel looking at what the model produces is really discomfort with the model, or the older discomfort of meeting the record without the story wrapped around it first.

The heap was always there. We never had a reader patient enough to take in the whole thing at once.