Taphonomic Bias

On why the fossil record is not a sample of life, why a training corpus is not a sample of what was said, and what we are deleting

#

Ernst Stromer went to the Bahariya Oasis in the Egyptian Western Desert between 1911 and 1914 and came back with the bones of animals nobody had seen. A sauropod. Two very large predators. And, sent to him from the field in 1912 by his collector Richard Markgraf, a set of dorsal vertebrae with spines nearly two metres tall, along with jaw fragments and teeth. He described it in 1915 and called it Spinosaurus aegyptiacus.

The specimens went on display in the Bavarian State Collection in Munich. Stromer was an aristocrat, a baron, and an outspoken opponent of the Nazi party, which cost him his post at sixty-five and made him a difficult man for the collection’s management to accommodate. As the war went on he petitioned repeatedly to have the fossils moved somewhere safer. The man in charge, a young party paleontologist named Karl Beurlen, declined, on the position that the Luftwaffe would keep Munich’s skies clear.

Some of Stromer’s colleagues took matters into their own hands and carried smaller specimens out of the building in suitcases. Vertebrae two metres tall do not go in a suitcase.

On the night of the twenty-fourth of April, 1944, the RAF bombed Munich. The Academy building took a direct hit and the collection burned. The Spinosaurus holotype went, and with it Aegyptosaurus, and the Carcharodontosaurus and Bahariasaurus material, all of them the only examples in the world.

Two of Stromer’s three sons were killed on the Eastern Front, one in 1941 and one in 1944. The third was taken prisoner and held in the Soviet Union until 1951. Stromer died the following year.

What a type specimen is for

#

A holotype is not the best example of a species or the most complete one. It is the reference. It is the single physical object that the name is anchored to, so that when two scientists disagree about what belongs inside a category, there is a thing in a drawer they can both go and look at. Every subsequent claim about the animal is measured against it.

Which is why its loss is not the loss of one fossil. Eighty years on, work on Spinosaurus is still obstructed by the 1944 raid. When new material surfaced in Morocco and was described in 2014, it did not settle the arguments. It reopened them, because there was no way to establish how the new bones related to the ones the name was attached to. What survives of the holotype is Stromer’s published description, his illustrations, and a photograph of it mounted in the gallery.

A reference is not a copy of the thing. It is the thing itself, kept so that future arguments have somewhere to land.

Taphonomy, running forward

#

Paleontology has a word for the reason the fossil record is not a sample of past life.

Taphonomy is the study of what happens to an organism between death and discovery: what decays, what scatters, what a river carries away, what gets buried fast enough to preserve. The record over-represents hard parts, animals that died near water, animals in subsiding basins, large animals, marine animals. It under-represents everything soft, everything upland, everything small, everything that lived where nothing was accumulating. No paleontologist mistakes the record for the world, and correcting for the bias is most of the discipline’s methodological work.

A training corpus has a taphonomy too, and unlike the geological one, it runs forward.

Each generation of models is trained on material that includes what previous generations produced. Not all of it. A vanishing fraction of it. What enters the record is what somebody kept: published, posted, adopted into a document, quoted, ranked highly, copied into a codebase, accepted into a draft and sent. The rest evaporates in the session it was made in and leaves nothing.

Adoption is the taphonomic event. What survives into the next corpus is not what the systems produced but what people were willing to keep.

That is a filter with a shape, and the shape is knowable. It favours what read well enough to be accepted. It favours what fit an existing document. It favours the register that gets ranked and the register that gets shared. It disfavours everything that was generated, glanced at, and closed.

Drift with a direction

#

Which makes model drift a different kind of object than it is usually treated as.

The comfortable version is a random walk. The models change a bit each generation, in no particular direction, and the changes average out over enough of them, and the whole thing is noise around a stable centre. Random walks are reassuring because they do not accumulate.

Taphonomic bias is not noise. It is a systematic pressure with a direction, applied at every generation, and it compounds. If each pass keeps disproportionately what was accepted, and what was accepted is disproportionately what already sounded right, then each generation is trained on a slightly narrowed version of the previous one’s output, and the narrowing is in a consistent direction.

Nothing about that requires anyone to intend it. It follows from the fact that a corpus is made of kept things and keeping is a choice made under conditions.

The compounding is the part worth being precise about, because a single generation of it is negligible and that is what makes it easy to wave off. Suppose each pass narrows the distribution by an amount too small to detect in any individual comparison. Ten passes later the cumulative narrowing is large and there is no single transition anyone can point at, because there wasn’t one. Every step was within noise. Only the sum is not.

This is the standard shape of a compounding bias and it is the reason disciplines that face one build instruments rather than relying on judgement. Nobody can eyeball a two percent annual drift. Everybody can read a gauge.

A change too small to notice at any single step and too large to reverse after enough of them is not a subtle problem. It is the ordinary way large changes actually happen.

There is a nearby argument that this one is not making. The question of whether the thing that hears you is continuous with the thing that answers your grandchildren is a question about counterparty, and it belongs elsewhere. This is a question about measurement.

The instrument problem

#

To measure a drift you need a fixed point, and there is exactly one class of fixed point available.

A set of weights that will never change again is the only reference specimen this field has. A description of a model will not do. The weights are the model, in the sense that matters: the actual object, capable of being queried indefinitely, producing the same output distribution in 2040 that it produced in 2026. Everything else, the benchmark scores, the eval suites, the published samples, the papers, is Stromer’s illustrations.

Deprecated weights are the calibration equipment. They are being deleted.

The deletion is not malicious and it is barely a decision. Storage costs something, serving costs more, old checkpoints accumulate, legal exposure attaches to models with known behaviours, and nobody’s quarterly objectives include maintaining a reference collection for a measurement problem that will become urgent in fifteen years. Paleontology keeps type specimens in climate-controlled drawers with catalogue numbers, permanently, under institutional obligation, precisely because the reference has to outlive every argument it will be used to settle. Software has no equivalent institution and no equivalent instinct. Deprecation is a line item.

A discipline that deletes its reference specimens can observe that things have changed and can never say by how much, or in which direction, or whether the change is still going on.

We wonder whether anyone will be able to tell, twenty years from now, that a narrowing happened at all, or whether the only available evidence will be a widespread sense that older writing reads strangely, which is exactly what every generation has always said about older writing.

What the drawings cannot do

#

Stromer’s records survived. They are excellent. They are not the specimen, and the reason is not the obvious one about resolution.

Stromer measured what 1915 knew to measure. Lengths, proportions, the articulation of the vertebrae, the shape of the jaw. He was meticulous and he recorded what a paleontologist of his moment understood to be the data. He did not section the bone and examine its density, because the question that made bone density interesting had not been asked yet. Ninety-nine years later, dense bone tissue associated with reduced buoyancy was central to the argument that Spinosaurus spent its life in water. That measurement could have been taken from the holotype. There is no way to take it now, and there never will be.

Anyone archiving anything is in Stromer’s position. We preserve what we currently know to measure. The eval suites being saved today capture the properties that seemed important in the year they were designed, and the drift question that matters in 2045 will require a measurement nobody is currently taking, on an object nobody currently thinks needs keeping.

The only defence against that is keeping the object rather than the measurements, and the object is a few hundred gigabytes, and it is being deleted to save the few hundred gigabytes.

What would answer this is not complicated and does not require anyone to agree about anything contested. A designated set of weights per generation, frozen, catalogued, never served to the public, never updated, held under an obligation that outlives the organization that made them, queryable by anyone with a measurement to run. Herbaria do this. Seed banks do this. Standards bodies keep physical artefacts in vaults for exactly this reason and have since the metre was a bar of metal in a basement outside Paris.

The reason it does not exist is not that anybody argued against it.

Wolfgang Stromer came home from the Soviet camp in 1951 and his father died in December of the following year. In 1995, more than forty years after that, Wolfgang gave the Munich museum a set of photographs of his father’s destroyed Spinosaurus.

They are photographs of a mounted skeleton in a gallery that no longer exists, of an animal nobody has seen since, taken by a man who could not have known what anyone would later want to see in them. It is the most that could be salvaged and it is a fraction of what was in the room. Everyone involved understood exactly what was at risk, said so, was overruled, and did what they could with the suitcases they had.

The title is the technical term for the reason a fossil record over-represents some things and loses others, and the essay defines it in full before using it. This essay is the deepest single clock in the ensemble described in The Many Clocks: depreciation, examined from inside. Its mechanism is the forward taphonomy of training corpora, and its finding is an instrument problem rather than a prediction. The question of whether a succeeding model is continuous with its predecessor as a counterparty, rather than as a reference, is the question of The Two Tempos. It stands in conversation with The Distillation Problem and The Monoculture.