The Intent — Summary
In 2003, a grant review panel at a major international health agency evaluated proposals for AI-assisted diagnostic screening. Dr. Anjali Rao, who had spent fourteen years running community health programs in Bihar, asked whether any proposal studied the interaction effects between the conditions being screened for and the conditions of the lives in which screening would occur. The panel chair thanked her. Three proposals were funded. None studied interaction effects. The interaction effects were not in the request for proposals because they were not in the strategic plan because they could not be measured within a funding cycle.
The skeptic questions the categories. The traditions identify the type of insufficiency. Neither asks the most dangerous question: who wrote the specification? Who funded the research that produced the evidence? Who decided what was worth studying, in which populations, using which methods? The bias is not in the algorithm. It is in the genealogy of the evidence the algorithm was trained on.
The triage system’s training data was produced by fifty years of decisions about which diseases to study, in which populations, funded by which institutions with which priorities. At each step, decisions narrowed what would eventually be known. The RCT was chosen as the gold standard because it isolates causal mechanisms, encoding the ontological commitment that causes are isolable and context can be controlled for. Conditions whose causes are interactive and slow to manifest are structurally disadvantaged by this choice. Populations that were geographically remote or institutionally disconnected were not studied. Outcomes that mattered to the people living the conditions, quality of life with chronic pain, household economic cascade from illness, were not measured because they could not be measured cheaply or in clean numbers.
Each decision was made by people doing their best within institutional constraints they did not create. The genealogy is not a conspiracy. It is the accumulated consequence of reasonable decisions made within structures that reward certain kinds of knowledge production and ignore others. The intent is not malicious. It is administrative. And it is no less consequential for being boring.
The self-healing question: can a system detect when its own framing has been captured by the interests it was supposed to interrogate? Not from inside. The auditor co-opted by the firm does not know it has been co-opted. Regulatory capture is not an anomaly. It is the equilibrium. Self-healing requires structural adversarialism: multiple systems interrogating each other’s intent, funded separately, governed separately, with affected populations as the irreplaceable check the architecture cannot generate for itself.
Dr. Rao retired three years ago. She consults occasionally. She notices that the deployment plans have become more sophisticated. She also notices that her question from 2003 still does not appear. Not because anyone argued against it. Because the institutional architecture that produces the plans has not changed. She keeps a photograph of a stone bridge her grandfather helped build in Tamil Nadu in the 1940s, built without engineering drawings by people who knew the river. It has stood for eighty years. She thinks about the difference between building something that works and building something that looks like it should work.