The Intent

TAM-INS.03 · The Insufficient · The Approximate Mind

In 2003, a grant review panel at a major international health agency evaluated seventeen proposals for AI-assisted diagnostic screening. The panel had a budget. The budget had priorities. The priorities had been set by a strategic plan. The strategic plan had been shaped by the agency’s donors, its board, its institutional memory, and its theory of change, which was, as most theories of change in global health are, oriented toward interventions that could be delivered at scale and measured within a funding cycle.

One of the panelists, a woman named Dr. Anjali Rao, who had spent fourteen years running community health programs in Bihar, asked a question during deliberation. She asked whether any of the proposals studied the interaction effects between the conditions being screened for and the conditions of the lives in which the screening would occur. Whether any of them looked at what happens when a tuberculosis screening tool is deployed in a community where the barriers to treatment completion, transportation, stigma, wage loss during treatment, family disruption, are not incidental obstacles but structural features of the disease ecology.

The panel chair thanked her for the question. The panel funded three proposals. None of them studied interaction effects. The interaction effects were not in the request for proposals. They were not in the request for proposals because they were not in the strategic plan. They were not in the strategic plan because they could not be measured within a funding cycle. They could not be measured within a funding cycle because the cycles are three to five years long, and the interaction effects between screening, treatment barriers, household economics, and disease outcomes unfold over decades.

Dr. Rao went back to Bihar. The three funded proposals produced three AI screening tools. The tools worked. Sensitivity and specificity were good. They were deployed in settings similar to Bihar. Screening rates went up. Treatment completion rates did not.

Nobody studied why.

Upstream of Everything

#

The previous essays in this series introduced a skeptic architecture and populated it with seven philosophical operations. Each operation catches a different kind of insufficiency in a specification. Each one extends the frame, questions the categories, tests the assumptions.

But none of them asks the most dangerous question.

Who wrote the specification? Who funded the research that produced the evidence the specification is built on? Who decided what questions were worth asking, in which populations, using which methods, over which time horizons? Who benefits from the answers the system is designed to produce, and who bears the cost of the answers it is not designed to find?

The skeptic questions the categories. The traditions identify the type of insufficiency. This essay asks who put the categories there. Not in the conspiratorial sense. In the structural sense. The categories in any AI system are the endpoint of a chain of decisions that stretches back years, sometimes decades, through funding priorities, research agendas, institutional incentives, political pressures, and the ordinary administrative logic of organizations that must justify their budgets.

The bias is not in the algorithm. It is in the genealogy of the evidence the algorithm was trained on.

The Genealogy

#

Consider the evidence base for any AI diagnostic system deployed in a low-resource setting. The clinical guidelines it was trained on were produced by studies. The studies were funded by grants. The grants were awarded by agencies. The agencies had strategies. The strategies were shaped by theories of change. The theories of change reflected the intellectual commitments, the institutional incentives, and the measurability requirements of the people and organizations that developed them.

At each step, decisions were made about what to study. Each decision was reasonable in isolation. Each one narrowed what would eventually be known.

The randomized controlled trial was chosen as the gold standard because it isolates causal mechanisms. That choice was not neutral. It encoded an ontological commitment: that causes are isolable, that context can be controlled for, that what matters can be measured within the trial’s time horizon and population. Conditions whose causes are interactive, context-dependent, and slow to manifest, which is to say most conditions affecting the populations that most need the system’s help, are structurally disadvantaged by this choice. Not because the RCT is wrong. Because it is insufficient for the causal architecture of the phenomena it is being applied to.

The populations chosen for study were chosen because they were accessible, because institutional relationships existed, because the logistics of enrollment were manageable. Populations that were geographically remote, politically unstable, institutionally disconnected, or simply too expensive to reach were not studied. Not because their health conditions were less important. Because the research infrastructure could not reach them.

The outcomes measured were the outcomes the funders wanted to see. Mortality reduction. Disease incidence. Screening rates. These are important outcomes. They are also the outcomes most amenable to short-term measurement and most legible to the political constituencies that sustain the funding. Outcomes that matter enormously to the people living the conditions, the quality of a life lived with chronic pain, the social consequences of a stigmatized diagnosis, the household economic cascade triggered by a breadwinner’s illness, were not measured because they could not be measured cheaply, quickly, or in ways that produced the kind of clean numbers a strategic plan requires.

Each of these decisions was made by people doing their best within institutional constraints they did not create. The genealogy is not a conspiracy. It is the accumulated consequence of reasonable decisions made within structures that reward certain kinds of knowledge production and ignore others.

What the Green Revolution Was For

#

The earlier essays in the series used the Green Revolution as a case study in optimization failure. Yield per hectare was maximized. Soil health, groundwater, farmer autonomy, dietary diversity, and the social fabric of rural communities were not measured and were devastated.

But the previous treatment was incomplete. It described what the optimization missed. It did not ask who was doing the optimizing and why.

The Rockefeller Foundation and the Ford Foundation funded the foundational research. USAID provided the policy infrastructure for adoption. Seed companies and fertilizer manufacturers provided the commercial incentive for scale. The World Bank provided loans to governments that adopted the package. Each institution had its own reasons for wanting yield to increase. Those reasons were not identical to the reasons of the farmers whose lives would change.

The question “how do we maximize crop yield?” was not asked by the woman in Vidarbha growing cotton alongside tur dal alongside vegetables. It was asked by institutions whose theories of development, whose metrics of success, whose political and economic interests aligned with the answer. The farmer’s question, if anyone had asked her, might have been: “How do I survive the bad year?” The optimization that followed was an answer to someone else’s question applied to her land.

The objective function was not incomplete by accident. It was incomplete because the people who set it were optimizing for their goals, not hers. This is not malice. It is the structure of institutional knowledge production. And it is the most consequential form of bias in any system, because it determines what the system is for before the first variable is selected.

The Administrative Architecture of Ignorance

#

Robert Proctor coined the term “agnotology” for the study of culturally produced ignorance. He was writing about the tobacco industry’s deliberate manufacture of doubt. But the concept extends beyond deliberate manufacture to structural production.

There is a form of ignorance that is produced not by suppression but by administration. The grant structure that requires isolable outcomes produces ignorance about compound causation. The funding cycle that demands results within five years produces ignorance about mechanisms that unfold over decades. The research infrastructure that rewards depth within established fields produces ignorance about phenomena that cross field boundaries.

Nobody decided to be ignorant about the interaction effects Dr. Rao asked about. The ignorance was produced by the ordinary operation of institutions doing what institutions do: managing budgets, satisfying stakeholders, measuring what they can measure, and reporting results in formats their governance structures can absorb.

The Intersectional Systemic Harm Index was built to measure compound effects precisely because the existing infrastructure had produced systematic ignorance about them. Barriers were treated as individual problems not because anyone believed they were individual but because the policy architecture that funded interventions required isolable outcomes with measurable attribution.

“This intervention reduced transportation barriers by X percent.” That is a fundable finding. “This person’s transportation barrier interacts with her digital divide, her economic strain, her social isolation, and her language barrier in ways that produce an outcome none of them would produce alone, and the interaction is the mechanism, not the individual components.” That is not fundable. Not because it is not true. Because the funding architecture cannot process it.

The intent is not malicious. It is administrative. And it is no less consequential for being boring.

Self-Healing and Its Limits

#

The previous essays described an adversarial architecture: the skeptic questions the categories, the traditions identify the type of insufficiency, and the entire stack is structurally independent from the optimizer. This essay adds the question that makes the architecture complete, or reveals that it cannot be completed.

Can a system detect when its own framing has been captured by the interests it was supposed to interrogate?

The honest answer is: not from inside. The auditor who works for the firm does not provide independent auditing. The pharmaceutical company’s internal ethics review does not protect the populations the company’s incentive structure does not prioritize. The epistemic interrogator embedded within the institution learns to interrogate in ways the institution can absorb.

This is not a speculative risk. It is the documented history of every adversarial function ever embedded within the institution it was designed to challenge. Regulatory capture is not an anomaly. It is the equilibrium. The adversarial function begins independent. It is funded by the institution. It develops relationships with the institution’s personnel. It learns which questions are welcome and which produce friction. Over time, and the time is usually short, the friction is smoothed. The questions narrow to the ones the institution can absorb. The function becomes a compliance exercise rather than a genuine challenge.

I wonder whether the only honest architecture is one in which the components are designed to be replaced periodically, so that no adversarial function serves long enough to be captured, and whether that architectural honesty would be tolerable to any institution that has to live with the disruption of perpetual challenge.

What cannot be replaced is the human check. The affected populations, the people whose lives are being processed by the systems the architecture interrogates, are the ground truth the system cannot generate for itself. Dr. Rao knew what the panel was missing because she had spent fourteen years in Bihar. Her knowledge was not theoretical. It was accumulated through contact with the lives the system was designed to serve.

The self-healing mechanism is not a feature of any single system. It is an emergent property of adversarial architecture combined with genuine participation by the people the system affects. The architecture can be built. The participation requires something no architecture can guarantee: the political will to include voices that have no institutional power and whose testimony is inconvenient.

The Panel, Twenty Years Later

#

Dr. Rao retired from field work three years ago. She consults now, occasionally, for organizations that ask her to review AI deployment plans for rural health systems. She notices that the plans have become more sophisticated. The models are better. The screening tools are more accurate. The deployment logistics are more carefully planned.

She also notices that the question she asked in 2003 still does not appear in the plans. Not because anyone has argued against it. Because the institutional architecture that produces the plans has not changed. The funding cycles are still three to five years. The outcomes are still required to be isolable and measurable. The evidence base is still built from studies that controlled for context rather than studying it.

She keeps a notebook. She writes down the questions that the plans are not asking. It is a different notebook from the one described in Part 74 of this series, but the practice is the same. She writes: “The screening tool has 94% sensitivity. Nobody is measuring what happens to the 94% after they are screened. Whether they complete treatment. Whether the treatment disrupts their household’s income. Whether the disruption produces secondary health consequences that the screening tool was not built to see.”

She has a photograph on her desk, not of a patient or a colleague. It is a photograph of a bridge in her home village in Tamil Nadu, a stone bridge over a seasonal river that her grandfather helped build in the 1940s. The bridge was built without engineering drawings. It was built by people who knew the river, who had watched it in flood and in drought, who understood its moods in a way that no specification could capture. It has stood for eighty years.

She thinks sometimes about the difference between building something that works and building something that looks like it should work. The screening tool works, by every metric the system uses to evaluate it. The bridge works, by the only metric the river recognizes: it is still standing.

The question she asked in 2003 was not about the screening tool’s accuracy. It was about whether the system that produced the tool had the capacity to see what the tool would encounter in the field. Whether the intent behind the tool’s design, the research agenda, the funding priorities, the measurability requirements, had produced an instrument calibrated to the institution’s needs rather than the community’s reality.

She does not think the people who designed the tool were wrong. She thinks they were constrained. The constraints produced the tool they produced, and the tool does what it was designed to do. What it was not designed to do is the part that matters to the people in Bihar.

The bridge is still standing. The screening tools are being updated. The question is still not being asked.

This is the third essay in The Insufficient, a four-essay sub-series of The Approximate Mind. The first essay introduced the skeptic architecture. The second populated it with seven philosophical operations. This essay moves upstream from the specification to the commissioning decision, arguing that the most consequential bias in any AI system is not in the training data or the objective function but in the genealogy of decisions that determined what would be studied, funded, and known. The fourth essay, “The Retroduction,” provides the method for working backward from outcomes to the mechanisms that the insufficient empirical record has not captured.

References

#

Critical Realism and Social Science

Bhaskar, Roy. Scientific Realism and Human Emancipation. Verso, 1986.

Bhaskar, Roy. The Possibility of Naturalism: A Philosophical Critique of the Contemporary Human Sciences. Harvester Press, 1979.

Agnotology and the Production of Ignorance

Proctor, Robert N., and Londa Schiebinger, eds. Agnotology: The Making and Unmaking of Ignorance. Stanford University Press, 2008.

Proctor, Robert N. Golden Holocaust: Origins of the Cigarette Catastrophe and the Case for Abolition. University of California Press, 2011.

The Politics of Knowledge Production

Harding, Sandra. Objectivity and Diversity: Another Logic of Scientific Research. University of Chicago Press, 2015.

Foucault, Michel. The Archaeology of Knowledge. Pantheon Books, 1972.

Bowker, Geoffrey C. Memory Practices in the Sciences. MIT Press, 2005.

The Green Revolution and Development

Shiva, Vandana. The Violence of the Green Revolution: Third World Agriculture, Ecology, and Politics. Zed Books, 1991.

Scott, James C. Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed. Yale University Press, 1998.

Optimization and Institutional Incentives

Oreskes, Naomi, and Erik M. Conway. Merchants of Doubt: How a Handful of Scientists Obscured the Truth on Issues from Tobacco Smoke to Global Warming. Bloomsbury Press, 2010.

Winner, Langdon. “Do Artifacts Have Politics?” Daedalus, vol. 109, no. 1, 1980, pp. 121-136.

Muller, Jerry Z. The Tyranny of Metrics. Princeton University Press, 2018.

Global Health and Equity

Farmer, Paul. Pathologies of Power: Health, Human Rights, and the New War on the Poor. University of California Press, 2003.

Chambers, Robert. Whose Reality Counts? Putting the First Last. Intermediate Technology Publications, 1997.

Sen, Amartya. Development as Freedom. Anchor Books, 1999.