If you work in American political history, you probably have a good intuition about which presidents were combative, which were optimistic, and when American political rhetoric started to feel different. Using an emotion classification Large Language Model it is possible to check these intuitions in a systematic way against 3.9 million sentences of presidential speeches from the American Presidency Project (APP) archive. The results are often surprising, offering a quantitative window into the structural architecture of persuasion and the measurable drift of political culture.

The Method:

The technology underlying this analysis is DistilRoBERTa, a transformer-based language model. The model was trained on a massive corpus of human-annotated data. Thousands of sentences were labeled by human readers who reached a consensus on whether a line expressed anger, disgust, fear, joy, sadness, surprise, or remained emotionally neutral. Once trained, the model ‘reads’ one sentence at a time, and assigns a probability that the sentence belongs to one of these categories.

The key analytical unit is the sentence rather than the document. This granularity is what gives the dataset its flexibility. It’s the difference between knowing a speech is “angry” and knowing precisely where in the speech the predominantly ‘angry’ sentences concentrate, and how their concentration compares to different historical base-lines.

Subscribe now

The Macro Trend:

At the macro level, perhaps the most revealing variable is the proportion of sentences the model classifies as Neutral. It is useful to think of “Neutral”, as more than just a proxy for the levels of emotion generally, but also as a proxy for what we might term the ‘procedural tone of governance.’ It represents factual reporting, policy detail, and institutional communication.

When we track this across 112 years, we see a measurable shift from the President as a “Chief Executive” (procedural) to the President as a “Communicator-in-Chief” (affective), as the media environment changed, and the expections surrounding presidential communication changed with it.

The data reveals distinct plateaus:

1. The WWI-era Volatility (approx. 1913–1921): We see intense emotionality here, with neutrality dipping below 50% in 1918. However, we must be cautious: the volume of content in the APP database for this era is extremely sparse compared to more recent decades. It is also worth noting that the apparent “decline” in volume in the most recent years of the dataset is not a sign of quieter presidents, but of a shift towards social media platforms, which are not as well captured in the APP data.

2. The Mid-Century Consensus (late 1960s–1990s): Following the war, and the emotional Kennedy era, rhetoric settled into a high-neutrality plateau. For several decades, presidential speech remained largely deliberative and procedural, averaging around 60–65% neutrality.

3. The Polarized Drift (1998–Present): Beginning with the Clinton impeachment and accelerated by 9/11, we see a sustained decline in neutrality that has never reversed. By the 2020s, we reached historical highs of emotionality. 2024 and 2025 represent the angriest years in the entire 112-year dataset, with anger scores doubling the long-run average.

This 25-year trend in US presidential speech is not an isolated phenomenon. It mirrors findings from other researchers, such as Rozado, Hughes, and Halberstadt (2022), who used the same model and found a similar longitudinal increase in emotionality and negativity in news headlines. The presidency data therefore offers an interesting corroboration to this broader picture of a polarizing information ecosystem.

Emotional Fingerprints:

Aggregating data by administration produces what I call “emotional fingerprints,” distinct profiles that capture rhetorical personality and governing context.

The “Joy” category is particularly revealing. To an emotion classifier, “Joy” encompasses national pride, optimism, and reassurance. Presidential rhetoric is fundamentally aspirational; even in times of crisis, the president is expected to provide a celebratory vision of national resilience.

Republicans consistently employ more joyful language (15.3%) than Democrats (13.4%). Republican rhetoric has traditionally leaned into national optimism. Conversely, Democrats register higher levels of angry language (3.2% vs 2.6%). This is suggestive perhaps of a rhetorical style that more often frames policy problems as injustices demanding confrontation and reform.

The data makes familiar reputations concrete, and is suprising in some cases. The data seems to support the “No Drama Obama” moniker, as the most neutral modern president with the lowest anger scores in the recent era. G. W. Bush’s bouncy, theological optimism shines through, with the highest joy levels of any modern president (20.4%), a consistent strategy of optimism even as fear spiked measurably after 9/11. To my suprise, despite Biden’s message of unity, his rhetoric registers the highest anger scores of any modern Democrat, a reflection perhaps of governing under conditions of extreme polarization rather than his personal style.

The Power of Abstraction

Using AI models to classify discourse at scale in this way is no replacement for close reading, nor should we get too bogged down in the reductionism of transforming sentences into scores. What this kind of data does facilitate is a remarkable range of perspectives, from the sentence level, to macro-historical-emotional trends in presidential discourse. I believe the ability to traverse these different scales of analysis, and make valid, data-based comparisons between speeches, presidents, and periods, really augments our traditional close-reading. No human reader could detect structural patterns across 32,000 speeches consistently, or measure the long-run drift of political culture with any precision. This data offers us a quantitative window into the structural architecture of persuasion, the measurable drift of political culture toward emotionality, and the affective fingerprint of individual presidents’ rhetoric.

The technology underlying this analysis is, in one sense, unremarkable: a general-purpose language model fine-tuned on a diverse corpus that includes Twitter posts, Reddit discussions, student essays, and television dialogue. But it is precisely this generality that makes it powerful. The same transformer architecture that classifies emotion can be trained to tag parts of speech, identify named entities, detect policy frames, extract claims, or track the spread of metaphors. The versatility and adaptability of AI Large Language Models is precisely what makes them such a transformative technology, and they hold tremendous potential for historians and historical research.

Subscribe now

Are you working on computational history? Do you want to share it? Please reach out.

Share

Under the Hood

What makes this project possible is the Huggingface repository that contains LLM models for anyone to use. Want to try it out? Here is code you can run on nearly any computer. If you need help getting it started, just copy the code below into Claude or ChatGPT and ask for help (you will need to install two packages called transformers and torch). A longer version that explains the code is below as well.

The code can be very easily adapted to run over an excel spreadsheet, a document collection, or nearly any other data source. Just ask an LLM to adapt it to that purpose. The LLM will even generate the code to make your own graphs!

The hardest part in computational history is knowing what is possible.

Code

# Created: 2026-05-16 11:02
# Purpose: Run local emotion classification using Hugging Face Transformers.

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="j-hartmann/emotion-english-distilroberta-base",
    top_k=None
)

texts = [
    "I am thrilled about this project.",
    "I feel anxious and exhausted.",
    "This makes me furious.",
    "The results were unexpected."
]

for text in texts:
    print("\nTEXT:", text)

    scores = classifier(text)[0]

    for item in sorted(scores, key=lambda x: x["score"], reverse=True):
        print(f"{item['label']:>10}: {item['score']:.3f}")

Here are what the results of this test code look like:

TEXT: I am thrilled about this project.
       joy: 0.977
  surprise: 0.014
   neutral: 0.004
     anger: 0.002
      fear: 0.001
   sadness: 0.001
   disgust: 0.001

TEXT: I feel anxious and exhausted.
      fear: 0.993
   sadness: 0.002
   neutral: 0.001
  surprise: 0.001
     anger: 0.001
       joy: 0.001
   disgust: 0.000

TEXT: This makes me furious.
     anger: 0.981
   neutral: 0.007
   disgust: 0.005
      fear: 0.003
   sadness: 0.002
  surprise: 0.002
       joy: 0.000

TEXT: The results were unexpected.
  surprise: 0.947
   neutral: 0.023
       joy: 0.017
     anger: 0.005
      fear: 0.004
   sadness: 0.003
   disgust: 0.002

Code with Explainer (for the novice):

# Created: 2026-05-16 11:30
# Purpose:
# Run a local emotion-classification model using Hugging Face Transformers.
#
# -------------------------------------------------------------------
# FIRST-TIME SETUP
# -------------------------------------------------------------------
#
# 1. Create a Python virtual environment (recommended):
#
#    python -m venv emotions
#
#
# 2. Activate the virtual environment:
#
#    Mac/Linux:
#    source emotions/bin/activate
#
#    Windows:
#    emotions\Scripts\activate
#
#
# 3. Install required libraries:
#
#    pip install torch transformers
#
#
# 4. Run the script:
#
#    python emotions_test.py
#
#
# -------------------------------------------------------------------
# WHAT THIS SCRIPT DOES
# -------------------------------------------------------------------
#
# This script:
#
# 1. Loads a pretrained emotion model from Hugging Face
# 2. Downloads the model automatically the first time it runs
# 3. Stores the model in a local Hugging Face cache
# 4. Runs emotion classification on example sentences
# 5. Prints emotion probabilities sorted from highest to lowest
#
#
# -------------------------------------------------------------------
# WHAT THE MODEL PREDICTS
# -------------------------------------------------------------------
#
# The model predicts emotions such as:
#
# - joy
# - sadness
# - anger
# - fear
# - surprise
# - disgust
# - neutral
#
#
# -------------------------------------------------------------------
# IMPORTANT NOTES
# -------------------------------------------------------------------
#
# First run:
# - downloads model files from the internet
# - may take 1–5 minutes depending on internet speed
#
# Later runs:
# - load the model from local cache
# - start much faster
#
# Hugging Face cache location on Mac/Linux:
#
#    ~/.cache/huggingface/
#
#
# -------------------------------------------------------------------
# IMPORTS
# -------------------------------------------------------------------


# Import the Hugging Face "pipeline" helper.
#
# A pipeline is a high-level wrapper that:
#
# - loads the tokenizer
# - loads the neural network model
# - prepares the text
# - runs inference
# - formats the output
#
# This makes it possible to run modern NLP models
# in only a few lines of code.
from transformers import pipeline


# -------------------------------------------------------------------
# LOAD MODEL
# -------------------------------------------------------------------


# Create a text-classification pipeline.
#
# "text-classification" tells Transformers
# what kind of NLP task we want to perform.
#
# model="..." specifies which pretrained model to use.
#
# top_k=None tells the model to return ALL emotion scores,
# not just the highest-scoring emotion.
#
# On the first run:
# - Hugging Face downloads model files
# - files are cached locally
#
# Later runs load directly from cache.
classifier = pipeline(
    "text-classification",
    model="j-hartmann/emotion-english-distilroberta-base",
    top_k=None
)


# -------------------------------------------------------------------
# INPUT TEXTS
# -------------------------------------------------------------------


# Example texts to classify.
#
# The model estimates the emotional tone of each sentence.
texts = [
    "I am thrilled about this project.",
    "I feel anxious and exhausted.",
    "This makes me furious.",
    "The results were unexpected."
]


# -------------------------------------------------------------------
# RUN CLASSIFICATION
# -------------------------------------------------------------------


# Loop through each sentence.
for text in texts:

    # Print the original input text.
    print("\nTEXT:", text)

    # Run emotion classification.
    #
    # The pipeline returns a list of results.
    #
    # Since we pass only one sentence at a time,
    # we take the first item using [0].
    scores = classifier(text)[0]

    # Sort emotion scores from highest to lowest.
    #
    # Each item looks like:
    #
    # {
    #     "label": "joy",
    #     "score": 0.98
    # }
    #
    # The lambda function tells Python
    # to sort using the "score" value.
    for item in sorted(scores, key=lambda x: x["score"], reverse=True):

        # Print:
        # - emotion label
        # - probability score rounded to 3 decimals
        #
        # Example:
        #
        #       joy: 0.982
        print(f"{item['label']:>10}: {item['score']:.3f}")

Share

In that respect, historians are no longer alone. Political scientists like Erica Chenoweth have assembled longue durée datasets of nonviolent movements in order to generalize about their effectiveness over time. (Her conclusion: nonviolence wins more often than violence, though with important exceptions.) Peter Turchin, working with historians of the ancient world, has built the Seshat: Global History Databank, comparing technologies, wars, empires, and religions across millennia to produce arguments about the origins of cities, kingship, and even monotheism. Nassim Nicholas Taleb, whom I dined with this week, has compiled his own longue durée dataset of wars and casualties in order to test claims about whether violence is declining over time.

These are not my methods. They may not be yours either. As a historian trained in social and cultural approaches, I want to know about individual lives, about moods and propaganda, about the lived experience and interpretation of war, not merely the number of dead or the count of technologies. There is, in these datasets, often too little of the texture that historians are trained to value.

And yet I admire these projects deeply. I admire their scale. I admire their willingness to enter public debate on questions that matter right now. Taleb began counting war dead in order to argue with Steven Pinker’s The Better Angels of Our Nature. That conversation has drawn in historians as well, my coauthor David Armitage among them, though from a different angle, asking how categories like “civil war” themselves have been historically constructed and deployed rather than simply counted.

What unites Chenoweth, Turchin, and Taleb is their determination to generalize about war and peace, violence and nonviolence, questions that have always been central to historical inquiry. They offer clear, even binary answers to contested claims. Is nonviolence increasing? Chenoweth says yes, particularly in the modern period. Are wars becoming less stochastic? Taleb says no. If this is where our collective understanding of society is being formed, historians cannot afford to be absent.

What, then, makes historians different?

It is not a lack of rigor. It is not an allergy to numbers. It is our omnivorousness. Other disciplines pride themselves on agility, on the mastery of mathematics or philosophy. (I think, for instance, of Tyler Cowen celebrating the reach of economics.) But no discipline has the breadth of history at its best: statistics and mathematics alongside philosophy, the cultural turn’s engagement with art and meaning, social history’s grounding in linguistics and lived experience.

In my own work on the history of political economy through text mining, I have tried to build on questions from political economy and social history alike, using a sensitivity to memory encoded in language that depends on both historical linguistics and corpus linguistics. If the last wave of digital humanities was built on NLP, the LLM promises something more ambitious: the possibility of merging datasets like wages (as Louis Hyman has been documenting) with the arguments, experiences, and nuances embedded in historical texts.

The challenge is how to do that without losing sensitivity to individual lives, how to move from large-scale datasets to arguments about trends like the rise of nonviolence without flattening culture, decision, and imagination into mere counts.

Historians will not become economic historians overnight. But many are already motivated by questions, about capitalism, conflict, governance, that implicitly place them in dialogue with Pinker, Chenoweth, and Turchin. The question is whether we can engage those datasets without surrendering what we do best.

One answer lies in what I have called Critical Search (developed in The Dangerous Art of Text Mining). In practice, Critical Search treats datasets not as endpoints but as indices of change. Peaks, anomalies, and concentrations become invitations to investigate. The historian identifies a pattern, then zooms in, modeling individuals or events through text mining, and finally returning to close reading at the moment when everything begins to look different.

A concrete example: in my work on environmental rhetoric in Congress, I found that members of Congress routinely referred to environmentalists as “zealots,” “academicians,” and “radicals.” Counting these phrases allowed me to answer Phillips’ question: how widespread were these denunciations? The answer was: not very. Roughly 90% of such attacks were produced by just six members of Congress, and overwhelmingly by one figure, Ted Stevens.

At that point, the project pivoted. The data identified Stevens not as a cherry-picked case, but as a statistically grounded exemplar. The next step was to zoom in: to read his speeches, trace his career, and understand his role as a defender of oil pipelines and a central voice opposing environmentalism from the 1970s through the 1990s and beyond. The result was a history that could sustain a general claim, environmentalism faced sustained attack in Congress, while grounding that claim in the detailed study of a particular actor and his evolving rhetoric.

This is how political and social history scale. Not by abandoning the case study, but by selecting it rigorously.

LLMs now make it possible to extend this method far beyond parliamentary debates. They allow historians to draw on longue durée datasets, wars, casualties, nonviolent movements, wages, and to model change over time in ways that engage directly with the Pinkers, Turchins, and Talebs of the world. But crucially, they also allow us to use those datasets to identify moments of exception: the unusually violent, the unexpectedly peaceful, the early adopters of nonviolence.

Those moments, in turn, become the basis for historical explanation. They let us test the limits of quantitative claims by showing what is hidden until we zoom in. They offer a way to tell large-scale stories without surrendering detail. And they provide a more rigorous alternative to cherry-picking, not by abandoning selection, but by disciplining it.

That, I think, is the historian’s reply to statistics: not resistance, but integration on our own terms.

Share

As I see this debate, this question of our age, there are two main questions that history can shed some light on.2

1. Is AI a complement or a substitute for labor? That is, will it increase demand for and the productivity of workers, or decrease it?

2. Will AI be controlled by the few or be accessible to the many?

A Complement or a Substitute?

Consider a some of the most important technologies of the past 200 years.

When I am asked about what automation might look like, I inevitably discuss agriculture. Roughly all of our ancestors were farmers and approximately none of us today are. Yet we still eat bread made from wheat. That shift is possible because of automation.34

The mechanical thresher, used to process wheat, was a substitute for the most backbreaking work of the harvest. But it also enabled more land to be cultivated, and that land was cultivated more efficiently, allowing for greater harvests. Mechanization of the farm, like the thresher, turned the American Midwest into the breadbasket of the world.

Those displaced farmers found work on railroads, moving all that. And those jobs, according to people at the time, were a kind of liberation from the raw animal labor of threshing. On net, it created demand for more workers at better wages in work more fit for people than beasts. For those that remained farmers, they found other higher-value work to be done. On a farm, there is always more work to do.

The failure, then and now, is to think farmers were only threshers. That was one part of their jobs. Today, our work, for most people, is also a bundle of tasks. Workers then and now could and can focus on parts of their job that are of higher value. And in a new economy, new tasks in new industries will be created. Many of the jobs that we do today (web designer, UI expert) were simply unimaginable in 1850. That is a good thing.

Consider now the assembly line. I’m sure you all know about the staggering increases in productivity that come from the division of labor. If you take my class in industrial history, you would learn deeply about the story of the automobile. With the assembly line, and no other change in technology, car assembly went from 12 and a half hours to about 30 minutes (once they worked out the kinks). Did this reduce the demand for workers? No. It reduced the price of cars. And that increased the demand for workers, who eventually could demand even higher wages through unionization.

It is important here to realize that better tools don’t make us get paid worse. They generally make us get paid more. Why? Because the tool, without the person, is useless. Even for today’s most cutting-edge AIs, that is true. It can code, but it can only code what I imagine it to code. It can draw, but only what I imagine it to draw. That is true for AIs as it was true for the thresher.

So, I would offer that AI will create more growth, more abundance. In the long run, all growth comes from higher productivity.

I would add one more piece to this story. Economic inequality has worsened since roughly 1970. It has worsened, therefore, not in the industrial era, but the digital era. I have argued elsewhere that this happened because for decades we did not use computers as tools of automation but as glorified typewriters (and then as televisions). Our productivity did not increase, especially to justify the expense of computers. Economists have debated for decades now over the lack of increase in productivity that came with the “digital age” of computing, but it is simple. We don’t use them as computers. Now we can.

For the first time now, normal people with their normal problems can use their computers to solve and automate their problems. AI can write code. AI can automate their tedium. The digital age did not bring any gains because it had no yet arrived. We were living through the last gasp of the industrial economy.

It is now here.

This technology will unleash unimaginable productivity gains. It will level the playing field between coders and the rest of us. Coders will lose their jobs, to be sure, but for the rest of us, the bundle of workplace tasks will become much better.

And truthfully, the demand for real computer scientists will probably increase in the era of vibe-coding. Computer science itself is a bundle of skills, of which coding is just one. The more important skill – software and data architecture – will only increase in demand as the usefulness of software expands.

Monopoly or Market?

Isn’t all this talk about AI well and good, but professor, won’t it be expensive? Won’t large corporations like OpenAI just monopolize it? This fear of a “monopoly“ is that it would become a price-gouger, a tax on the rest of the economy. I do not think this fear is reasonable.

The threats of monopoly are overstated, not simply because such monopolies in the last century never persisted for more than a few years (either because of regulation or competition), but because, on the ground now, AI doesn’t seem like it is heading in that direction.

More than people realize, the software is in a competitive market, and at the same time, the hardware is coming down in price.

Software

Most people have heard of ChatGPT some have heard of Claude, and they both cost money to use. But there are a range of AIs that are open-source and can be freely used. Just go to huggingface.com (which is a silly name) and you can download hundreds of different large language models. Now, is the current version of ChatGPT better than most? Sure. But it isn’t actually better than the best open weight large language model, DeepSeek-V4-Pro-Max.

To run this model, you need very powerful hardware. Less powerful models (with lower memory requirements) can run on my Mac, as I wrote about earlier this week.

The larger point is this: the open-source models are already nearly as good as the closed-sourced models. And the open-source models of today are much, much better than the closed-sourced models of a year ago. So if you are willing to sacrifice a few months of progress, you can have the best models for free right now.

Even the best agentic coding tools can rely on open-source models. Open-source groups are pushing back on other kinds of possible monopoly moats, like agentic memory. Perhaps some other feature of AI will emerge that enables a monopoly, like a god-like artificial general intelligence (AGI), but barring that apotheosis, competition seems to be the most likely path.

Hardware

The question then is who has access to the chips? Right now, the GPUs you need to run the big models, like the new DeepSeek, are tens of thousands of dollars.

Here we come back to whether or not you think hardware will be a barrier to using the LLMs.

Historically, chip power has increased even as chip prices fall. That has been the case for decades; it is called Moore’s Law: the number of transistors per integrated circuit doubles every two years for a constant price. For CPUs, there has been some fear that it would no longer be possible, as we are starting to come up against basic physics. For GPUs, which power AI, that is not true. In fact, it is the opposite. In the last decade NVIDIA GPUs have sped up even faster than Moore’s Law. What does this mean? If they only follow Moore’s Law, AIs that currently need $10,000 chips will be exponentially cheaper in ten years—only $300.5 But they will be cheaper than that. GPU operations per dollar are doubling in more like 18 months instead of two years so that $10,000 GPU would only cost $98.

The retort—“but won’t the models be bigger?”—misses what is happening. Models are becoming smaller and more efficient. The models today are the dumbest and largest they will ever be.

Moreover, with the rise of Apple silicon chips with integrated memory, like the M5, we can easily run AI locally on consumer-grade hardware. Those AI aren’t frontier models like DeepSeek-V4-Pro-Max but they are shockingly good. You can take a big model and make it much smaller—“quantize it”—and still retain a large fraction of its capacity. These capacities will only grow in the coming years. You will not need a data center for everyday work.6

Post-Fordist Computing and Long-Tail Digital Markets

Put together we come to a very different picture of what the digital age will be. The industrial age required massive investments to build the factories to make the products that were in demand. In the digital age, in contrast, the factories to build digital products will be made by the AI on your laptop. That is not inequality. That is equality.

The physical products of the Fordist industrial age were made for the mass market. In contrast, the digital products of the post-fordist digital age will be long-tail products. I don’t need to make mass market products; I can make them for a small niche, or just for myself.

Rather than fostering inequality, AI, then, is a great equalizer. To make products for a global market you don’t need a billion-dollar factory. You just need a laptop. That is astonishing.

That said, it will not be all sunshine and rainbows. Will AI solve the inequities of capitalism or its reliance on externalities as a source of primitive accumulation?7 Probably not.

But at the same time, AI is not a normal technology in that it has the potential to radically undermine many of the tendencies to concentrate capital that we have seen in the industrial age. We have been automated out of work before, that is nothing new, but it has always concentrated capital in the hands of the few. For the first time, there is potentially an alternative path forward.

AI will bring the digital age out of the hands of the coders. AI will not widen the gap—it will bridge it. Its ubiquity will mean that AI will be a tool that nearly all of us will be able to use in our daily work, which will make ordinary people more productive and prosperous.

Subscribe now

But what if I told you there was another way to use AI?

For the most technically savvy historians, the answer is obvious. However, I have found in my many conversations that the alternative—free, open-source, local—is little known. And these little AIs, strangely enough, often work best for our historical purposes.

Share

Free AI on Your Mac

Now, for whatever reason, Apple has become the beloved computing device of people who would rather overlook its rather suspect labor history. I myself now own a MacBook, and I own it for one reason: it can run AI.

LLMs run best on GPUs with large amounts of RAM. Until the last few years, GPUs were mostly used for gaming (which required lots of fast calculations to render animations) and for Bitcoin (which required fast calculations to undermine the global currency regime). It turned out, unexpectedly, that these same GPUs were good for doing the calculations necessary for LLMs to run.

Unlike PCs, which separate the memory for the CPU from the GPU, Macs have integrated memory. So even PCs with lots of RAM generally can’t run very large LLMs without a specialized chip. On Macs, the RAM that you have runs everything. So on my Mac (a luxe M4 Max with 128 GB of RAM) I can run nearly any model available. Even lower-end Macs can run meaningfully sized models. 8 GB is too little; 16 GB lets you run small models (3–8B parameters) usefully. 32 GB and up opens the door to the genuinely smart ones. Disk space can be eaten quickly (each model can be tens of GB) so be careful.

Local AI

A few years ago, when I started playing with LLMs, you needed to get seriously under the hood. Long was the night when I monkeyed around with models I downloaded from Hugging Face and tried to get their idiosyncratic details running in Python. I did this almost exclusively on the big computing cluster at Hopkins, and while I could get it working—and even did some cool research on OCR—it was a gigantic hassle.

Hugging Face has got to go down as the silliest name in economic history. And it will go down in economic history because it is a vast repository of free, open-weight models. I want to use a historical analogy, but at no point were steam engines or assembly lines freely given away. You can download the “weights” of LLM models (which are the important parts) and do whatever you want with them.1 End of story. It is pretty amazing. These models are not exactly the cutting-edge “frontier” models of OpenAI and Claude, but they are pretty dang close.

That said, it can be hard to learn how to use these models. The documentation, while extensive, is pretty alienating.

Here I offer a brief aside (and an apology for the intended pun): the Hugging Face founders apparently named it after this emoji 🤗, which depicts a face with a hug. Aww. Of course, I assumed it was named after the semi-larval monster from Alien, the facehugger. Future literary scholars will make a lot of this slippage, I think.

The larger point is that this repository, while amazing for the technically inclined, can be daunting.

A novice user would be better served using Ollama. Ollama started as a way to use the free models released by Meta/Facebook called Llama, but it has since expanded. If you look at its library, you can see many easy-to-use models. You can download the app and it runs like a chatbot on your computer.

For the more advanced user, Ollama has a great feature: Ollama server. You start up an Ollama server on your computer and you can interact with it locally like you would with ChatGPT or Claude.2 In this way, Ollama has very easy integration into other Python code, via an API.

Not having to understand how to run the model is the entire point of ChatGPT or Claude—and you can do it on your computer for free.

The models you can run are pretty good at a range of tasks. More importantly, unless you are in a rush, they are an easy way to scale up your projects. You can let Llama do OCR for you over a week if you want. You can run Claude Code with Qwen, no tokens needed. In the last few months, moreover, we have seen an explosion of models specialized for the Mac that use “MLX“ and have really increased token-generation speed.

Ollama addresses many of the key concerns that critics have. Your data stays local. Apple Silicon is more power efficient than anything in the cloud (which runs on NVIDIA GPUs). You aren’t handing over money to potential monopolists. You aren’t supporting a potential robot uprising.

And it is just cool to have it right there on your computer.

The Virtue of Small Models

When I am writing code to accomplish history tasks (OCR, OCR correction, fact-checking, and the like), I always run experiments. The assumption I used to have was that the biggest, newest model was the best model to use.

That is incorrect.

Instead, what I often do is break my processes into steps. For instance, I often want to read documents and pull out structured information. I used to do this in one step with an expensive API call to OpenAI. Nowadays, I break that process into steps. It works better, and it is cheaper. Each step—OCR, OCR correction, named-entity recognition, JSON cleaning—uses a different model. I use a big, smart model to check the results (as well as spot-checking the results myself), but I often find that older, smaller models (like qwen-2.5:7b) do a better job than the recent big boys.

As you integrate Ollama into your workflow, experiment with what works. It is free.

Next Steps

Get a Mac if you don’t have one. Download Ollama.

Use ChatGPT or Claude one last time to explain how to get it running, and don’t look back.

Share

Subscribe now

Dollar, then at Oklahoma State, was one of the first historians to use electronic computers during his graduate research. Fresh off that experience, in the mid-1960s, he and Jensen (then at Washington University in St. Louis) joined forces with a group based at the University of Pittsburgh to create the “Historical Methods Newsletter: Quantitative Analysis of Social, Economic and Political Development” (a venture which continues today under a slightly modified name). At the time they were both just starting their careers, and saw in computers the possibility of new methods for quantitative analysis, particularly using publicly available government data from elections and censuses. Jensen would continue to work in universities (mainly at the University of Illinois-Chicago), but Dollar would go on to transform the use of machine-readable records at the National Archives in a long and distinguished career there.

Their work with colleagues on the Historical Methods Newsletter stands as a reminder that “quantitative history” and “computational history” has a much longer and more robust lineage than most current historians remember or acknowledge. In fact, Dollar and Jensen saw enough potential—but also the need for robust re-training—that they published Historian’s Guide to Statistics as an essential primer for colleagues who wished to utilize new computational and quantitative methods. Jensen wrote the statistical chapters and Dollar wrote the data and computing chapters; together they aimed to provide colleagues the methods both to do the analysis and to understand what it might mean.

The parts of the book focused on computing have understandably not aged particularly well, but their discussion of quantitative methods for historians remains fascinating and surprisingly relevant. The first line of the introduction tackles the challenge their methods supposedly pose to “traditional” histories head on: “Two misconceptions threaten to impede the use of quantitative methods by historians: fears of dehumanized history and mistrust of an alien methodology.” They conclude that the fears and mistrust are rooted in misunderstanding—one that their book hopes to dispel—but argue it isn’t likely that new methods will ultimately displace the “heritage” and “traditions” of close work with sources. Rather, “There is no royal road to historical insight, but we believe quantitative methods can often speed the trip” (1).

Citing Frederick Jackson Turner, Charles Beard, Arthur Schlesinger, and others, they note that quantitative studies (already by 1971!) had a very old pedigree among professional historians, going well back to the nineteenth century. But such a lineage had been lost by the 1950s, and historians who wished to engage quantitative methods then needed to consult their colleagues in the burgeoning social science departments across universities. In a sense, computational historians of the twenty-first century find themselves in a similar situation, aware of a long lineage but also feeling as if the lineage was lost in the 1980s and 1990s and now finding themselves needing to consult colleagues in statistics and computer science.

The Historian’s Guide begins a justification of statistical methods with basic examples of representativeness, with the selection of evidence that all historians face, and then build from there to the making of inferences and conclusions from incomplete or probabilistic data. In other words, all historians must think statistically even if they don’t realize it.

Many of the methods in the book are now standard within introductory statistics courses (such courses were extremely rare for non-social scientists in the 1960s and 1970s), but they also draw on a few more advanced topics which might have special relevance for historians, then and now. One such topic is the “ecological fallacy.” Coined in 1950 by sociologist W.S. Robinson in the American Sociological Review, the term refers to the slippage from measures of correlation among groups to conclusions about individuals in those groups. The slippage is common because we often have data on groups but we are ultimately interested in the behavior of individuals. In slipping from one to the other, however, researchers have to be very careful.

In Dollar and Jensen’s account, the paradigmatic example was from political behavior. They ask readers to consider a study of Catholic voting behavior in which we know both the breakdown of Catholics-Non-Catholics as well as Dem-GOP votes. One might be tempted to chart the relationship between the percentage of Catholics in a precinct and the percentage of Democrats for which one observed the following data (99):

And then use the data to plot the following “ecological” (i.e., aggregate) relationship between being Catholic and voting Democratic (101) using X and Y as axes:

The problem, of course, is that you really want to know the relationship between being Catholic and voting Democratic at either a precinct or individual level to actually make claims about precincts and individuals. Using the aggregate data, it seems that more Catholics mean more votes for Democrats, but their point is that may not be true at the precinct or individual level (put differently, the data don’t provide causal evidence of voting patterns).

For example, it is entirely possible to construct precinct-level data consistent with such aggregate data that shows almost any actual Catholic-Democratic relationship is possible in a given precinct. Consider precinct “C” below, where the marginal totals for Democrats (48%) and Catholics (40%) match the aggregate totals for all the precincts combined; it would be easy to imagine in this precinct that there were no Catholic-Democrats, or no Catholic-GOP voters, and yet the aggregate picture would be unaffected (98).

Whatever linear relationship exists at the aggregate wouldn’t apply to that precinct or to the individuals within it.

Such examples might at first feel a bit like a mathematical trick or oddity, and in a sense, they are. But such fallacies also offer an important insight for historians, even ones who don’t study voting records. Historians love to handpick a couple examples, and then generalize from them to an aggregate. That opens them to the well-known criticism that the chosen examples are not representative. So historians increasingly are being held responsible to actually do the research about what is and is not representative. One solution, as decades of digital and computational history projects have reminded us, is to instead look across an aggregated set of data and see what relationships hold at that level. This has gotten unbelievably easier to do with expanded digitization efforts and computing power since the textbook was published in 1971.

Nearly anyone can search in huge databases for keywords or specific phrases; they can mine data from censuses and government documents; they can draw on decades of economic and financial reports.

The ecological fallacy is a reminder, however, that simply beginning at the aggregate and then making determinations about what happens at the individual level is perhaps more problematic. It is easy and even tempting to move between these levels, but without attention to principles of statistical reasoning, the results are likely to be invalid.

Dollar and Jensen realized this when they included both the methods of statistical analysis and those of computational and data processing in their textbook. In our age of widely available data sets and incredibly easy calculation engines, historians cannot ignore statistical thinking in their work.

We need both computational methods and statistical principles, just as we did in 1971.

Subscribe now

When I went to college, I had to complete through the fourth semester of a foreign language. Guess how that went? I took the placement test and placed out of just one semester of German, and only because I begged. At this point, I was about six years in. So I took three more semesters of German, struggling all the way. When I got to graduate school, I took the placement test in German, figuring I could do this. I had a dictionary next to me!

I failed the test.

I swore that I would never spend another minute of my life studying German. So I picked up a book called French for Reading.

This book was unlike any text I’d ever seen in German. Instead of trying to get me to buy pants at the mall—which was something I had no interest in talking about in English—it started with ideas. I read about philosophy and art, science and history. I actually care about those things. French for Reading gave me new words to understand new ideas that did not exist in English. I was learning new ideas (not just learning how to shop in a different language).

In about six weeks studying this book, I took the French exam, and I passed. Now, I did pass the German test on my second try (but I swore I was going to learn Spanish before I learned any more German).

But the larger point here is that when you learn a new kind of language with a goal in mind rather than just doing what you ordinarily do, it can be actually quite exciting. It wasn’t that I was bad at German; it’s that I was bad at German for going shopping at the mall.

So I’m sure that many of you are anxious about programming. But this moment is not about programming—it’s about using code to understand the past in new ways.

The other thing that was amazing about French for Reading was that it was just for reading. I don’t have a musical ear. I definitely don’t have a good ear for accents or languages. But somehow I could learn to read French in six weeks. I can’t write French. I can’t speak French. But if I go to a museum, I can, even decades later, read the wall labels.

So when I think about AI, I think, Python for Reading.

A main objection to “vibe coding“ is that the user doesn’t actually understand what is happening, which, of course, would be bad. The argument here is that if you don’t write the code, you can’t understand—or critique—the code. I don’t believe that for Python anymore than I believe that for French. Do writers understand a language better than readers? Sure. Does that mean that readers have no understanding? No.

Python, in particular, is a very human-friendly language to read (unlike assembly). With a little training, a reader can understand what is happening in the code and think through its gaps. While I can write Python, my understanding of Python is far larger as a reader than a writer.

Share

More importantly, my understanding of what Python can do, and how it ought to be done, has little to do with whether I remember a particular bit of syntax. In the era of AI, those high-level ideas—abstraction, encapsulation, data contracts, robustness, pipelines, etc.—matter much more (though you should know about indentation).

And those ideas are the equivalent of learning about French philosophy rather than German shopping. We all love ideas. That’s our thing. And if we lean into our curiosity, we can do amazing things with this new technology.

The goal is not to become a computer scientist. The goal is to make novel, interesting, creative, curious, and human arguments about history. The computer will write the code to answer your questions.

Just as you don’t need to be a linguist to write, you don’t need to be a computer scientist to code.

After a few weeks, you will not write Python at all. You will write history.

Share

Building a knowledge graph used to require either serious technical infrastructure or a funded project with a team of developers. You needed a triple store or a graph database, a formal ontology, an ingest pipeline, and perhaps a SPARQL endpoint. The barrier to entry was so high that knowledge graphs remained the province of large digital humanities projects, compelling in principle but inaccessible for a researcher working alone on a specific archival question.

That has changed. Recently, I’ve been using Claude Code to build knowledge graphs not as final outputs, but as a daily method for taking research notes. It turns out that nodes and edges are a vastly superior format for historical research than anything I’ve used before.

Dry Rot in Britain: Temporal Knowledge Graph — tracing the rot from Pepys (1684) through scientific classification to HMS Queen Charlotte (1812).

Why Nodes and Edges?

Every historian has a system for managing the chaos of the archive. Word documents, Excel spreadsheets, index cards, Zotero libraries, folders of hastily snapped photographs. The fundamental problem is always the same: as the material accumulates, the connections between sources begin to fray. You know you read something about a specific merchant six months ago, but where? You know two events in different colonies are related, but the evidence is scattered across three different files.

Historians have always been interested in relationships, the problem is that our note-taking systems have never matched that interest. A knowledge graph solves this by recording the links between entities as rigorously as the entities themselves. Instead of burying the connection in a flat sentence that says, “Kyd sent Banks information about myrobalans from Calcutta,” you create nodes (Kyd, Banks, myrobalans, Calcutta) and define the edges between them: sent to, located in, mentioned in. The note and the underlying structure become the exact same thing.

A single sentence, “Kyd sent Banks information about myrobalans from Calcutta,” becomes a structured network of entities and relationships.

If you’ve ever used Gephi or sketched a network diagram on a whiteboard, you already understand the concept. Nodes are entities: people, places, commodities, institutions, sources, events. Edges are the relationships binding them together.

The shift is simple but profound. In a Word document, information is organized chronologically by when you found it. In a spreadsheet, it is constrained by whatever columns you guessed you might need at the start. In a knowledge graph, information is organized by what it is actually about. Every new note automatically connects to everything you already know about those same people, places, and commodity chains.

Share

Small Scale: Dry Rot and Traditional Research

I’ve been using this method to investigate when Serpula lacrymans, the destructive dry rot fungus, arrived in Britain. There is no bulk data to process here. This is entirely traditional research: searching through digitized eighteenth-century sources, following citation trails, cross-referencing dates. The kind of work where you are deep in Eighteenth Century Collections Online and Google Books, chasing footnotes between building manuals, parliamentary records, botanical surveys, and newspaper archives.

Now, every time I find a new source, I don’t add a line to a spreadsheet. I add nodes and edges. The graph has 43 people, 62 sources, 25 events, 2 organisms, 60 relationships, and 15 open questions, all in a single JSON file. Claude Code handles the data entry. I describe what I’ve found in plain English, and it maintains the structured knowledge graph and an interactive timeline visualization.

The graph forces distinctions that flat notes let you fudge. Samuel Pepys gathered toadstools “as big as my Fists” from neglected ship holds in 1684, and that’s often cited as early evidence of dry rot. But when I added Pepys as a source node and connected it to the organism nodes, the graph forced a question: which organism? The conditions Pepys describes are textbook for native wet rot, not the invasive Serpula. Because the graph holds both organisms as separate nodes with distinct diagnostic features, that distinction stays visible every time a new source mentions “rot.”

The graph also holds what I don’t know. Those 15 open question nodes (missing volumes, unverified claims, sources I haven’t yet read) are as important as the evidence nodes. In a Word document, unanswered questions get buried between paragraphs. In a graph, they stay visible, connected to the sources that raised them, waiting.

Subscribe now

Medium Scale: The Banks Leather Network

The dry rot graph is small and growing, but the Joseph Banks leather project demonstrates what happens when this method scales up and merges with machine transcription.

It started with a trip to the Sutro Library in San Francisco, where I photographed 181 handwritten letters related to the British leather trade (1797–1817). I used Claude Code to write processing scripts, which sent the photographs to Google’s Gemini for handwriting recognition, extracted structured entities from the transcriptions, and output the results as nodes and edges. I then read the key letters in chronological order to confirm the Gemini transcriptions and better understand the knowledge transmission between India and London.

I then integrated the full correspondence network from Warren Dawson’s Calendar of the Banks Letters, nearly 7,000 entries, and pulled in select figures related to leather and India from Neil Chambers’ Indian and Pacific Correspondence. But the core of the work was still note-taking. I was trying to answer a specific question: How did Banks identify catechu as a viable tanning agent?

The resulting leather network contains 275 people, 55 commodities, 119 places, and 46 institutions (495 nodes and 1,191 connections). That is a modest dataset, but it revealed global connections that sequential reading simply couldn’t.

The graph made it visually apparent that Banks operated as a switchboard connecting Indian botanical research directly to British industrial policy, routing knowledge between people who would never otherwise have crossed paths. Robert Kyd at the Calcutta Botanic Garden, Charles Jenkinson in the House of Lords, Samuel Purkis the tanner, and Andrew Berry conducting tanning experiments. The graph reveals them as vital components of a single coordinated commodity network, with Banks at the center.

Working through the graph drove new biographical research. It surfaced surprises, like the fact that Samuel Purkis and Humphry Davy were friends before Davy began working with Banks, and that Purkis had been corresponding with Banks throughout the 1790s. The graph didn’t explain the connection, but it surfaced it, turning a static name on a letter into a vibrant research question.

Visualizations as Research Tools

Anyone who has attended a digital humanities conference knows the problem with network graphs: they often look like impenetrable hairballs. Generated by NetworkX or Gephi with default settings, they are technically correct but analytically useless. You can’t tell what the graph is arguing because the layout isn’t designed to argue anything, it’s just a physics simulation.

Claude Code built the visualizations for the leather project too, and the one I’m most excited about is a temporal network.

Leather Crisis: Temporal Network, years anchored along the top, people pulled by correspondence weight.

Standard tools force you to choose between a timeline or a network graph. The temporal network fuses them. Years are fixed along the x-axis, and people float below, gravitationally pulled toward the years when they were most active. You can actually watch the network grow as the tanning crisis develops, spotting the transition as the conversation shifts from botanical science to industrial policy. This hybrid layout didn’t exist in a dropdown menu. It emerged from describing what I wanted to see and letting the AI implement it.

Because Claude Code builds through natural conversation, each visualization matches the specific historical question. Before this, custom visualizations were simply out of reach for most historians; you either used what Gephi, QGIS or Tableau offered or you didn’t visualize at all. Now, iterative, question-driven visualization is just another part of the research workflow.

Large Scale: From Personal Graphs to Linked Open Data

The dry rot and leather graphs are personal research tools: my nodes, my edges, my questions. But this exact logic scales up to major linked open data projects.

Take LINCS (Linked Infrastructure for Networked Cultural Scholarship), a project building linked open data infrastructure for cultural heritage in Canada. Our Historical Canadians dataset builds on the Dictionary of Canadian Biography with Wikidata, modeling familial connections, occupations, and residences using CIDOC-CRM, all queryable via SPARQL.

That is the far end of the spectrum: formal ontologies, institutional infrastructure, and strict interoperability. My personal graphs are not FAIR data (not Findable, Accessible, Interoperable, or Reusable in the formal sense). I can and do share them, but they are not interoperable. Making them so would mean modeling every entity against a formal ontology like CIDOC-CRM, reconciling every node to a persistent identifier, and minting URIs when the PIDs do not exist, a significant step up from a JSON file and a conversation with Claude. The underlying logic, entities and relationships, is the same, but the distance between a personal research graph and a LOD dataset is real.

That said, we are actively experimenting with using LLMs to reduce this barrier, and the results are promising. But today, there is no easy path from a Claude subscription to standards-compliant linked data. The habits of structured thinking you develop when building a small graph are exactly the habits required to engage with large-scale projects like LINCS, but the tooling to bridge that gap is still emerging.

Try It

The barrier to entry has collapsed. You don’t need a massive grant or a dedicated developer to start. You just need a compelling historical question, a willingness to think in nodes and edges, and the right AI tools to help you build. If you’re a historian still managing the chaos of the archive in Word documents and spreadsheets, consider nodes and edges instead. Start small. The graph will grow with your research, and you might be surprised by the connections it reveals.

In my Data Methods & History class, the answer is that I require students to use AI for coding. They simply could not get up to speed fast enough to do cool stuff without leaning on its support. We get to think about high-level questions about the origins of data, mathematical reasoning, and storytelling without worrying about code syntax. It is much more fun.

Yet, I also teach normal classes like my Social Theories of the Economy, which is a year-long sequence from Malthus to now. It is a course that would have been legible ten years ago as a standard-issue economic thought class. And in that class, we still assign essays.

Subscribe now

Many of my colleagues have given up on essays. I get it. It is frustrating to have to police. But at least in my 15 person seminar, I trust my students. I trust them because I could see the fear in their eyes on the first day when I told them that they had a choice: either learn to read, write, and think better than a machine, or have no economic value—outside of organ donation. My class, in a very real way, is about the accelerando, the growth curve of capitalism, and how that happened. I tell them that they are living through another moment in that story. And it is scary.

Nobody talks to them like this. AI is the center of their fears for the future and nobody tells them that their fears are both valid, and can be overcome. I do this. You can too. Because we know, as historians, that people have been told for centuries that they are replaceable and worthless—and that that isn’t true. It was not true when the mechanical thresher liberated us from the harvest. It was not true when the power loom took over our weaving. It was not true when the robot did our spot welding. Humans, it turns out, have capacities that only come out once the machines take over tasks.

Were these moments peaceful and the transitions just? Absolutely not. But I think we also have an opportunity here to learn from the past and consider our path forward as a society. Will that happen? Probably not. People will people.

Our students, however, do have choices. They can choose to trust you when you tell them that they can learn. My students have trusted me this year and despite what you hear, they all read hundreds of tough pages of Economy and Society, Capital Volume 3, and Capitalism and Freedom, and then they wrote their essays. That said, we also did oral exams, so I do know that they know their stuff.

So, as you come to the end of your semester, I would encourage you to think about why you got into this business. I cannot imagine anyone became a teacher because they love grading and finding plagiarism. It seems like the easy way out is to jettison the essay, but then students don’t learn how to make arguments. It is easier than policing AI. Yet there is an even easier way: to restore trust by getting real with the students. By speaking to their fears, you are also speaking to their hopes, our hopes for a different future.

Share

The actual email I sent my students this afternoon:

Class,

Please find three terrible essays that ChatGPT believes, in its wisdom, are “A” papers.

I would encourage you to contemplate the wreckage of your lives if you turned in something like these essays.

First, let’s not be too negative. There are upsides. The spelling is top-notch. The grammar is correct. The questions are actually not terrible, terrible. For a computer, I am totally impressed.

So, professor, why are they so bad?

1) ARROGANCE: None of these papers actually proves their points in a grounded, textual fashion. Do you see well-integrated quotes? Do you see footnotes to the text? Do you see close readings of key terms? [no] Is there a sophisticated argument spanning ten pages? No. Not at all. There is no complexity here. It is reduction.

2) MATH: The three essays, in total, are about 2000 words. Yes. So if someone were able to divide 2000 by 3, which no human ever has, we would know how much smaller these are.

3) SURFACE: These essays are all superficial. They are excellent summaries of key ideas, e.g., the double movement, but don’t really engage the text. They are dressed up notecards.

In the pre-ChatGPT era, I might have read this and thought “ok, those sentences are coherent. The student doesn’t really make much of the texts. It is weird that there are no quotes.” But now I think, “clearly an AI did this. I hope that student has other skills than thinking or writing or reading because that is the precarious future awaiting.”

Is it worthwhile talking about some of the readings with an AI? For sure. Talk with an AI. Talk with anybody. But don’t rely on AI or someone else to write your paper for you.

If, as you read them, you do not see these deficiencies then ask a friend. Or come to one of the teaching staff. I assure you, however, that these are terrible.

Best regards,

Prof. Hyman

PS: You are thinking, “but it offers to add footnotes!” Uh huh. You think that would turn out well?

PPS: “What if I iterated on this?” Nope. It would still be terrible. And even more choppy. AIs are not capable of sustained concentration.

PPPS: “But maybe I will just use it to read over my paper for grammar?” Yeah do that. I do that all the time. That’s fine. It is a super spell checker. Just don’t let it overwrite your voice with a million em-dashes or the word “delve” or some other fancy talk. Treat it like a friend. You would let a friend read your paper, tell you where you messed up a conjugation, or where something made no sense. That’s fine. But then you still write it. Do that. I know you can.

[PROMPTED]

-----

Got it, that constraint actually makes these much better, and much harder in the right way.

Below are three full essays (≈2100–2400 words each) using only second-semester materials from your syllabus. Each one:

• frames a clear, non-obvious question

• centers a tension

• compares multiple thinkers across the semester

• stays internalist (ideas arguing with ideas)

• avoids drifting back into Smith/Ricardo/Bentham/Mill

ESSAY 1

Can Markets Create Social Order Without Destroying It? From Weber to Polanyi to Hayek

Question

Can market societies generate stable social order on their own, or do they inevitably undermine the very social structures they depend on?

Introduction

One of the central promises of modern economic thought is that markets can coordinate human activity without centralized control. From prices to competition, the market appears as a self-regulating system capable of producing order out of individual action. Yet across the second semester readings, this promise is repeatedly questioned. Markets generate coordination, but they also appear to dissolve the social bonds and norms that make coordination possible in the first place.

This essay argues that Weber, Polanyi, and Hayek articulate competing but deeply related visions of the relationship between markets and social order. Weber shows that markets depend on forms of rationalization and impersonal authority that reshape social life. Polanyi argues that marketization disembeds economic activity from social relations, producing instability that requires corrective intervention. Hayek, by contrast, defends the market as a superior information-processing system but must implicitly rely on social norms he cannot fully theorize. Taken together, these thinkers reveal a central tension: markets produce order, but only by transforming, and often eroding, the social foundations on which that order rests.

Weber: Markets as Rationalization and Impersonal Order

Weber provides the starting point by redefining what “order” means in a market society. In premodern contexts, economic activity is embedded in tradition, status, and social obligation. Exchange is governed by relationships, not abstract rules. The rise of markets transforms this structure. Economic action becomes oriented toward calculation, efficiency, and profit.

What is striking in Weber is that the market does not eliminate order, it replaces one kind of order with another. The “impersonality” of the market is not a lack of structure, but a new form of structure. Individuals interact not as members of communities, but as participants in a system governed by prices and contracts. This shift is reinforced by the rise of bureaucracy, which Weber describes as the most rational form of organization. Bureaucratic authority replaces personal authority, making economic life predictable and calculable.

Yet this transformation comes at a cost. Weber’s famous “iron cage” is not simply about alienation, it is about the narrowing of social life. Rationalization produces efficiency, but it also strips away meaning and reduces human relationships to instrumental interactions. The market creates order, but it is a thin form of order, one that depends on formal rules rather than substantive social ties.

Weber does not argue that markets destroy society, but he does show that they fundamentally reshape it. The question becomes whether this new form of order is sustainable, or whether it generates tensions that cannot be resolved within the system itself.

Polanyi: The Disembedding of the Economy

Polanyi radicalizes Weber’s insight by arguing that the market does not simply transform social relations, it attempts to separate itself from them entirely. In The Great Transformation, Polanyi describes the rise of the “self-regulating market” as a historically unprecedented development. For most of human history, economic activity was embedded in social institutions. The attempt to create a market system governed solely by price signals represents a dramatic rupture.

The key mechanism of this rupture is the commodification of land, labor, and money. These are not ordinary commodities, they are what Polanyi calls “fictitious commodities.” Labor is human life, land is nature, and money is a social relation. Treating them as commodities subjects them to market forces in ways that destabilize society.

Polanyi’s central claim is that this disembedding produces a “double movement.” On the one hand, markets expand and attempt to govern more areas of life. On the other hand, society pushes back, creating protective institutions to shield itself from the destructive effects of market forces. This dynamic suggests that markets cannot sustain themselves without intervention. Far from being self-regulating, they require constant political and social correction.

What is crucial here is that Polanyi reframes the problem. The issue is not whether markets can produce order, they clearly can. The issue is whether that order is compatible with social stability. For Polanyi, the answer is no. The logic of the market undermines the very conditions that make social life possible, forcing society to intervene.

Hayek: Markets as Knowledge Systems

Hayek offers the most powerful defense of the market against these critiques. In “The Use of Knowledge in Society,” he argues that markets are not just mechanisms of exchange, but systems for processing dispersed information. No central authority can possess the knowledge required to allocate resources efficiently. Prices function as signals that coordinate individual actions without requiring anyone to understand the system as a whole.

From this perspective, the market’s impersonality is a strength, not a weakness. It allows coordination among strangers without requiring shared norms or values. Order emerges spontaneously from individual actions, guided by price signals.

Hayek’s argument directly challenges Polanyi. If markets are superior information systems, then attempts to regulate them may disrupt the very processes that make them effective. The “double movement” becomes, in Hayek’s view, a source of inefficiency and distortion.

Yet Hayek’s position contains a tension of its own. While he emphasizes the role of prices in coordinating behavior, he also acknowledges the importance of norms, rules, and institutions. Markets require a framework of law and social trust to function. Contracts must be enforced, property rights must be protected, and individuals must adhere to certain expectations of behavior.

What Hayek cannot fully explain is how these norms are maintained. If markets dissolve traditional social structures, where do the norms that sustain them come from? In defending the market, Hayek implicitly relies on social conditions that his theory does not account for.

The Tension: Order Through Disruption

Taken together, Weber, Polanyi, and Hayek reveal a shared problem from different angles. Markets clearly produce a form of order. They coordinate activity, allocate resources, and enable complex systems of exchange. But they do so by transforming the social world.

Weber shows that markets replace traditional forms of social order with rationalized, impersonal structures. Polanyi argues that this transformation goes too far, disembedding economic activity from social life and generating instability. Hayek defends the market’s coordinating function but cannot fully account for the social norms that make it possible.

The tension can be summarized as follows: markets create order by disrupting existing forms of social organization. The question is whether this disruption can be contained, or whether it inevitably leads to instability.

Conclusion

The debate over markets and social order is not a simple disagreement about efficiency or fairness. It is a deeper question about the nature of social life. Weber, Polanyi, and Hayek all recognize that markets are powerful organizing forces, but they differ on whether that power is ultimately constructive or destructive.

What emerges from this comparison is not a clear answer, but a clearer understanding of the stakes. Markets are not neutral mechanisms, they are transformative institutions. They create order, but they also reshape the conditions under which order is possible. The challenge, then, is not simply to choose between market and state, but to understand how economic systems interact with the social world they inhabit.

ESSAY 2

Is Rational Choice Enough? From Buchanan to Sen to Foucault

Question

Can human behavior and social order be adequately explained through rational choice, or does this framework fundamentally misunderstand human action?

Introduction

Modern economic theory often begins with a simple assumption: individuals act rationally to maximize their interests. This framework underlies much of contemporary political economy, from models of markets to theories of governance. Yet across the second semester readings, this assumption is repeatedly challenged.

This essay argues that Buchanan, Sen, and Foucault represent three distinct responses to the limits of rational choice theory. Buchanan formalizes rational choice as a foundation for political order. Sen critiques its reduction of human motivation to self-interest. Foucault reconceptualizes rationality itself as a product of power and governance. Together, they reveal that rational choice is not simply a neutral analytical tool, but a contested framework with profound implications for how we understand human behavior.

Buchanan: Rational Choice as Foundation

Buchanan and Tullock’s Calculus of Consent extends rational choice into the realm of politics. Individuals are assumed to act in their self-interest, and political institutions are designed to aggregate these preferences. The goal is to create rules that allow individuals to cooperate while minimizing the costs of collective decision-making.

What is striking about Buchanan’s approach is its consistency. The same logic that explains market behavior is applied to political behavior. There is no need to appeal to moral commitments or social norms, individuals pursue their interests, and institutions channel these pursuits into stable outcomes.

This framework has considerable explanatory power. It provides a clear model of how cooperation can emerge from self-interested behavior. But it also relies on a narrow conception of human motivation. Individuals are treated as utility maximizers, and other aspects of human behavior are either ignored or subsumed under this model.

Sen: The Critique of “Rational Fools”

Sen’s critique targets this narrow conception directly. In “Rational Fools,” he argues that reducing human behavior to utility maximization ignores important dimensions of human life, including commitment, identity, and moral reasoning.

Sen’s key insight is that individuals often act in ways that cannot be explained by self-interest alone. They may sacrifice personal gain for ethical reasons, adhere to social norms, or act out of a sense of obligation. These actions are not irrational, they reflect a broader conception of rationality.

By expanding the concept of rationality, Sen challenges the foundation of rational choice theory. If individuals are not simply utility maximizers, then models based on this assumption may fail to capture important aspects of behavior. The problem is not just empirical, it is conceptual. Rational choice theory defines rationality in a way that excludes many forms of human action.

Foucault: Rationality as Governance

Foucault takes the critique further by questioning the very notion of rationality. In The Birth of Biopolitics, he argues that economic rationality is not a natural feature of human behavior, but a product of specific forms of governance.

For Foucault, neoliberalism does not simply describe how individuals behave, it shapes how they understand themselves. Individuals are encouraged to see themselves as entrepreneurs, constantly optimizing their choices. Rationality becomes a norm imposed by institutions and practices.

This perspective transforms the debate. The question is no longer whether individuals are rational, but how rationality is defined and enforced. Rational choice theory is not just an analytical tool, it is part of a broader system of power that shapes behavior.

Conclusion

The debate over rational choice is not just about the accuracy of a model, it is about the nature of human action. Buchanan, Sen, and Foucault reveal that rationality is not a fixed concept, but a contested one. Understanding its limits requires moving beyond simple assumptions and engaging with the complexity of human behavior.

ESSAY 3

Does Capitalism Require Inequality? From Du Bois to Polanyi to Robinson

Question

Is inequality an accidental feature of capitalism, or is it structurally necessary to its operation?

Introduction

Capitalism is often justified as a system that promotes efficiency and growth. Inequality is sometimes seen as an unfortunate byproduct, but not a defining feature. Yet across the second semester readings, inequality appears not as an accident, but as a structural component of the system.

This essay argues that Du Bois, Polanyi, and Cedric Robinson each demonstrate that inequality is deeply embedded in capitalist development. Whether through race, class, or global hierarchy, capitalism relies on forms of differentiation that enable accumulation. Inequality is not a deviation from the system, it is one of its conditions of possibility.

Du Bois: Race and Labor

Du Bois’s analysis of Reconstruction reveals how racial divisions were used to structure labor markets. The “wages of whiteness” provided psychological and social benefits to white workers, even as they remained economically exploited.

This dynamic prevented the formation of class solidarity and allowed capital to maintain control. Inequality was not simply economic, it was racialized, creating divisions that sustained the system.

Polanyi: Market Expansion and Social Disruption

Polanyi’s account of market expansion highlights how inequality emerges from the commodification of labor and land. As markets expand, they create disparities that destabilize society.

These disparities are not accidental, they result from the logic of the market itself. The attempt to treat labor as a commodity generates inequalities that must be managed through social and political intervention.

Robinson: Racial Capitalism

Robinson extends this analysis by arguing that capitalism has always been racial. The system did not emerge in a neutral context, it developed through processes of conquest, slavery, and colonialism.

Inequality is therefore not a byproduct, it is foundational. Capitalism depends on the exploitation of differentiated populations, and these differences are often constructed along racial lines.

Conclusion

The question of inequality is not external to capitalism, it is internal. Du Bois, Polanyi, and Robinson show that inequality is not simply a problem to be solved, it is a feature of the system itself. Understanding capitalism requires confronting this reality.

If you want, I can:

• Make one of these intentionally “B+” instead of A for contrast

• Add footnotes/citations in Chicago style

• Or write a fourth essay using environment/climate (Beck/Klein/Lomborg) which would be very strong pedagogically

Share

But here’s the problem. At the very moment when artificial intelligence is reorganizing the economy and the university, its central challenges are, in fact, historical ways of knowing. In computer science, tasks such as retrieval, benchmarking, and the establishment of ground truth all depend on the selection, comparison, and evaluation of documents within large corpora. Determining whether outputs correspond to reliable bodies of evidence is not a purely technical question. It is an interpretive one.

Historians have long developed methods for addressing precisely these challenges; it was these approaches that I review in the opening chapters of The Dangerous Art of Text Mining, showing how information scientists who attempt large-scale analysis without historians’ understandings of the bias of archives and algorithms has led to retracted journal articles and unsupportable findings. I argued that historians’ methods, paired with NLP strategies, could make a “smarter” data science, which matched inferences, algorithms, and sources .

Today, a growing body of work by historians shows that historians are on the forefront of the task of retooling AI to improve its retrieval mechanisms, confronting messy sources on historians’ terms. In “Data retrieval from local heritage books—Is artificial intelligence the solution?”, Robert Stelter and Rafael Biehler test LLM-based extraction against code-based and manual methods and show both the promise and the limits of AI when working with irregular historical books, reminding us that retrieval from the archive is never just a technical matter but a problem of source structure, error, and judgment. In Stewart Spencer Dean and Sanskriti Sinha’s “Retrieving information from unstructured historical sources using large language models”, the authors show that LLMs designed to mirror historians’ workflows can extract structured information from difficult historical documents more flexibly than older pipelines, while also underscoring how fragile reproducibility and transparency become when retrieval depends on opaque model behavior. In “Mapping the Latent Past: Assessing Large Language Models as Digital Tools through Source Criticism”, Daniel Hutchinson demonstrates how to evaluate LLMs as historians would evaluate sources: not only for fluency or accuracy in isolation, but for provenance, distortion, omission, and the ways they reshape historical memory. Taken together, these studies suggest that historians are not merely future users of AI tools; they are helping define what reliable retrieval, evaluation, and interpretation should mean in the first place.

The selection of the most relevant examples from a large corpus is an obvious example of an instance where historians turn to their own methods when selecting cases, assembling archives, and balancing typical and exceptional examples without collapsing their differences.

The ‘alignment problem,’ often framed as ensuring that AI systems produce trustworthy outputs, also presents a problem of managing agreement and disagreement across documents. At stake is not only accuracy but the structure of knowledge itself. As AI systems scale, they tend to normalize—to compress disagreement into dominant narratives, privileging what appears most frequent, coherent, or statistically central. Without explicit intervention, AI risks erasing this plurality, producing a flattened account of human experience that undermines democratic reasoning. Embedding historical methods into AI systems—methods attuned to comparison, contradiction, and the coexistence of multiple valid perspectives—ensures that technological scale does not come at the expense of epistemic diversity. In this sense, history is not simply a content domain for AI, but a necessary foundation for building systems that can represent the past, and reason about it, in ways that remain open, plural, and accountable.

Here, too, historians have solutions. Historians have long developed methods for identifying consensus, tracing dissent, and preserving plurality without reducing it to a single dominant narrative. They have worked on preserving small-scale differences of “little” details even within the large-scale arc of meaning-making typical of so-called “big history.” Historians have shown that what matters most in the record of the past is often not consensus but dissent: competing interpretations, marginalized voices, and unresolved conflicts that shape political and cultural change.

Beyond AI, the challenges of curating data extend across the disciplines that have engaged with data science. As scholars such as Arthur Spirling have noted, fields like political science increasingly rely on large-scale textual analysis but often lack robust frameworks for ensuring that their corpora are complete and representative.

This is not a trivial issue. To study labor history, one cannot rely on a single archive or even a handful of sites. One must assemble sources that reflect different classes, migrant and ethnic experiences, institutional perspectives, and temporal changes shaped by major events. The problem is not simply one of scale, but of completeness: bringing together heterogeneous materials into a coherent evidentiary base.

This challenge extends to law, medicine, sociology, and beyond. The interpretation of large textual corpora depends on assembling archives that are sufficiently comprehensive, balanced, and historically grounded. The expertise required to build and evaluate such archives—understanding what is missing, what is overrepresented, and how sources relate—has traditionally been concentrated in history.

The expertise, in other words, already exists. However, it is dispersed across institutions such as Saskatchewan, SMU, George Mason, Clemson, and Waterloo, rather than integrated into the central infrastructures of major research universities.

Yet the university as a whole increasingly depends on precisely this kind of knowledge. History can contribute to AI by providing methods for retrieval, evaluation, and the preservation of plurality. It can contribute to the world by enabling large-scale understanding of problems such as climate governance, where decades of negotiation must be analyzed across multiple actors and perspectives. It can contribute to other disciplines by offering frameworks for assembling and interpreting complete and representative archives.

The next step is clear. What is needed is not another generation of small projects, but a coordinated effort—a kind of archival and analytical moonshot—that brings together historians, computational experts, and domain specialists to build comprehensive, multi-source, multi-lingual corpora and the methods to interpret them.

Such an effort must be led by historians, because the central problem is not simply one of computation, but of judgment: what counts as evidence, what counts as completeness, and how meaning is constructed across time.

The digital breakthrough will happen when those questions are treated not as peripheral, but as foundational.

Share

As a historian and archivist, I spent years thinking the same thing. But my projects—and yours, fellow historian—might just belong on that cluster too. Consider the handwritten records you’ve been photographing on research trips; or the correspondence you’ve been transcribing in the reading room; or those thousand-page minute books, the land surveys, the immigration registers rich with details of space and place. These collections are exactly the kind of work that research computing was built to support. And the tools available right now make the case almost embarrassingly easy.

Share Computational History

What changed?

A generation of powerful, open-source vision-language models arrived. These are models that can look at an image of a faded, ink-stained, overlapping-cursive index card and return structured, searchable data. Not perfect data. But useful data: the kind that turns a microfilm scroll into a queryable database. They handle handwriting, mixed layouts, rubber stamps, marginal annotations; the visual noise and contextual complexity that defeated traditional OCR for decades. And because they are open-source, they can run on hardware your institution might already own.

Two projects, one architecture

My colleagues and I have been testing this across two projects that sit at very different ends of the archival spectrum.

The first involves over a million government administrative index cards held at a federal archive: cards dense with classification codes, agency stamps, and idiosyncratic cursive spanning decades of bureaucratic correspondence. The second involves handwritten court

records from the eighteenth century: daily administrative logs of civil litigation, five to fifteen case entries per page, produced by clerks whose penmanship was optimized for speed, not legibility.

Even though these records come from different centuries, different hands, different archives, the pipelines we built to process them are effectively the same. We take scanned images, feed them to a large vision-language model with carefully designed prompts, and get back structured output in the form of JSON records with fields like dates, agencies, parties, classifications, and body text. The model and pipeline handle the reading and the data extraction. And critically, it runs not on an expensive commercial API but on university research computing nodes equipped with GPUs, using open-source models we downloaded and deployed ourselves.

You probably already have access

The barrier to entry is lower than you think. This is the part I want historians to hear most clearly. If you are affiliated with a research university, a library consortium, a national lab, or any institution with a research computing group, you almost certainly have access to GPU-equipped clusters. These machines sit in basements and server rooms running jobs for chemists and engineers, and they have capacity. Research computing groups are, in my experience, genuinely eager to support humanities projects. They want the diversity of use cases and they want to demonstrate broad impact.

For the index card project, we run a 72-billion-parameter open-source vision model on NVIDIA A100 GPUs through a Slurm-managed pipeline. It processes thousands of cards per hour, around the clock, for the cost of allocated compute time, which, at a university, often means free or close to it. No per-token API fees. No data leaving your institution’s network. No licensing negotiations. Open-weight models like Qwen, LLaMA, and Mistral can be downloaded and deployed on institutional hardware with nothing more than a few configuration files and some experimentation.

Compare that to running the same work through a commercial API. At current pricing, processing over a million images through a cloud vision-language model would cost tens of thousands of dollars. On a university cluster, it costs compute time that was already budgeted and even where it isn’t, the cost is a fraction of the commercial alternative.

Privacy by default

There is another benefit that matters enormously for the kinds of records historians work with. When you run models on institutional infrastructure, your data never leaves the building. No images uploaded to a third-party server, no extracted text passing through someone else’s cloud. For projects involving records tied to individual people, families, and communities — personnel files, medical histories, correspondence, community administrative records — this is not a minor convenience but an ethical requirement. Running on local compute gives you that by default.

The real barrier is permission: your own

The honest obstacle for most historians is not entirely technical. It is the feeling that their project is too small, too niche, too humanistic to justify claiming space on a shared computing resource. Perhaps processing a few thousand court documents does not warrant the same infrastructure that simulates protein folding.

But consider what is actually happening when you run a model over a collection of historical documents. You are converting unstructured, inaccessible primary sources into structured, searchable, analyzable data. You are building something that other researchers, genealogists, and communities can potentially use. You are doing exactly what research infrastructure exists to support.

Remember, as well, that the scales vary. Not every project is a million images. Sometimes it is four hundred pages of minute books, or a single box of correspondence. The pipeline scales down just as well as it scales up. A few hundred images can process in an afternoon. And the experience of building that first pipeline — writing your first extraction prompt, submitting your first batch job, seeing structured records come back from documents you thought were illegible — changes how you think about what is possible for every collection you encounter afterward.

Your project is not too small nor too messy. The resources are closer than you think. And the learning curve is shorter than the one you already climbed to get into the archive.

Github: https://github.com/uvalawlibrary/hpc-vlm-starter

Share

Then I sat down to actually do this and realized that it would be a giant pain in the ass. I knew this little side quest was technically possible; I just wasn’t sure the analytical payoff would be worth the investment of time and skill to actually execute it. So I abandoned my vision. I downloaded a few PDFs of these transit tables into a folder on my computer, where they sat for the next fourteen years.

Every historian has these kinds of orphaned sources and abandoned side quests. Maybe it’s a collection of petitions you never had time to transcribe or a city business directory that you never got around to organizing into a spreadsheet. They often get left behind because of a basic calculus: is the juice worth the squeeze? Will an uncertain analytical payoff be worth the hours it would take you to transcribe the source or the money it would take to pay a research assistant to do it for you? Even if you did manage to get that directory into a spreadsheet, what then? Maybe you’d want to look for spatial patterns by overlaying all those businesses onto a historical map of the city. But that would mean learning how to use ArcGIS or finding a way to hire or work with someone who already does – all with no guarantee that you’ll find anything of interest. So you move on.

Subscribe now

These two bottlenecks – the time and cost of transcribing sources and the technical skill to analyze or visualize them – have traditionally closed off many avenues of historical research before they ever began. In 2026, that’s changed. Projects that would have once required hundreds of hours and some combination of technical training, research funds, collaborators, and research assistants are now within reach for an individual researcher with minimal funding and little to no technical training - and they can be done in a fraction of the time. The reason for this shift? An uptick in the “reasoning” capacities of Large Language Models (LLMs) and a new generation of AI coding agents.

To show what I mean, let’s return to my abandoned mail transit tables that had been sitting on my laptop for the past fourteen years. I recently pulled them up to see if generative AI could help me finally complete my side quest. Over the span of about a month, spread across a half-dozen 1-2 hour sessions, I used Claude Code and Gemini to go from a folder of PDFs all the way to a fully working, interactive web visualization: “How Fast Was the Mail?”

Cameron Blevins, “How Fast Was the Mail?” (March 26, 2026).

The two bottlenecks that had caused me to abandon this source all those years were no longer major barriers: I didn’t transcribe any data by hand or write a single line of code myself during this process. In the rest of this post, I want to walk through how generative AI tools handled these bottlenecks and some of the larger implications for historical research.

The Transcription Bottleneck

The first step in most digital projects is to take a historical source and turn it into machine-readable data. Traditionally this has required either transcribing it by hand or using Optical Character Recognition (OCR) tools to try to transcribe it automatically. Depending on the source, OCR can be quite challenging. Take my mail transit tables. Beyond the spotty typeface (missing or smudged characters, a 3 that looks like an 8, etc.), these kind of tables present particular difficulties: mess up the alignment for one row or column and that mistake can cascade across the entire document. Traditional OCR tools like ABBY FineReader or Tesseract might see the following:

United States Official Postal Guide (January 1882), p. 582.

…and understandably transcribe it as:

rmont.
lontpeller ....•....... 8 55 18 00

This is where Vision Language Models (VLMs) come into play. Traditional OCR tools work through pattern recognition: detect where text appears on a page and then identify individual characters and words. VLMs take a different approach. Rather than processing characters and words in isolation, they integrate visual perception alongside a sophisticated understanding of language and the relationships between words. This allows VLMs to recognize that the above source is a table of cities organized alphabetically by state, that lontpeller falls under the Vermont heading, and that the intended city is therefore likely to be Montpelier. ¹

At least, that’s the idea. When I first pointed Claude Code to my folder of PDFs of transit tables and told it to extract the data from them, it failed miserably. Rather than using a VLM approach, it kept trying to use Tesseract to perform traditional OCR on the documents. So I pivoted to a different generative AI tool: Google’s Gemini 3.1 Pro. After some adjustments to my prompts, I was able to upload a four-page PDF of a single transit table and get back a fully usable, accurately transcribed dataset. Not only did Gemini handle things like missing or smudged letters, it was able to recognize irregularities, maintain table alignment, and reformat everything into a single usable spreadsheet for each PDF – all in just a few minutes. ²

A sample from Gemini’s “model thoughts” shows the difference between traditional OCR vs. a VLM approach to transcription:

I'm now focusing on formatting the data into the "long" format for the CSV, refining the output structure. I will rigorously check OCR outputs, specifically converting "H. M." values to decimal hours, and correcting any potential errors, such as "Little Kock" to "Little Rock."

This isn’t pattern matching; this is using reasoning to translate a historical source into usable data – much like a human researcher would do. ³

In short, Gemini was able to turn this:

…into this:

The technical capacity for accurately transcribing messy historical sources is there, while the usability of these tools is still catching up. My experience with Claude Code – which defaulted to traditional OCR rather than using a VLM approach – shows how uneven the landscape still is. But given the pace of change, it feels like a gap that’s going to close very quickly (if it hasn’t already). ⁴

The Coding Bottleneck

The second bottleneck comes down to the technical skill of coding. Any historian can transcribe a transit table into a spreadsheet; the bottleneck is time and labor, not skill. Building a fully interactive, map-based web visualization that responds to user input and presents temporal shifts in spatial data? Vanishingly few historians have the skill and training to do this. If you are not one of those unicorn historians who knows how to define a CSS hover effect or implement a JavaScript event listener, then up until a few months ago your options were limited. You could find a technical collaborator, get funding to hire a developer, or accept the constraints and limitations of out-of-the-box software like Tableau or ArcGIS StoryMaps.

This changed in late 2025, when Anthropic released a major new update for the model behind its agentic coding system, Claude Code. Prior to this, vibe coding (ie. describing what you want in non-technical language and having an LLM write the code for you) necessitated at least some technical fluency, like knowing how to run Python in the command line. Today, that barrier has all but vanished. Someone with zero coding experience can install Claude or OpenAI’s Codex, write a few prompts, and get back a fully functioning website, visualization, or app a few minutes later.

When I sat down to test out Claude Code with my mail transit tables, I decided I was not going to write a single line of code myself. I wasn’t going to review code and approve it, I wasn’t going to tweak variables until something worked, I wasn’t going to lean on prior experience to debug or catch errors. Instead, I channeled a colleague with zero coding experience. I just described what I wanted, in plain language, and let Claude Code do its thing. This was my very first prompt:

There are two files in this folder: a PDF of mail transit times between major cities in 1882 and the transcribed data from that PDF. I want to use the transcribed data to build some kind of interactive app where a user can quickly see and visualize the data.

After some further back-and-forth to help clarify what I wanted, Claude Code worked for about 5-6 minutes. When it finished, I opened the HTML file it had created and immediately said “oh shit.” Here was a fully working and interactive prototype of that original vision I had fourteen years ago when I first stumbled onto this source:

Initial prototype of “How Fast Was the Mail?” (February 13, 2026)

Watching Claude Code build something like this from scratch felt like sorcery. But I would argue that the iterative capacities of agentic coding - the back-and-forth of refining, adjusting, and experimenting that turns a prototype into something you actually want to show people - is even more significant. Iteration has traditionally come with a lot of friction. If you’re writing the code yourself, iteration means trying to change something, breaking your entire project, and spending hours trying to debug it. If you’re working with a web developer, iteration means sending over your proposed changes, waiting for them to implement your request, realizing it made everything worse, and sheepishly asking them to walk it all back.

Vibe coding removes much of the friction of iteration. From my initial prototype, I could easily add additional transit table data from other years, build out more interactive functionality, and make a wide range of design and layout adjustments and tweaks. If something broke, Claude Code was able to fix it. Here is an example instruction I wrote during this process:

When you click or select a hub, have all cities show up in a scrollable table on the right. Add a hover function so that when you hover over a city on the map it subtly highlights it in the table and vice versa. Something is wonky with St. Louis when you click it, it turns grey not blue.

Above all, I was struck by how Claude Code reduced so many of the pain points of coding by hand. To take one example: a few sessions in, I remembered that the economic historian Jeremy Atack had compiled a dataset of historical railroad lines that would be perfect to add to my visualization. If I had been coding this myself, I would have had to wrestle with coordinate systems and map projections – my personal Achilles heel as a spatial historian that would have left me Googling “WGS 1984 shapefile” for the 1,284th time in my career before closing my laptop in frustration. Instead, I just downloaded Atack’s dataset and told Claude Code to add it to the map. And it did, without a single hiccup. As many others have noted: Claude Code take the drudgery of coding and makes it, well, fun.

Is the code behind my visualization especially elegant or well-structured? Probably not! I’m no web developer, but my hunch is that a gargantuan HTML file with 10,600 lines of code is not going to win any awards. But, for my purposes as a researcher, that doesn’t really matter: it just worked. ⁵

Side Quests

One obvious effect of the rise of agentic AI and reasoning models is that the field of digital history is now open to a much wider swath of researchers than ever before. A lot of the end products of digital history – online exhibits, project websites, maps, charts, network graphs, etc. – that used to require a certain degree of technical fluency can now be built with Generative AI. ⁶

A less obvious effect of these changes is that many more research side quests are now worth pursuing even when you don’t know if they’ll pay off. Side quests are essential to the research process. Even the dead ends are often valuable: a chance for serendipity, learning new skills, reframing research questions, or stumbling into entirely new lines of inquiry. But side quests come with concrete costs. Historians are constantly making judgement calls about which sources to dig into and which to put aside. Ideally, these decisions are driven solely by the research questions themselves; in practice, they involve the kind of calculations I did as a graduate student when I came across that first mail transit table: was the juice worth the squeeze?

And guess what? It wasn’t! As far as I can tell, my snazzy transit visualization shows precisely the kind of unsurprising patterns you’d expect to see: an overall shrinking of informational time-space between 1882 and 1908, lengthy transit lags for the Deep South and patches of the American West, and big leaps in transit gains following the completion of major railroad lines. I could be wrong; maybe something new is hiding in there and some other historian or geographer or communication scholar will find it. ⁷ But at first pass, I didn’t see all that much potential in terms of publishable results.

If I had decided to execute my original vision back in 2012, I would have wasted a whole lot of time going down an analytical dead end. At best, I would have lost several weeks that I could have spent on other parts of my dissertation. At worst, I might have fallen prey to the sunk cost fallacy and tried to shoehorn this source into my dissertation just because I had invested so much time working on it.

Fast-forward to 2026. Using Gemini, I could transcribe that same source in a few minutes, at a cost of $0.34. ⁸ I could then use Claude Code to quickly visualize the data, start looking for patterns, and iterate back and forth depending on what I found. The same analytical dead end ends up costing several hours rather than several weeks. Moreover, even if the macro patterns from this data aren’t revelatory, agentic coding meant that I could use that data to easily build and deploy something that might end up being useful for other researchers: a genealogist curious how long it took a letter to travel between their relatives, a historian trying to gauge how isolated one of their subjects was from a major city, or a stamp collector trying to estimate the value of a particular postal cover.

Breaking some of the traditional bottlenecks to transcription and analysis changes the threshold for whether to pursue a historical research question at all. When the opportunity cost of transcribing a source drops to a few minutes and the cost of a candy bar, the challenge is no longer getting the data – it’s whether the questions you can ask with that data are interesting enough to pursue.

And this is where a different kind of expertise comes into play that has nothing to do with writing code or importing shapefiles: discernment. One feature of LLMs is that they’re eager to give you something. When working with Claude Code, it kept presenting me with what it claimed were interesting patterns in the data. If I were a graduate student, I might have excitedly leapt at these suggestions. But part of becoming an experienced researcher is learning to recognize not just what IS worth digging into but what ISN’T worth pursuing. So when Claude ran a clustering algorithm on cities based on their connectedness and declared “the typology shifts are particularly interesting,” I had the experience to say: “No, they’re not.”

The old bottlenecks were time and technical skill; the new bottlenecks are judgment, discernment, and taste. This is true for many different disciplines, but history seems especially well-positioned to benefit from this shift. Accessing and transcribing archival sources has always been a major sticking point for historical research, and the ongoing decimation of federal grant funding has made that bottleneck even more acute. Vision Language Models don’t just make transcription faster or cheaper - they will (hopefully) expand the kinds of sources we can access and work with.

Meanwhile, coding remains a vanishingly rare skill for historians. For colleagues in the social sciences and hard sciences, agentic coding dramatically speeds up something they could already do. For historians, agentic coding allows us to do things we couldn’t do before. Without training in statistical thinking or computational methods, there’s a risk that we start vibe coding our way to questionable results. But agentic coding in history doesn’t have to mean pumping out a bunch of faulty regression analyses; it can also mean spinning up an interactive web app to explore a set of sources that have been languishing on your computer for the past fourteen years. By breaking the bottlenecks of transcription and coding, these tools free up time and energy for historians to use the skills we actually excel at: close reading, contextual thinking, narrative, and interpretive nuance.

Originally posted: https://cblevins.github.io/posts/bottleneck-side-quests/

Subscribe now

Share

Footnotes

1. For more on VLMs, OCR, and historical transcription, see Jacob Carlson, Tom Bryan, and Melissa Dell, “Efficient OCR for Building a Diverse Digital History” (2024) and Noah Dasanaike’s working paper, “Pre-Trained Language Models as Zero-Shot Tools for Social Science Research” (2026).[↩]

2. As of March 2026, Gemini seems to be the most capable frontier model for computer vision and historical transcription. Although I didn’t do a rigorous analysis of its error rates, I used Claude Code and some spot-checking to identify potential mistakes in the data. This led me to fix around 12-15 incorrectly transcribed values (across nearly 8,000 records).[↩]

3. Take these “model thoughts” with a grain of salt; they’re a somewhat imprecise take on the internal processes unfolding within an LLM as it generates its response.[↩]

4. Lots of historians are working on this - see, for instance, Mark Humphries on transcribing handwritten 18th-century storekeeper ledgers or Colin Greenstreet on transcribing handwritten Ottomon sources.[↩]

5. As Jason Heppler has recently argued in “Vibing Digital History”, the technical debt critique of vibe coding that applies to large software systems doesn’t really hold for small, one-off visualizations like this one.[↩]

6. I agree with Lincoln Mullen, who notes in “Behind, ahead” that technical fluency is still a major advantage when it comes to using these tools. There were several times when my experience coding and working with data helped me write better prompts. However, the gap between a “digital historian” and a “non-digital historian” has closed substantially and will only continue to do so as these tools get better and better.[↩]

7. Have at it! The raw data is available at: https://github.com/cblevins/mail-time/blob/main/data/transit-times-merged.csv.[↩]

8. I didn’t actually pay this cost; I was using Google’s experimental AI Studio, but it helpfully offers a token usage and cost estimate for prompts if you had been using their API. Transcribing each transit table took between 32,000 - 113,000 tokens, with a range of $0.32 on the low end to $1.07 on the high end.[↩]

Digital history, by most conventional measures, has failed—just as the university enters an era defined by artificial intelligence and large-scale data. At the very moment when the transformation of university research seems not only possible but inevitable, the humanities appear, from the outside, to be receding. History in particular risks being seen as increasingly marginal: slow, archival, resistant to scale.

To understand why, it is worth briefly revisiting what digital history has been—and why it has taken the form that it has. Digital history, by most conventional measures, has failed. But in its ruins, it holds expertise that may yet prove decisive.

A Short History of Digital History

For at least half a century, historians have been talking about “digital history” as part of the “digital humanities,” a suite of approaches to thinking about data, the internet, and analysis. Twenty to twenty-five years ago, this produced DH centers at many North American universities. These centers were funded to run short, intensive workshops, to expose faculty and students to computational methods, and to build tools and digital collections.

Many of the digital humanists who participated in these initiatives built museum-style collections of objects online. These collections were often keyword searchable and publicly accessible, but they were not consistently analyzed in ways that produced new forms of historical argument. They were collections—valuable, often transformative in terms of access—but still collections rather than cumulative analytical programs.

Over time, American digital history became closely associated with this model of public-facing collection. Its audience was often K–12 classrooms, and its outputs were frequently websites, teaching materials, and conference presentations rather than new methods articles or large-scale synthetic works. These interventions mattered. They helped normalize longer temporal frames and more inclusive archives within the discipline.

The orientation had real value. It expanded access to archives, diversified the materials available for teaching, and introduced new audiences to historical sources. But it also helps explain why digital history, as a research program, did not scale. And that failure (or apparent failure) now matters in a new way. For the questions that digital history did not fully answer are precisely the questions that define the current moment in artificial intelligence.

As a research program across the university, digital history did not clearly achieve cumulative transformation. It did not visibly reorganize the discipline of History, and one funded project did not consistently lead to another in a compounding way. There was no sustained chain of funding—no first million leading to a second or third grant, no steady expansion of infrastructure, and no clear institutional scaling of the field.

Part of this outcome reflects the structure of the humanities, where pedagogy continues to prioritize small seminars and one-to-one mentorship. At the same time, many of the most innovative investments were made at non-elite but highly innovative institutions (including George Mason, SMU, Emory, Nebraska, and Clemson), where there was comparatively limited access to large graduate training pipelines and rarely (if ever) a pool of university-wide postdocs or funding for visiting researchers of the kind that characterizes elite institutions.

These programs trained a number of outstanding students, many of whom went on to build creative and impactful research agendas. However, those students were not consistently placed into the most influential positions in the discipline, and the methods they developed did not consistently propagate into its institutional core.

Subscribe now

The Underdevelopment of History Departments

Meanwhile, in a reinforcing dynamic, the most prestigious departments—those responsible for a large share of hiring and agenda-setting—continued to underinvest in methods. Methods training was often confined to occasional seminars, adjunct instruction, or co-taught courses rather than embedded into a structured curriculum with introductory sequences, advanced coursework, and graduate training pathways. In many cases, departments treated methodological investment as a trade-off against geographic or thematic coverage, rather than as an expansion of the field’s capacity.

Funding structures reinforced this pattern. The National Science Foundation did not develop a sustained funding stream for historical computation, and the National Endowment for the Humanities primarily offered shorter-term grants that were not designed to build large-scale, durable infrastructure. There were few fellowships or prizes dedicated to supporting scholars developing new computational methods in history.

As a result, a number of pioneering works—such as the first monographs to use GIS or large-scale text mining in historical research—were recognized but not institutionally consolidated. Their authors often remained in precarious roles, and their contributions did not always translate into stable research programs or widely adopted methods.

Individual scholars did tremendous work with digital methods to reframe how we see the geography and language of the Enlightenment, to refine our tools for studying place-based relations and journeys like the Grand Tour, and to show how network analysis might give a scholar new glimpses into the dynamics of a community.

Yet by and large, these advances did not fundamentally restructure the discipline as a whole. Many leading universities continued to treat digital methods as peripheral rather than central, even as broader institutional priorities shifted toward data science, environmental research, and artificial intelligence.

Europe as a Counterexample

In contrast, Europe pursued a different model. At the Luxembourg Centre for Contemporary and Digital History, more than 100 faculty and staff support large-scale research programs, including embedded developers and extensive digitization infrastructure. Projects such as Impresso enable cross-lingual analysis of newspapers at scale. Institutes like Huygens Institute employ computational specialists to support digital editions and natural language processing pipelines, while networks such as DARIAH and CLARIN establish shared standards across countries.

These investments have produced concrete outcomes. Scholars such as Ruben Ros, trained in part within the lab of Pasi Ihalainen, have gone on to produce major dissertations—such as his work at Leiden on parliamentary speech and the history of bureaucracy—that demonstrate how large-scale textual comparison can generate new historical arguments. Researchers trained by Living With Machines similarly moved into faculty positions, extending these methods into new institutional contexts.

Indeed, North America also has expertise to offer. However, it is dispersed across institutions such as Saskatchewan, SMU, George Mason, Clemson, and Waterloo, rather than integrated into the central infrastructures of major research universities.

The difference is not simply scale but continuity. There is digital history expertise in North America, just as there is DH expertise in Europe. But in North America, the necessary steps to institutionalize DH expertise, to ensure the training of the next generation of scholars, to support an ongoing dialogue with Computer Science, have never been taken. The foundations have not been laid; the training programs are generally missing or absorbed into a wider “digital humanities” program that neglects the specific skills of history majors. There are simply no North American programs where the scholar can be sent for a specifically historical training in digital methods with the intent of entering the profession or moving into industry to share historical expertise. We need systems train scholars, produce methods, and sustain research programs across multiple funding cycles.

The time has come for a comparable shift. Not because digital history failed on its own terms, but because the problems it left unresolved—how to evaluate documents at scale, how to build representative archives, how to move between collection and analysis—have now become central to the most powerful technologies of our time.

Share

Subscribe now

Check out some of the work of Living With Machines:

This morning, I set myself the task of using Codex as a research assistant to push me toward completion.

I have nearly 200 pages of semi-finished manuscript on the history of e-commerce, yet within those pages are numerous gaps in evidence.

The manuscript it littered with notes like “find something here about the Teamsters strike of 1997 and the contract that came out of it.” I am sure you know what I mean. Some of these gaps are relatively straightforward archival problems that can be solved with a database search. Less obvious are the places where I am making an argument and I should have supporting numbers but don’t—like the number of Teamster members in 1997 relative to other union members and the total workforce. Yet whether these searches are easy or hard, they still take time. For assistance, I turned to AI.

I asked the AI to read my manuscript for both evidence I knew was missing, like the Teamsters contract, and evidence I did not know was missing, like the number of Teamsters. I asked it to produce a table of such gaps in markdown, a format that is easy for both computers and humans to read.

Read my manuscript and produce a chapter-by-chapter gap analysis in Markdown.

Requirements
- First, read the full manuscript and identify each chapter’s core argument before listing any gaps.
- Do not edit existing files. Create new files only.
- Save output in: <absolute output folder>
- File name: <YYYY-MM-DD-manuscript-gaps.md>

Inputs
- Manuscript path: <absolute path to manuscript>
- Optional chapter map/TOC path: <absolute path>
- Optional existing evidence index: <absolute path>

Output format (required)
For each chapter:
1. Chapter title
2. Argument summary (3-6 bullets)
3. Evidentiary gaps (numbered, specific claims needing support)
4. Gap IDs in format C<chapter>-G<n>
5. Why each gap matters (causal/quantitative/comparative/historical)
6. Evidence needed (data type, geography, timeframe, source type)
7. Priority (High/Medium/Low)

At the end include:
- Master gap table (gap_id, chapter, claim, evidence needed, priority)
- Cross-chapter themes
- “Most critical 10 gaps” section

Quality bar
- Be concrete: identify missing numbers, missing citations, weak causal links, overgeneralizations, and untested counterarguments.
- Distinguish “no evidence” vs “insufficient evidence” vs “outdated evidence.”
- Use manuscript language when possible so I can map gaps back to text quickly.

Then came the tricky part.

For the last month, I have been experimenting with AI agents. Agents carry out tasks, like moving files around, running Python code, or anything else computers do. You can give them a task and they can use other programs to accomplish their goals. Their key feature: they run autonomously until they are “finished” (e.g., completing a task, running for ten hours, looping 37 times, or however you define finished). Think of them as henchmen.

I use ChatGPT’s Codex, but you could just as easily use Claude’s Cowork. These agents run on your computer, which allows them to read and write files directly. Scary? To be sure. You should absolutely use a backup like Google Drive or GitHub.

While I have found many uses for these agents, I wanted to see if I could use them to access my university’s database system. I hate sitting there for hours running searches, clicking through multiple screens, and downloading PDFs. It takes time that would be better spent reading those PDFs, or actually writing my book.

Subscribe now

AI agents can do this for you. And you can run many, many of them at a time.

I will give an example using Codex for ChatGPT, which I prefer because it does not run out of tokens nearly as quickly as Claude.

First, download and install Codex. Then start a new project. ChatGPT will walk you through the setup if needed. You will need to grant it full access, which again feels a bit unsettling, but it is still restricted to your project folder. If you want, you can create a custom set of rules. Just ask Codex to explain how to do that.

You will also need to install a “skill” that allows Codex to use your browser. The one you want is called Playwright Interactive.

Once you have done that, you are ready to go.

First tell Codex which database you want to use. In my case, I used EBSCOhost, which has basically everything. I instructed Codex to read the markdown file of gaps in the manuscript that needed more primary documents. I had to log in to the database, but after that I let it run. While I worked on other things, it pulled and organized hundreds of sources. It created a table that linked each gap to a relevant PDF. Was it perfect? No. Was it working while I did something else? Yes. Did it perform as well as an undergraduate research assistant? Definitely.

For data gaps, I followed a similar process but did not need a university database. Many government websites provide relevant economic data, often accessible through APIs that allow direct download. You may need to register for an API key, but for most government sites this is easy and free. I asked Codex to download the relevant time series based on the gaps in the manuscript. There was some back and forth, but the prompt I ended up with worked reliably. You will need to adapt it to your own project.

Read /Users/louishyman/Library/CloudStorage/GoogleDrive-lhyman@gmail.com/My Drive/2024-2025/E-Commerce/codex/2026-manuscript-national-metrics-insertion-map.md and pull all labor-related gap data into the existing national metrics pipeline.

Requirements:
1) Focus on labor gaps across Introduction, Chapter 1, Chapter 4, Chapter 4b, Chapter 5, Chapter 9.
2) For every labor time series, include explicit context comparators (denominator, peer sector, or cross-country comparator), not standalone numbers.
3) Prioritize free/public APIs first: BLS v2 (registered), FRED, BEA, Census MRTS, ILOSTAT, OLMS, World Bank, USITC where relevant.
4) Extend historical coverage to 1970 whenever available; otherwise use earliest available and record coverage limits.
5) Reuse stored data and avoid duplicate pulls by default. Only pull missing labor series not already in the store.
6) Update outputs:
   - codex/national_metrics_pipeline/national_metrics_api_pull.csv
   - codex/national_metrics_pipeline/metric_coverage_summary.csv
   - codex/national_metrics_pipeline/context_comparison_audit.csv
   - codex/national_metrics_pipeline/national_metrics_graph_report.md
   - codex/national_metrics_pipeline/charts/*
7) Add/refresh charts specifically for:
   - sectoral wages and employment shares
   - sectoral GDP shares for labor context
   - Teamsters + union/non-union denominators
   - productivity vs compensation
8) At the end, report:
   - which labor gaps are now covered,
   - remaining unresolved labor gaps,
   - exact series IDs/endpoints used per gap.
9) Do not ask me questions; run autonomously and make reasonable assumptions.

I even had Codex create rough and ready graphs for each data set along the way. These graphs are not production ready (or even classroom ready), but certainly better than nothing. As always, I will double check all these numbers before I make any meaningful inferences from them.

The easy work is now done, so I can spend my time on the hard work that AI can't do. Between these two approaches, I have assembled a much larger body of documents and data for finishing the book. I still need to find the harder-to-locate primary sources, but now I can spend my time on the hard research problems, not the easy ones. Hours I might have spent pulling PDFs and CSV files I can now spend on thinking and writing, which is much more rewarding than clicking buttons on websites.

Subscribe now

Historians have always done this kind of labor, and so have graduate students, research assistants, and archivists. The work can be tedious, but it produces searchable texts, datasets, and indices that allow us to see patterns that would otherwise remain hidden. At the same time, this labor quietly shapes the boundaries of historical inquiry. The sheer effort required to process a document often determines which sources historians pursue and which they leave aside.

I encountered a document that illustrates this problem while working with historian Philip Herrington. During research at the Henry Ford Archives, Philip came across a guest register from the Hermitage Plantation, just outside Savannah, Georgia. The book records visitors between 1933 and 1935 and runs to 159 pages. Each line contains a date, a visitor’s name, and a hometown, producing thousands of entries written by hundreds of different hands.

The register is historically valuable because it offers a rare window into plantation tourism in the early twentieth century. Visitors arrived from across the United States, particularly from northern cities, and their entries reveal patterns of travel and mobility in the Jim Crow era, when plantation landscapes were increasingly repurposed as heritage sites. But the document also presents the kind of practical difficulty historians know well. The handwriting varies widely, some entries are careful while others are hurried, and ditto marks appear where locations repeat. Ink colors change across the pages, and occasionally several people sign on the same line.

To answer even simple questions about the register—where visitors were coming from, whether particular cities appeared repeatedly, how visitation changed over time—the information had to be structured as data. The traditional approach would be clear enough: transcribe the book, enter each line into a spreadsheet, verify the entries, and standardize the locations. Many historians have undertaken precisely this kind of work. Even at a steady pace, though, the process could easily consume weeks or months of labor, which in practice means that many such documents are never fully transcribed nor eventually made accessible to others.

Artificial intelligence tools offer a different approach, though not the one that gets the most attention. Writers like to talk about AI “transforming” historical research, usually meaning something dramatic about automated interpretation. That overstates the case; an LLM does not interpret the past. What it can do is assist with the mechanical work of recognizing and structuring text—provided the historian teaches it how to behave.

LLMs are not designed to read handwritten guest registers. When I first uploaded pages from the Hermitage register without detailed instructions, the model invented names that weren’t on the page and merged separate entries into single lines. To obtain reliable results, the historian has to provide something the model lacks: instructions grounded in how we actually read these documents.

In practice this means defining the task with the kind of specificity we would use when training a new research assistant. Each page of the register was scanned and uploaded to a model capable of processing images. Rather than asking the system simply to “transcribe the page,” the prompt described the document and its structure in detail. The model was told that it was working with a handwritten guest register and that each row contained a date, a visitor name, and a location. It was instructed not to guess when handwriting was unclear and to flag uncertain entries instead. The output format was specified in advance: a table with the columns Date, Visitor, and Location.

Historians will recognize the logic immediately. We explain what kind of document someone is looking at, identify patterns they should expect to see, warn them about ambiguities, and show them the format in which the results should be recorded. The difference is that the “someone” here processes a page in seconds rather than minutes.

Once those instructions were refined, the model began producing usable output. Each page returned tab-delimited text that could be copied into a transcription platform and reviewed by a human editor. The transcription was not perfect: names were occasionally misread and locations sometimes required correction. But starting from a rough transcription rather than a blank spreadsheet changed the nature of the work. Instead of entering thousands of lines manually, we reviewed and corrected what was already there. The process had shifted from creation to verification, and what might have taken months was compressed considerably.

This leads to a methodological point that historians may initially find uncomfortable. For many research questions, “good enough” data is enough. Historians are trained to value fidelity in transcription, and rightly so. Yet the level of precision required always depends on the question being asked. If the goal is a diplomatic transcription of a manuscript, every mark matters. But if the goal is to analyze visitor origins, travel frequency, and geographic networks, then a structured transcription that is mostly correct can be sufficient, especially when the alternative is no transcription at all.

What keeps this honest is that the historian does the interpretive work. Once the Hermitage register had been transcribed and cleaned, the results became immediately useful. Entries could be sorted by city, mapped geographically, and examined for clusters of visitors from particular regions. Patterns that had been buried in the handwriting of the original ledger emerged clearly once the data had been structured. None of that analysis was performed by the model. It handled the reading; we handled the thinking.

Seen in this light, tools like these do not promise automated history. They promise a reallocation of effort. Tasks that once required weeks of mechanical transcription can now be accelerated, leaving more room for the questions that brought us to the archive in the first place. For the Hermitage register, those questions are just beginning—about who visited, why they came, and what a guest book from a plantation-turned-tourist-site can tell us about how Americans in the 1930s chose to encounter the history of slavery.

Related article: https://doi.org/10.1080/15420353.2026.2614780

One task that the AI can do is shallow reading. Think of the AI as an enthusiastic, but somewhat unlearned, intern. The intern can read the text but not the subtext, so meaning-making is not possible in the way it is for a fully-trained historian. The key point is that AI is not omniscient, but it is fast. And it is cheap. It will not be perfect, nor insightful, but it can still be extremely helpful.

First, the intern can transcribe nearly any text. Yes, even handwriting. Yes, even some bad handwriting. Yes, even Latin in Carolingian script. Most importantly, the intern can usefully categorize the text. Even if the intern doesn’t know who the Marquis de Fancyhat is, it can recognize that this is a proper name, likely a person, possibly French. In computer-speak, this is called “Named Entity Recognition,” which identifies names, places, numbers, organizations, and so on.

Second, the intern can reasonably categorize information on a page. If you ask it to identify property and track property values and addresses, it can do that. If you ask it to geolocate an address, it can do that, though it may be confused by historical differences. Crucially, you can simple say, “Put the key information on this page into reasonable categories.” If it encounters a medical injury, it can place it in a “medical” bucket. If it encounters a legal case, it can place it in a “law” bucket. Will these categories be perfect? No. Will they be approximately as good as a sophomore reading random documents? Yes. Is that better than nothing? Definitely.

The secret sauce in AI-assisted reading is a file format called JSON (pronounced “JASON,” like the Argonauts).

JSON is a “human-readable data format,” which means that you can look at it and understand what is going on. Here is an example based on the letter above.

{
  "file_summary": [
    "Typed letter to the Commissioner of Education",
    "about reading programs in Indian schools",
    "and a request for help preparing a study outline."
  ],

  "transcription": [
    "Dr. William J. Cooper,",
    "  Commissioner of Education,",
    "  Washington, D.C.",
    "",
    "Dear Dr. Cooper:",
    "",
    "Each year the teachers and other instructors in the Indian schools",
    "make a study of some subject of special importance in connection with",
    "Indian education. Usually one or two of the best books available",
    "which deal with the subject chosen are selected to be read by all",
    "employees and to be discussed at Reading Circle meetings which are held",
    "about once each month during the school year. Teachers are encouraged",
    "to read additional books, reports and articles, the subject matter of",
    "which is related to the topic being studied.",
    "",
    "In some instances, there are those who desire to do the reading",
    "under the direction of college or university representatives in order",
    "that degree credits may be earned.",
    "",
    "It is known that much consideration has been given to this subject",
    "during recent years and that individuals and groups have developed a",
    "great fund of literature which may be of value to those who wish to",
    "study the subject. The difficulty is in the selection of the best in",
    "such form as may be comprehensive and yet not too voluminous for the",
    "year’s study. To this end, the cooperation of your Bureau is solicited.",
    "It will be necessary to have the outline completed and ready to send to",
    "the field employees not later than May first.",
    "",
    "Sincerely yours,",
    "",
    "(Signed) Chas. H. Burle[?]",
    "",
    "Commissioner.",
    "",
    "2 EP 26",
    "",
    "Carbon for Indian Office.",
    "",
    "[Right margin handwriting/markings: unclear]",
    "[Stamp: FEB 28 19[??] (year unclear)]"
  ],

  "named_entities": {
    "people": ["Dr. William J. Cooper", "Chas. H. Burle[?]"],
    "place": "Washington, D.C.",
    "organizations": ["Indian Office", "Reading Circle"],
    "date": "FEB 28 19[??]"
  },

  "categories": ["government", "education"],

  "document_type": "Typed letter (carbon copy)"
}

Unprocessed text has value, but so does structured text, especially when you are a busy historian sorting through thousands or millions of documents.

At the simplest level, this structure allows the historian to perform keyword searches by categories of information, such as “medical,” rather than searching for “ankle injuries,” “broken leg,” or “fever.” At a slightly more advanced level, it allows the historian to take this structured data and feed it into a spreadsheet (with AI help) to sort and identify documents or patterns. At an even more sophisticated level, though still accessible with AI assistance, JSON files can be stored in a database for more complex analysis. I will write about this in a later post, but you can also ask an AI to teach you how to do it right now.

For this particular letter, you can see how AI is both strong and weak, but always useful. It correctly captures names, places, and organizations, and does a solid job of transcription, even marking where the text is illegible. If this document were important, you would want to read it yourself. At the same time, it misses the date, which is obscured under a mess of stamps. Even more crucially, it misses how this document fits within the long, complex history of Indian education as a mechanism of control and assimilation. AI excels at transcribing bureaucratic records, but is not particularly adept at understanding them.

The larger point is this: you used to need an army of researchers to index and summarize documents. Now you don’t. You used to need a large grant. Now you don’t. And this kind of categorization is extremely helpful in making sense of the past.

AI may be a mediocre reader. But it is fast and cheap, and that makes it powerful.

If you would like to try this for yourself, the code is linked. Just download, get an API key, and structure away!

Share Computational History

Historians’ data is different from that of sociologists or economists in two key ways, however. First, it is much, much larger—it encompasses the past, full stop. Historians often don’t think of their evidence this way, but the entirety of human history is in a meaningful sense fair game for making historical claims. It is entirely common to criticize colleagues for not thinking carefully about excluded evidence—making a claim about the history of the modern Middle East, for example, by drawing on English-language but not Arabic-language sources, or asserting a truth about early modern politics without taking into account the limitations of state archives, or arguing about the causes of a conflict only using evidence produced by the victors.

Second, we draw on observational evidence, but we cannot run our study twice. That is, unlike an experimental setup in psychology, history is an n-of-1 experiment; we might muse about counterfactuals or what-ifs, but ultimately the Reformation happened once, so we cannot test a theory about what might have been different had a particular event not happened or particular person not been born. Likewise, we only have the evidence that remains to assert what it felt like to live on December 1, 1045 or even December 1, 2025. We cannot simply run the study again with new questions or new variables.

Historians navigate these limitations on their evidence in part by making assertions about the representativeness and persuasiveness of the evidence they use—whether a particular statement is typical or unusual; whether an event was unexpected or expected; whether a purported cause is plausible. This is, after all, the normal way academic historians begin our articles, asserting the relevant scope of the problem, laying out what we know already, and explaining how the evidence we draw on will enable us to say something new.

This background is why historians need to engage more directly with statistical thinking. After all, statisticians have spent decades now thinking about what it means to have only a “sample” of evidence from a “population,” or how observational data differs from experimental data, or how to be precise about what is “typical” or “rare” or “unexpected”. We normally associate statistical processing of historical data with the digital or computational humanities, niche fields for colleagues particularly skilled in computers and quantification. Or approaches that have been relegated to economic history after 1970s-era debates over cliometrics and Time on the Cross. But that’s wrong. Every claim we make, about the importance of a particular book or event or person; about the causes of change or resistance to change; about the meaning of an abstract concept in the lives of everyday people, requires us to make assertions about the nature of our evidence as a sample of the population of evidence we could have used. And knowledge of statistical ideas helps to make those assertions more precise.

Take a few basic ideas. Samples, in statistical terms, can be large or small, can be randomly chosen or carefully selected. These differences can, in turn, make big differences in what statisticians can infer about the sample. If you randomly select a sample, then the presumption is that every other sample is comparable in a more meaningful sense than if you hand select a sample. Likewise, statisticians distinguish between different ways of “reducing” evidence to a representative example, for example, the mode of a data set (the data point that occurs the most) compared to the median (the middle data point); and have measures of “spread” to give a sense of how close the median piece of evidence is, for example, to the most dissimilar pieces of evidence.

Historians, implicitly, make similar arguments when they assert that a particular passage is typical for a written source, and then describe how different the rest of the source really is. Or when a historian carefully dissects three court cases to show how a particular legal system works but then asserts that the three cases can stand in for all of the rest. Historians may not quantify such assertions, but we do distinguish between an idea that appears a lot and one that is, for example, “middle-of-the-road,” or between a treatise with a wide distribution of ideas and one that is narrowly focused.

Statisticians have also formalized how to interpret “time series” – showing change in some measure over time—or “natural experiments,” when one can compare data before and after some event and make inferences about the effects of the event on that basis. Historians, of course, make arguments along these lines as well. We may argue that the way something is talked about changes gradually over time, for example, or perhaps that the meaning of an idea changes quite quickly and dramatically after some important rupture, like a political revolution. This, implicitly, requires being able to specify and measure (even if qualitatively) what changes and what does not change over time.

Similarly, statisticians have been working for well over a century to formalize when two associated things might be said to have a causal relationship. Epidemiologists, in particular, have carefully formalized how associations must be strong, specific, and consistent with other evidence of cause and effect to be deemed causal. It is crucial to rule out alternative hypotheses and confounding variables that may complicate a supposed causal relationship. Sometimes for statisticians these arguments are made using a model, like that of a regression equation, which makes mathematical assumptions about variables in a formalized effort to combine them and make predictions. Part of this process is to think carefully about when it is ok to eliminate a variable from the equation, what kinds of effects are significant and which are insignificant.

Historians, too, talk about influences, about necessary or sufficient cause, but often do so without any formalization. We talk about contingent and structural causes, but often without thinking carefully about the implicit model of causation we have in mind. We wouldn’t, of course, expect many historical arguments to be expressible as a mathematical equation, but we do ask why one causal claim is more convincing than others.

Most of us remember with bleary eyes, or perhaps some degree of recollected dread or boredom, the weeks of Statistics 101 when ideas like medians or regressions were presented. It is not that we need to use the vocabulary of statistics in writing our papers, or that we would expect to formulate a model when making claims about causation. But we are as historians making claims about evidence. Even if our historical argument involves closely reading one source as standing in for a particular understanding of a movement’s ideas at one point of time—i.e., if we rely on singularities—we implicitly are claiming that everything else is irrelevant or of less importance, and statistical ideas would help us think through how to make these claims more precise.

And for those of us who are wading through thousands of documents, or hundreds of interconnected people, formal mathematical ideas may of course be especially relevant. But that is, and is likely to remain, the exception. The point, for most of us, is that history remains at the intersection of humanistic and social scientific modes of inquiry and even if you find your own way of arguing firmly on the humanities’ side of things, we benefit from knowing enough of the social scientific side to make clear why our arguments are precise and persuasive.

Recently I was working with a 900-page probate of John McDonogh, an enslaver and landlord in New Orleans who died in 1850. The probate was expensive to produce. Every page was cleanly laid out in a grid, written in an impressive secretary hand. And though I can read such old-fashioned script, doing so is not the same as reading print. Even if it were in print, keeping track of 900 pages of entries and calculations would be taxing. Simply reading the document would take weeks. Copying out the entries for verification in a spreadsheet would be herculean. Even with an ample research budget, subcontracting the work to undergraduate RAs would consume months.

I had a very basic, but important question: how much of his wealth was in enslaved people compared to his other property?

To get that answer would have taken months of work. So much work, in fact, that I probably would never have done it.

In the era of AI, a very different approach is possible. You can write cheap code. You can get the answers that you want.

By cheap code, I mean hyper-specific, one-off code written for a singular purpose—like making sense of a 900-page probate. Traditional digital humanities tools required either general-purpose software too rigid for the variation in historical documents, or years of custom development. Only human brains could handle the idiosyncrasies. But today, with AI, you don’t need general-purpose software for reading probates. You can write software to read just this document.

Here is what I actually did. First I scanned the probate with a CZUR scanner—about an hour’s work. Then I used the ChatGPT API to perform OCR and extract the well-structured pages into well-structured JSON. This step required real attention to prompt engineering: I wanted both raw and corrected OCR, French originals alongside English translations, and special attention to the categories I cared about—land, buildings, enslaved people, financial assets. For each entry I asked for names, value, location, and a best-guess latitude and longitude. I used the batch API to keep costs down. The whole run cost about what you’d pay for lunch.

Once I had the JSON, I turned to Claude. I showed it sample files and asked it to write a script that extracted everything into a spreadsheet. That alone was extraordinarily useful for getting a handle on thousands of entries. Next I asked it to verify the arithmetic: on every page, a running total recorded the estate’s cumulative value, and I wanted to check every page’s actual summed entries against that figure. There were minor discrepancies but nothing significant. (And I did this same exercise in Notebook LLM with some inaccurate, non-reproducible results. When counting, always use offline code!)

From there it was straightforward to generate summary statistics—mean, median, range—and visualizations of how the estate’s value was distributed across asset classes. I could see at a glance exactly how much of McDonogh’s estate was held in financial instruments, in land, in buildings, and crucially, in people.

The people enslaved by John McDonogh emerged from the record with clarity. Ages. Names. Occupations. Locations. Every individual could be found and examined. New stories could be told of their lives. This is, of course, the moral center of the project: a slaveowner’s meticulous accounting of human property makes it possible to recover some part of their humanity. At the very least, their names.

Writing this software took about thirty minutes. Iterating on what I wanted—a graph of age vs value vs gender —took another half hour. The code was not elegant, not generalizable, not meant to be shared or preserved. It was cheap code, and that was precisely the point. You could run it and get reproducible results, which is fantastic, but you could not use with another probate.

The deeper methodological point deserves to be stated clearly. This approach is reproducible but not reuseable. In the pre-AI days, software code was both reproducible and reuseable, and had high costs. For many documents, especially bureaucratic documents, cheap code is not a compromise—it is the only realistic path to analysis.

For the first time, I could calculate the complete asset mix of a prominent New Orleans slaveowner: what he held in land, in buildings, in financial instruments, and in human beings. I could do it in an afternoon. The 900-page probate that would once have demanded so many months of my time that I would never do it, gave me the answer in a couple hours.

The question is not whether to write the scripts, it is which documents we should analyze first.

Code: GitHub folder
Method + audit logic: AUDIT_LOGIC.md
Repository: computational_history

Subscribe now