Over the past several months I extracted several hundred thousand biographical statements from two Victorian reference works, the Colonial Office List and the India Office List, printed almost every year from the 1860s into the 1960s.
The promise of true interoperability in the all-too-often fractured edifice of historical knowledge is certainly exciting. I will be linking my datasets to Wikidata in the future before publishing them. I also have to say that your 'Two Services, One Empire' visualisation is simply awesome; I must have watched it 10 times! I'd be very interested to hear more about the AI pipeline that I assume is powering this major project in the background. One question that has lived with me for some months now, from the inimitable Javier Cha, is: "Can we use AI not only to achieve greater breadth but also greater interpretative depth?" Is automation pushing us to macroscopic perspectives?
Thanks for this thoughtful comment Jacob. The project was possible with the high quality OCR from Chandra 2 and Infinity Parser 2. It was then a Claude Code and Qwen pipeline where I used Claude to build a pipeline using an open model to parse the abbreviated bios into structured data. Building the visualization proved very useful to help identify duplicate officials. Once I had all the work figured out from the COL, it took 24 hours to reOCR and extract the IOL data. I need to decide whether I want to publish a methods paper based on the pipeline or just post it to Working Papers in Critical Search.
The second part of the question is more difficult. I started this project to test whether I could automate the extraction of a useful historical knowledge graph at scale. I didn’t start with a historiographical question. But watching the visualizations and exploring the data suggests there’s a lot here. So I need to figure out next steps. Clearly automation makes scale possible, but we need to learn how to ask and answer the right questions.
The promise of true interoperability in the all-too-often fractured edifice of historical knowledge is certainly exciting. I will be linking my datasets to Wikidata in the future before publishing them. I also have to say that your 'Two Services, One Empire' visualisation is simply awesome; I must have watched it 10 times! I'd be very interested to hear more about the AI pipeline that I assume is powering this major project in the background. One question that has lived with me for some months now, from the inimitable Javier Cha, is: "Can we use AI not only to achieve greater breadth but also greater interpretative depth?" Is automation pushing us to macroscopic perspectives?
Thanks for this thoughtful comment Jacob. The project was possible with the high quality OCR from Chandra 2 and Infinity Parser 2. It was then a Claude Code and Qwen pipeline where I used Claude to build a pipeline using an open model to parse the abbreviated bios into structured data. Building the visualization proved very useful to help identify duplicate officials. Once I had all the work figured out from the COL, it took 24 hours to reOCR and extract the IOL data. I need to decide whether I want to publish a methods paper based on the pipeline or just post it to Working Papers in Critical Search.
The second part of the question is more difficult. I started this project to test whether I could automate the extraction of a useful historical knowledge graph at scale. I didn’t start with a historiographical question. But watching the visualizations and exploring the data suggests there’s a lot here. So I need to figure out next steps. Clearly automation makes scale possible, but we need to learn how to ask and answer the right questions.