Methods in Digital History II

Wed · Aug 20 — Introductions and syllabus overview · R in VS Code review · R vs Python overview

Readings, tutorials, or videos (to be completed before class)

• GitHub CoPilot in VS Code Documentation
• If you haven’t already, please download and install the following:
  • Download Python. To check to see if its already installed on your machine type which python3. Use the commands above to verify your installation worked. If you don’t have python installed, you can install it using homebrew (Mac OS only): brew install python3. Linux users can use their package manager to install python. Windows users should use the official installer from python.org.
  • Download (or update) R
  • Download VS Code
  • Sign up for GitHub Education which allows you access to the GitHub Copilot extension for VS Code for free.

Mon · Aug 25 — Database design and relationships

Readings, tutorials, or videos (to be completed before class)

• Read through Geek for Geeks, SQL Tutorial
• DeBarros, Practical Guide to SQL, Ch 1-8
• Install VS Code Extensions
  • Search for: SQLite by alexcvzz
  • Search for: SQLite Viewer by Florian Klampfer

Tech Table mini-assignment · Build a schema for a simple relational database

• Draft a schema for a relational database based on your own research and primary sources (preferred) or use one of the datasets in the DMData package. Note that if you are using a dataset from Digital Methods Data, you’ll need to load the dataset and then write it to disk in R first. If you are using a very large dataset, consider taking a random sample to make the dataset a bit more manageable for experimentation (50-100 rows).
• First, draft what you think an ideal set up would look like.
• Describe it to CoPilot and work to refine your set up and build a SQLite3 database.
• Ask CoPilot to create a Data Model Summary and Schema Diagram and write them to a ReadMe file.
• Be prepared to discuss in class on Wednesday. Please make a reasonable start on this assignment and we’ll troubleshoot and discuss if necessary in class.
• Keep notes about how you use CoPilot to refine your setup and post a 1 paragraph overview along with you tech table questions to slack.

Wed · Aug 27 — Tech Table

Due this week

• Wed · Aug 27 — Meaningful start on schema for relational database
• Wed · Aug 27 — Questions and paragraph overview of relational model (see above) to Slack by 10am

Mon · Sep 1 — No Class – Labor Day

Wed · Sep 3 — Advanced SQL queries for historical analysis

Readings, tutorials, or videos (to be completed before class)

• DeBarros, Practical Guide to SQL, Ch 9-20

Tech Table mini-assignment · Build SQL queries

• Using the relational database you built last week (Week 2), practice writing more complex SQL queries to explore your historical data in new ways.
• Write 3-5 SQL queries that go beyond basic SELECT statements. Choose from these techniques:
  • using JOIN queries that connect multiple tables, Aggregate functions (COUNT, MAX, MIN, AVG) with GROUP BY, Filtering with conditions (WHERE clauses with AND/OR), or Sorting and limiting results (ORDER BY, LIMIT)
• Post each SQL statement to slack and describe what question you were trying to answer. Note any trouble you had.
• Since there is no tech table this week due to Labor Day, this tech table assignment and related questions are due Sep 10th.

Due this week

• Wed · Sep 10 — SQL queries mini-assignment and related questions (no Tech Table this week due to Labor Day)

Mon · Sep 8 — Flask web interface for database management

Readings, tutorials, or videos (to be completed before class)

• Miguel Grinberg, The Flask Mega-Tutorial pts 1-4
• Real Python, Flask Project Tutorial
• Flask Documentation

Wed · Sep 10 — Tech Table: SQL and Flask review/troubleshooting

Project · Database Project & Analysis

• Create a relational database for your historical research project using SQLite with a minimum of 30 entries, at least one junction/lookup table, and demonstrate its utility by writing at least one query that addresses a specific historical question relevant to your research.
• Write a 2-4 page paper documenting your database structure, explaining the historical context and research questions that informed your design, and analyzing the results of your queries.
• Create a GitHub repository with clear documentation and code.

Due this week

• Wed · Sep 10 — Meaningful start on a Flask Interface
• Wed · Sep 10 — Questions to Slack by 10am
• Sun · Sep 21 — Database Project & Analysis, by 11:59pm

Mon · Sep 15 — OCR with Tesseract

Readings, tutorials, or videos (to be completed before class)

• “Tesseract,” Wikipedia Entry.
• “Using the Tesseract OCR Engine in R,” CRAN Vignette.
• Documentation for “Pytesseract”

Wed · Sep 17 — Tech Table

Due this week

• Wed · Sep 17 — Questions to Slack by 10am

Mon · Sep 22 — OpenAI/Claude Vision API for OCR and transcription of historical documents

Readings, tutorials, or videos (to be completed before class)

• Read: OpenAI GPT-4 Vision
• Read: Eileen Chen, Exploring OCR Tools with Two 19th Century Documents
• Mark Humphries, Practical AI for Historians (And the Arts): Part 1

Wed · Sep 24 — Tech Table

Due this week

• Wed · Sep 24 — Questions to Slack by 10am

Mon · Sep 29 — No Class - Work on AI OCR and Handwriting Recognition

Wed · Oct 1 — Tech Table

Project · OCR + AI Transcription Project & Analysis (20%)

• Using 5-10 pages of historical sources related to your research, create processing pipelines using both Tesseract OCR and AI-powered transcription services.
• In a 2-4 page analysis, compare and contrast these methods’ effectiveness with your specific source materials, discuss limitations unique to your documents, and propose strategies for improving transcription quality.
• You must obtain approval for your sources and AI prompts before beginning, document all code in a GitHub repository, and test at least two different AI models.

Due this week

• Wed · Oct 1 — Questions to Slack by 10am
• Sun · Oct 12 — OCR + AI Transcription Project & Analysis, by 11:59pm

Mon · Oct 6 — Computer vision & facial recognition

Readings, tutorials, or videos (to be completed before class)

• Taylor Arnold and Lauren Tilton, Distant Viewing: Computational Exploration of Digital Images (2023)
• Sarah Oberbichler, “Large-Scale Research with Historical Newspapers: A Turning Point through Generative AI”

Wed · Oct 8 — Tech Table

Due this week

• Wed · Oct 8 — Questions to Slack by 10am

Mon · Oct 13 — No Class – Fall Break

Wed · Oct 15 — Named entity recognition

Readings, tutorials, or videos (to be completed before class)

• Cameron Blevins, “Space, Nation, and the Triumph of Region: A View of the World from Houston”, Journal of American History, Volume 101, Issue 1, June 2014, Pages 122–147, https://doi.org/10.1093/jahist/jau184
• Clément Besnier and William Mattingly, “Named-Entity Dataset for Medieval Latin, Middle High German and Old Norse” in Journal of Open Humanities Data, (2021).
• Torsten Hiltmann, Martin Droge et. al, “NER4all or Context is All You Need: Using LLMs for low-effort, high-performance NER on historical texts. A humanities informed approach,” arxiv.org

Mon · Oct 20 — Methods for tracking text reuse · N-grams and shingling

Readings, tutorials, or videos (to be completed before class)

• Lincoln Mullen, “Detecting Text Reuse in Nineteenth-Century Legal Documents,” March 11, 2015.
• Josh Catalano, “Digitally Analyzing the Uneven Ground: Language Borrowing Among Indian Treaties”, Current Research in Digital History, 2018.
• David Smith, Ryan Cordell, and Elizabeth Maddock Dillon, “Infectious Texts: Modeling Text Reuse in Nineteenth-Century Newspapers,” published in the Proceedings of the Workshop on Big Humanities (IEEE Computer Society Press, 2013).

Wed · Oct 22 — Tech Table

Due this week

• Wed · Oct 22 — Questions to Slack by 10am

Mon · Oct 27 — Sentence transformers for semantic similarity and text reuse detection (catch up from last week)

Wed · Oct 29 — Tech Table

Due this week

• Wed · Oct 29 — Questions to Slack by 10am

Mon · Nov 3 — Neural language models with BERT

Readings, tutorials, or videos (to be completed before class)

• Ted Underwood, “Do Humanists Need BERT?”, July 15, 2019.
• Ted Underwood, “Mapping the latent spaces of culture: To understand why neural language models are dangerous (and fascinating), we need to approach them as models of culture.”, October 21, 2021.
• Transformer Models, Wikipedia
• BERT, Wikipedia

Wed · Nov 5 — Temporal word embeddings and semantic change

Readings, tutorials, or videos (to be completed before class)

• Sandeep Soni, Lauren F. Klein, Jacob Eisenstein, “Abolitionist Networks: Modeling Language Change in Nineteenth-Century Activist Newspapers,” Journal of Cultural Analytics 6, no. 1 (2021).
• William L. Hamilton, Jure Leskovec, Dan Jurafsky, “Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change”, arXiv.org

Mon · Nov 10 — Tech Table

Project · Computer Vision or Advanced Text Analysis Project & Analysis (20%)

• Select either computer vision or one advanced text analysis method (NER, text reuse detection, topic modeling, or vector alignment) and apply it to your historical sources.
• Write a 2-4 page analysis discussing your chosen method, the historical questions you investigated, your findings, and a critical assessment of this approach’s effectiveness for your research area.
• You must propose your method choice for approval and maintain appropriate dataset sizes for meaningful analysis.

Wed · Nov 12 — Shiny

Readings, tutorials, or videos (to be completed before class)

• Shiny Basics Tutorial
• Look at the gallery of examples on the Shiny Website. Be sure to view the code and the app.
• Hadley Wickham, Mastering Shiny, Introduction and Ch 1-6

Due this week

• Mon · Nov 10 — Questions to Slack by 10am
• Sun · Nov 16 — Computer Vision or Advanced Text Analysis Project & Analysis, by 11:59pm

Mon · Nov 17 — Tech Table

Wed · Nov 19 — Flask web applications with Plotly visualizations

Readings, tutorials, or videos (to be completed before class)

• Explore Plotly Open Source Graphing Library for Python

Due this week

• Mon · Nov 17 — Questions to Slack by 10am

Mon · Nov 24 — No class - work on your final visualization project.

Wed · Nov 26 — Thanksgiving Break – No Class

Mon · Dec 1 — Flask styling, layout, and hosting options

Readings, tutorials, or videos (to be completed before class)

• TBD.

Wed · Dec 3 — Tech Table

Project · Visualization Project & Analysis (20%)

• Design and build either a Shiny (R) or Flask with Plotly (Python) web application that serves as a research tool for historical inquiry, whether visualizing analytical results, providing database interfaces, or offering other research functionality.
• Deploy your application online and write a 2-4 page paper introducing your visualization, explaining its research applications, demonstrating how it advances historical understanding through digital methods, and justifying your choice of framework (Shiny vs. Flask) based on your project goals and technical considerations.
• You must discuss your concept for approval before development and include user documentation.

Due this week

• Wed · Dec 3 — Questions to Slack by 10am
• Tue · Dec 9 — Visualization Project & Analysis, by 7pm

Tue · Dec 9, 7-9pm — Final Exam Period

Showcase of Data Visualization Apps and Projects.

Due this week

• Sun · Dec 14 — Final Reflection, by 11:59pm