Corpus Engineering for AI

What you'll learn

Every language model is, first, a dataset. The research record is consistent: data quality, deduplication and filtering decisions move model quality as much as architecture does. That was the lesson of C4, The Pile, RefinedWeb, FineWeb, Dolma and DCLM. Yet most practitioners have never built a corpus end to end, measured a filter's precision, tuned a deduplication threshold, or written a datasheet.

This course teaches corpus engineering as a measurable discipline. It runs from corpus-linguistics foundations to a production pipeline:

• Sourcing: Common Crawl and WARC files, dumps, APIs, polite crawling, robots.txt and AI opt-outs, provenance.
• Extraction & normalization: HTML boilerplate removal, Unicode, mojibake repair.
• Filtering: language identification; the Gopher, C4 and FineWeb heuristics, re-implemented and scored against ground truth; model-based quality classifiers.
• Deduplication: exact hashing, MinHash-LSH (with the math), SimHash.
• Safety & privacy: PII and secret redaction, toxicity, bias audits of filters.
• Tokenization: train BPE tokenizers; fertility, compression and the multilingual "token tax".
• Instruction & synthetic data: SFT formats, Self-Instruct-style generation, validation, preference pairs, model-collapse risks.
• Evaluation: decontamination, data mixtures, epoch budgets, ablations.
• Governance & scale: licensing, copyright and opt-out law (EU DSM, EU AI Act), GDPR, datasheets, versioning, scaling with sharded pipelines.

Students work on a synthetic web crawl with ground-truth labels for every page (good articles, spam, boilerplate, duplicates, foreign-language pages, PII, benchmark leaks), so every pipeline decision gets a precision/recall score, not an opinion. The skills then transfer to real Common Crawl data.

Learning outcomes

By the end of the course, you can:

1. Describe the corpus types used in AI (pretraining, SFT, preference, evaluation, retrieval) and the landmark corpora's design decisions.
2. Acquire web data responsibly: parse WARC/WET, honor robots.txt and AI opt-outs, record provenance.
3. Extract and normalize text from HTML, repairing encoding damage without destroying information.
4. Implement and measure language ID and heuristic quality filters (Gopher, C4, FineWeb) with precision/recall against labels.
5. Deduplicate at document and near-duplicate level with MinHash-LSH, and choose parameters from the S-curve math and measurements.
6. Build model-based quality classifiers, audit them for bias, and redact PII and secrets.
7. Train and evaluate tokenizers (fertility, compression, multilingual fairness) and compute corpus statistics.
8. Create instruction and preference datasets with validation, deduplication, diversity metrics and leakage-free splits.
9. Detect benchmark contamination, and plan data mixtures and ablations with epoch budgets.
10. Govern a corpus: licensing and opt-outs, privacy, datasheets, versioning, and scalable, resumable pipelines.

Who it's for

• ML/data engineers preparing pretraining, fine-tuning or RAG corpora
• Researchers and analysts who need defensible, documented datasets
• Technical leads accountable for data governance and compliance in AI projects

Prerequisites

Intermediate Python and basic command line. No ML background required.

Equipment and software

Python 3.10+, NumPy; optional tokenizers, trafilatura, ftfy, fasttext.

The full setup guide, with verification steps, is in your learning materials when you enroll.

Format and study time

40 guided hours (10 modules × 4 h) plus ~12 h capstone. Instructor-led cohort, 5-day intensive, or self-paced with office hours.

Each scheduled class below states its own format, every meeting date and time, and its time zone.

What a seat includes

Curriculum

1. 01
   What a corpus is: from corpus linguistics to LLM data

   4 h · Lab: A statistical profile of the crawl vs real text
2. 02
   Sourcing & acquisition: WARC, crawling, robots, provenance

   4 h · Lab: A polite fetcher + provenance manifest + sourcing plan
3. 03
   Extraction & normalization

   4 h · Lab: Extraction audit + mojibake repair report
4. 04
   Language ID & heuristic quality filtering

   4 h · Lab: Per-rule precision/recall + tuned thresholds
5. 05
   Deduplication at scale

   4 h · Lab: Tuned MinHash-LSH with measured trade-offs
6. 06
   Model-based filtering, PII & safety

   4 h · Lab: A quality classifier + bias audit + PII report
7. 07
   Tokenization & corpus statistics

   4 h · Lab: Trained tokenizers + token-tax analysis
8. 08
   Instruction, synthetic & preference data

   4 h · Lab: An audited SFT dataset + preference pairs
9. 09
   Evaluation, contamination & data mixtures

   4 h · Lab: A contamination report + mixture/ablation plan
10. 10
    Governance, documentation & scaling

    4 h · Lab: A full pipeline run + datasheet + governance memo
11. ★
    Capstone clinic & demo day

    Lab: Capstone project

Assessment and completion

• Labs 40% · Quizzes 15% · Final exam 15% · Capstone 30%. Pass ≥ 70% (capstone ≥ 70, ≥ 8 labs) · Merit ≥ 80% · Distinction ≥ 90% (and a capstone ablation showing a measured effect of a data decision). Certificates: certificate/README.pdf.

Simply opening a lesson or a video never counts as completing an assessment. Completion is recorded only when the published requirements are met and your instructor has confirmed the result.

Course policies

• Late work: −10%/day up to 3 days; one free 48-hour extension.
• Data ethics: crawl only what you're allowed to crawl; honor robots.txt and opt-outs; never share personal data found during labs; report anything illegal to the instructor without opening or redistributing it.
• AI assistants: allowed with disclosure; you must explain your work. Fabricated measurements are misconduct.
• Accessibility: searchable PDF materials (Markdown sources available on request); accommodations on request.

Upcoming classes