Local Voice AI Engineering

What you'll learn

Voice is the most natural interface there is, and the least forgiving. A chat assistant that misreads a word shows the mistake on screen; a voice assistant that mishears "five" as "nine" sends a robot to the wrong aisle. Building voice systems that work, on local hardware and with privacy intact, takes a working understanding of the whole chain: how sound becomes numbers, how speech is found and recognized, how errors are measured, how fast a reply can be, how speech is synthesized, and how an agent should act on what it heard.

This course builds that chain from first principles. Students implement, in NumPy, WAV input and output, short-time spectra and Whisper-style log-mel features, an energy voice-activity detector, a source-filter speech synthesizer with pitch and formant analysis, a dynamic-time-warping word recognizer with wake-word evaluation, word error rate with alignment and intervals, a streaming endpointer and latency budget, a text-to-speech front end, and a voice agent with confirmation. The speech comes from the synthesizer, so every lab runs offline on a laptop CPU and reproduces the instructor's numbers exactly. Each module then shows the production tools doing the same jobs: Whisper and whisper.cpp, faster-whisper, Silero VAD, openWakeWord, Piper and Kokoro.

Learning outcomes

By the end of the course, you can:

1. Explain sampling, quantization and aliasing, read and write PCM audio, and mix speech and noise at a controlled signal-to-noise ratio.
2. Compute short-time spectra, mel filter banks, log-mel features (as Whisper does) and MFCCs, and say why each step exists.
3. Build and evaluate a voice activity detector, and explain the trade-off between missed speech and false alarms.
4. Describe speech with the source-filter model, and measure pitch and formants.
5. Build a template word recognizer, and evaluate a wake word by false-accept and false-reject rates.
6. Measure speech recognition correctly: word error rate, normalization, corpus versus per-utterance rates, intervals and paired comparisons.
7. Design a recognition pipeline that avoids the documented failure modes of large models: hallucination on silence, chunk-boundary errors, unsegmented noise.
8. Budget the latency of a voice turn, from endpointing to the first audio of the reply, and design for streaming.
9. Build a text-to-speech front end and choose between local neural TTS engines.
10. Design a voice agent that acts safely on imperfect recognition, and handle consent, privacy and retention for voice data.

Who it's for

• Software engineers adding voice input or output to a product, a kiosk, a vehicle or an industrial system
• Teams who must run speech locally, for privacy, latency, cost or offline operation
• Technical leads who must decide whether a voice interface is reliable enough to act on, and prove it

Prerequisites

Comfortable Python and basic NumPy, the command line, and secondary-school trigonometry (sine waves) and logarithms. No signal-processing or machine-learning background is required. RNK-LLM-301 (local model serving) and RNK-AGT-401 (agents) are helpful for Module 10 but not required.

Equipment and software

See the setup guide. Python 3.12+, NumPy, Git and a text editor. A media player to listen to the WAV files the labs write is recommended. The optional at-scale tools (whisper.cpp or faster-whisper, Silero VAD, Piper) are needed only for the production exercises.

The full setup guide, with verification steps, is in your learning materials when you enroll.

Format and study time

42 guided hours (10 modules × ~4 h + capstone clinic) plus ~12 h capstone build. Instructor-led cohort, 5-day intensive, or self-paced with office hours.

Each scheduled class below states its own format, every meeting date and time, and its time zone.

What a seat includes

Curriculum

1. 01
   Sound as data

   4 h · Lab: WAV files, quantization noise at three bit depths, aliasing and its cure
2. 02
   Spectra and features

   4 h · Lab: Whisper-style log-mel features; do features describe the word or the speaker?
3. 03
   Voice activity detection

   4 h · Lab: An energy VAD measured from clean audio to 0 dB SNR; hangover and threshold trade-offs
4. 04
   Speech production and analysis

   4 h · Lab: Vowels synthesized for two voices, then their pitch and formants recovered
5. 05
   Recognizing words

   4 h · Lab: A DTW recognizer against noise, normalization and speaker mismatch; a wake-word DET table
6. 06
   Measuring speech recognition

   4 h · Lab: WER alignment, the normalization trap, corpus WER and a paired bootstrap
7. 07
   Recognition pipelines

   4 h · Lab: Rejection, VAD gating and chunking measured; hallucinations on silence counted
8. 08
   Streaming and latency

   4 h · Lab: Endpointing timeouts, a latency budget for batch vs streaming, real-time factor
9. 09
   Speech synthesis

   4 h · Lab: A TTS front end, prosody measured, intelligibility by round trip
10. 10
    Voice agents, privacy and deployment

    4 h · Lab: Task outcomes under recognition error, by policy; a retention plan
11. ★
    Capstone clinic & demo day

    2 h · Lab: Capstone project

Assessment and completion

• Pass: ≥ 70% overall, with the capstone ≥ 70% and at least 8 of 10 labs submitted
• Merit: ≥ 80% overall
• Distinction: ≥ 90% overall, and the capstone's evaluation shows a measured improvement over its baseline, with intervals

Simply opening a lesson or a video never counts as completing an assessment. Completion is recorded only when the published requirements are met and your instructor has confirmed the result.

Course policies

• Late work: −10% per day, up to 3 days; after that, by arrangement. One free 48-hour extension per student.
• Collaboration: discuss freely; write your own code and reports. Capstone teams of two document who did what.
• AI assistants: allowed for coding, with disclosure in each report. You must be able to explain everything you submit. Fabricating results is academic misconduct.
• Recordings and consent: the labs use synthetic speech only. If your capstone records real people, you must have their informed consent, record only what you need, store it securely, and delete it when the course ends; recording a person without consent may be illegal where you live. Never clone or imitate a real person's voice.
• Compute and hardware: the course runs on a CPU and needs no microphone. The optional "at scale" exercises need a few GB of disk for models; a GPU helps but is not required.
• Accessibility: materials are searchable PDFs; their Markdown sources are available on request. Contact the instructor for accommodations; voice technology is itself an accessibility tool, and examples from students who use it are welcome.
• Attendance (A/B): ≥ 80% required for certification; missed sessions can be made up by submitting the lab with a 1-page reflection.

Upcoming classes