Local LLM Engineering with Ollama & llama.cpp

What you'll learn

Running a language model on your own hardware has gone from a hobby to a serious engineering option. You get data that never leaves your network, predictable cost, offline operation, and full control over the model. But "ollama run" is where most people stop, and production needs much more: knowing why a model is slow, why it ran out of memory, which quantization to pick, how to get valid JSON every time, how to serve 20 concurrent users, and how to put an adapter you trained into production.

This course teaches the whole local inference stack from first principles:

• What the hardware is actually doing: tokens, the KV cache, memory math, and why decode speed is set by memory bandwidth.
• Ollama in depth: CLI, server configuration, the REST API, structured outputs, tool calling, Modelfiles and templates.
• llama.cpp in depth: building for your GPU, the GGUF format byte by byte, quantization (k-quants, i-quants, imatrix) measured with perplexity and KL divergence, sampling and grammars, and performance tuning (offload, flash attention, KV-cache quantization, speculative decoding, parallel slots).
• Production: OpenAI-compatible serving, TLS and authentication, systemd and Docker, metrics, load testing, and security (exposed inference servers are a real and recurring incident class).
• Customization: LoRA fine-tuning, merging, converting to GGUF, and deploying with Ollama or llama-server.

Every lab runs on a laptop. A GPU makes it faster but is not required, and a bundled mock server lets all client code run without any model.

Learning outcomes

By the end of the course, you can:

1. Estimate memory and speed for any model/quantization/context/hardware combination, and explain the estimates (KV cache, memory bandwidth).
2. Operate Ollama as a service: configure context, parallelism, keep-alive, model storage and the KV cache type, and troubleshoot GPU offload.
3. Integrate local models through the Ollama and OpenAI-compatible APIs, with streaming, structured outputs, tool calling and embeddings.
4. Package models with Modelfiles: system prompts, parameters, chat templates, few-shot messages, adapters.
5. Build llama.cpp for CPU/CUDA/Metal/Vulkan/ROCm and use its tools (llama-cli, llama-server, llama-bench, llama-quantize, llama-perplexity).
6. Explain the GGUF format, convert Hugging Face models, and choose quantizations based on measured quality (perplexity, KL divergence) and size.
7. Control generation with sampling parameters and grammar or JSON-schema constraints.
8. Tune performance with GPU offload, flash attention, KV-cache quantization, batching, threads and speculative decoding, and benchmark correctly.
9. Deploy a secure, observable, load-tested local inference service.
10. Fine-tune a model with LoRA and ship it through GGUF to Ollama or llama-server.

Who it's for

• Developers and DevOps/platform engineers who need to run LLMs on-premises or on edge devices
• Engineers building products on local models (assistants, extraction, coding tools, RAG)
• Technical leads evaluating "local vs API" with real numbers

Prerequisites

Comfortable on the command line; basic Python; basic understanding of HTTP APIs. No ML background needed.

Equipment and software

The full setup guide, with verification steps, is in your learning materials when you enroll.

Format and study time

40 guided hours (10 modules × 4 h) plus ~12 h capstone. Instructor-led cohort, 5-day intensive, or self-paced with office hours.

Each scheduled class below states its own format, every meeting date and time, and its time zone.

What a seat includes

Curriculum

1. 01
   How LLMs run locally: tokens, KV cache, memory & speed

   4 h · Lab: Memory/speed predictions for 3 models, checked against reality
2. 02
   Ollama fundamentals: CLI, server & configuration

   4 h · Lab: A configured Ollama service + a troubleshooting log
3. 03
   The Ollama API: streaming, JSON, tools, embeddings

   4 h · Lab: A tool-calling, schema-validated client
4. 04
   Modelfiles, templates & packaging

   4 h · Lab: Three custom models, including an imported GGUF
5. 05
   llama.cpp from source

   4 h · Lab: A GPU-accelerated build + a llama-server smoke test
6. 06
   GGUF & quantization, measured

   4 h · Lab: A quantization ladder with perplexity/KLD vs size
7. 07
   Sampling, grammars & structured output

   4 h · Lab: Sampling experiments + 100% valid JSON extraction
8. 08
   Performance engineering

   4 h · Lab: A tuning report with before/after benchmarks
9. 09
   Serving in production

   4 h · Lab: A TLS + auth deployment, load-tested, with metrics
10. 10
    Fine-tune to deploy (LoRA → GGUF → Ollama)

    4 h · Lab: A fine-tuned model served locally, evaluated against its base
11. ★
    Capstone clinic & demo day

    Lab: Capstone project

Assessment and completion

Simply opening a lesson or a video never counts as completing an assessment. Completion is recorded only when the published requirements are met and your instructor has confirmed the result.

Course policies

• Late work: −10%/day for up to 3 days; one free 48-hour extension.
• Hardware fairness: grading never rewards owning faster hardware. Labs are graded on method, correctness and explanation; absolute speeds don't matter.
• AI assistants: allowed with disclosure; you must be able to explain your work. Fabricated measurements are misconduct.
• Licenses: download only models whose licenses you accept; note them in reports.
• Safety: don't expose lab servers to the internet; follow Module 2/9 security rules even in labs.
• Accessibility: searchable PDF materials (Markdown sources available on request); accommodations on request.

Upcoming classes