Private AI on Your Own Hardware with llama.cpp, Ollama, and Python

Read before you buy: the PDF here is a free sample — the first 20 pages of the 420-page book (payments, sales and reviews on this demo site are sample data). Get the full e-book at thenexbooks on Gumroad.

Use AI without sending your confidential data to someone else's servers.

Law firms, clinics, consultancies, and engineering offices all face the same problem: AI tools are useful, but client and patient data cannot simply be pasted into a cloud service. Open models offer another way. You can download a capable model, run it on a laptop, a workstation, or a server in your own office, and keep every prompt and every document inside your own network.

This book shows you how to do it properly. It starts with the arithmetic that decides what you can run at all, works through the main local runtimes, and then builds a complete private assistant in Python, from the first model call to a secure, monitored, multi-user office server. Every trade-off is explained honestly: what smaller models get wrong, what hardware really costs in memory, and why "local" is not automatically "secure".

What's inside

• Why and when to run models locally, and an honest look at the costs: hardware, administration, and smaller models
• How language models work from an operator's point of view: tokens, context windows, parameters, prefill and decoding, sampling
• Hardware sizing you can do on paper: weights memory, KV cache, context and concurrency, and the memory-bandwidth limit on speed
• Choosing an open model: model families and sizes, model cards, revisions, provenance, and reading licenses
• Model formats and quantization: safetensors, GGUF, Q8_0 and Q4_0 block math, K-quants, GPTQ, AWQ, and measuring the quality cost yourself
• llama.cpp and llama-server, Ollama and Modelfiles, and how to evaluate desktop chat applications for professional use
• Running open models with Hugging Face transformers on an ordinary CPU, and tuning generation and performance
• Prompting small models: chat templates, system prompts that work, structured JSON output, and token budgets
• OpenAI-compatible APIs and your own FastAPI wrapper with API keys, streaming, queues, and tests
• Local embeddings, a private document index, and document Q&A that shows its sources and knows when to say "I can't find that"
• Tool calling with local models, where the model asks and your code decides
• Drafting and summarizing long documents safely, with human review built in
• Testing and evaluating with golden questions, and a simple private web page for staff
• The office AI server: hardware plan, services, multi-user serving, rate limits, TLS, threat modeling, privacy-preserving audit logs, monitoring, model updates with evaluation gates and rollback, and governance

One running example, start to finish. You help Ashcombe Legal, a fictional small law firm whose AI policy forbids sending client information to cloud chatbots, build AshDesk: a private assistant that answers questions about firm policies and client documents with sources, looks up matter details through tools, drafts text for attorney review, and keeps an audit log that never stores the prompt text. It starts on one laptop and ends on a shared office server for the whole firm.

Verified and honest. The Python code was run on an ordinary CPU-only computer with a small open model, and the printed results come from those runs. Commands for llama.cpp, Ollama, and GPU servers are shown as documented usage for recent versions, with their output clearly labeled as illustrative. The book contains no invented benchmarks or speed claims; instead, it teaches you how to estimate and measure performance on your own hardware. Every chapter includes Tips, Notes, Warnings, exercises, and Key Takeaways, and the appendices give you a cheat sheet, templates and reference tables, a production checklist, and a glossary.

Format: PDF, 420 pages (A4), linked table of contents and bookmarks. English. First Edition, 2026. Published by thenexbooks. This book is independent and is not affiliated with, sponsored by, or endorsed by any model provider or software project mentioned in it. Each open model is subject to its own license.

Table of Contents

Part I — Foundations

• Ch 1 · Why Run LLMs Locally: Privacy, Control, Cost, and Trade-offs
• Ch 2 · How LLMs Work for Operators: Tokens, Context, Parameters, and Generation
• Ch 3 · Sizing Your Hardware: RAM, VRAM, Bandwidth, and the Memory Math
• Ch 4 · Choosing an Open Model: Families, Sizes, Model Cards, and Licenses
• Ch 5 · Model Formats and Quantization: Safetensors, GGUF, GPTQ, and AWQ

Part II — Local Runtimes

• Ch 6 · llama.cpp: Building, Running, and Serving GGUF Models
• Ch 7 · Ollama: Pulling, Running, and Customizing Models with Modelfiles
• Ch 8 · Desktop Apps and GUIs: Local Chat Without the Command Line
• Ch 9 · Transformers on the CPU: Running Open Models from Python
• Ch 10 · Tuning Generation and Performance: Sampling, Context, Threads, and Speed

Part III — Building on Local Models

• Ch 11 · Prompting Small Local Models: Chat Templates, System Prompts, and Structured Output
• Ch 12 · OpenAI-Compatible APIs: One Client, Many Local Backends
• Ch 13 · Building Your Own API: An OpenAI-Compatible FastAPI Wrapper
• Ch 14 · Streaming, Concurrency, and Queues
• Ch 15 · Local Embeddings and a Private Document Index

Part IV — Private Applications

• Ch 16 · Private Document Q&A: Answers with Sources from Firm Documents
• Ch 17 · Tool Calling with Local Models: Letting the Model Ask, Letting Code Decide
• Ch 18 · Drafting and Summarizing Safely: Long Documents, Templates, and Human Review
• Ch 19 · Testing and Evaluating a Local Assistant
• Ch 20 · A Web Interface for Staff: A Simple, Private Chat Page

Part V — Office Deployment

• Ch 21 · The Office AI Server: Hardware, Operating System, and Installation Plan
• Ch 22 · Multi-User Serving: Authentication, Rate Limits, and Fair Queues
• Ch 23 · Security and Privacy: Threat Model, Network Isolation, Logging, and Retention
• Ch 24 · Monitoring and Operations: Health Checks, Metrics, Logs, and Capacity

Part VI — Production and Beyond

• Ch 25 · Updating Models Safely: Versioning, Evaluation Gates, and Rollback
• Ch 26 · Licensing, Compliance, and Governance for Open Models
• Ch 27 · Adapters and Fine-Tuning: When a Local Model Needs More
• Ch 28 · Case Study: AshDesk Goes Live at Ashcombe Legal

Appendices
A · Local LLM Cheat Sheet
B · Templates and Reference
C · Production Checklist
D · Glossary

Book details