Core ML, Vision, Natural Language, Speech, Create ML, and Apple's On-Device Foundation Models

Read before you buy: the PDF here is a free sample — the first 20 pages of the 458-page book (payments, sales and reviews on this demo site are sample data). Get the full e-book at thenexbooks on Gumroad.

Build ML features that run on the iPhone — private, offline, and honest about what they know.

Sending user photos, voices, and notes to a server is not the only way to add intelligence to an app. Modern iPhones run machine learning models right where the data lives, and Apple's frameworks — Core ML, Vision, Natural Language, Speech, Sound Analysis, and, in iOS 26, Foundation Models — let you build features that work without a network and without uploading anything.

This book is written for iOS developers who already know Swift and SwiftUI. It explains the machine learning ideas you need without heavy math, then shows exactly how to put them to work in a real app. It also covers what tutorials skip: deciding when a model is good enough, choosing confidence thresholds, designing a "not sure" state, keeping inference fast and battery-friendly under Swift 6 concurrency, testing ML code, protecting privacy, and shipping and updating models after release.

What's inside

• On-device ML concepts for app developers: training versus inference, datasets and labels, confidence, softmax, overfitting, and transfer learning
• Core ML in depth: model formats, generated classes, inputs and outputs, MLModelConfiguration, compute units, async loading, and Swift 6 concurrency patterns
• Vision: classifying photos with your own model, orientation and coordinates, text recognition, QR codes, faces, body pose, animals, feature prints, and the newer Swift-only Vision API
• A real-time camera pipeline with AVFoundation: frame throttling, stable live predictions, rotation, and thermal awareness
• Create ML: planning and preparing data, training an image classifier plus sound and text models, and avoiding data leakage
• Evaluation that leads to better products: confusion matrices, precision, recall, F1, top-3 accuracy, thresholds, coverage, and error analysis
• Converting PyTorch models with coremltools, and optimizing them with quantization, palettization, and pruning — then re-evaluating
• Natural Language, custom text classifiers with NLModel, on-device speech recognition, and bird-song recognition with Sound Analysis
• Combining photo and sound evidence responsibly
• Apple's Foundation Models framework (iOS 26+, Apple Intelligence-capable devices): availability checks, sessions, prompts, guided generation with @Generable, tool calling, and responsible design with fallbacks
• Production: privacy by design, testing ML features, measuring performance and energy with Instruments, and downloading, verifying, and versioning models

One running example, start to finish. You build Featherfield, a fictional bird-identification field guide app. It identifies backyard birds from photos and the live camera, honestly says when it is not sure, listens for bird songs, reads trail markers, turns spoken field notes into text and tags them, blurs people out of shared photos, writes an on-device journal summary of your day on supported devices, and updates its model safely after release. Everything runs on the device.

Current and honest. The code uses the Swift 6 language mode with Xcode 26-era tooling and an iOS 18 deployment target; iOS 26 features are clearly marked and gated with availability checks. The book contains no invented benchmarks, latency figures, or accuracy claims — the evaluation numbers belong to a fictional test set, are labeled as illustrative, and are internally consistent so you can recompute them. Every chapter includes Swift (and, where relevant, Python) code, Tips, Notes, Warnings, exercises, and Key Takeaways, and the appendices give you a cheat sheet, templates and reference, a production checklist, and a glossary.

Format: PDF, 458 pages (A4), linked table of contents and bookmarks. English. First Edition, 2026. Published by thenexbooks. Requires a Mac with Xcode to build the examples; Foundation Models features need iOS 26 on an Apple Intelligence-capable device. This book is independent and is not affiliated with, sponsored by, or endorsed by Apple Inc.

Table of Contents

Part I — Foundations of On-Device ML

• Ch 1 · On-Device Intelligence: What Machine Learning Can Do Inside Your App
• Ch 2 · ML Concepts for App Developers: Models, Training, Inference, and Evaluation
• Ch 3 · Meet Featherfield: The Project, the Data Model, and the ML Layer
• Ch 4 · Core ML Basics: Models, MLModel, Inputs, Outputs, and Generated Classes
• Ch 5 · Running Predictions Well: Configuration, Compute Units, Async Loading, and Concurrency

Part II — Seeing with Vision

• Ch 6 · Vision Fundamentals: Requests, Handlers, Observations, and Orientation
• Ch 7 · Identifying Birds in Photos with Vision and Core ML
• Ch 8 · Reading the World: Text Recognition and Barcode Scanning
• Ch 9 · Faces, Bodies, and Animals: Detection and Photo Privacy
• Ch 10 · The Real-Time Camera Pipeline: AVFoundation Capture and Live Inference

Part III — Building Your Own Models

• Ch 11 · Create ML Essentials: The App, Templates, and Preparing Data
• Ch 12 · Training the Featherfield Bird Classifier
• Ch 13 · Evaluating Models: Metrics, Confusion Matrices, Thresholds, and Error Analysis
• Ch 14 · Converting Models with coremltools: From PyTorch to Core ML
• Ch 15 · Model Optimization: Quantization, Palettization, Pruning, and Trade-offs

Part IV — Language, Speech, and Sound

• Ch 16 · The Natural Language Framework: Language, Tokens, Tags, and Embeddings
• Ch 17 · Custom Text Models: Classifying Field Notes with Create ML and NLModel
• Ch 18 · Speech Recognition: Dictating Field Notes On Device
• Ch 19 · Sound Analysis: Recognizing Bird Songs
• Ch 20 · Combining Signals: Photo, Song, and Context Together

Part V — Apple's On-Device Foundation Models

• Ch 21 · The Foundation Models Framework: Availability, Sessions, and Prompts
• Ch 22 · Guided Generation: Structured Output with @Generable
• Ch 23 · Tool Calling: Letting the Model Use Featherfield's Data
• Ch 24 · Designing Responsible Generative Features: Instructions, Safety, Fallbacks, and Testing

Part VI — Production ML

• Ch 25 · Privacy by Design for Machine Learning Features
• Ch 26 · Testing and Evaluating ML Features in Your App
• Ch 27 · Performance and Battery: Measuring and Tuning Inference
• Ch 28 · Shipping and Updating Models: Bundling, Downloading, Versioning, and the Featherfield Case Study

Appendices
A · On-Device ML Cheat Sheet
B · Templates and Reference
C · Production Checklist
D · Glossary

Book details