Requests, httpx, Beautiful Soup, APIs, and Responsible, Reliable Data Pipelines

Read before you buy: the PDF here is a free sample — the first 20 pages of the 518-page book (payments, sales and reviews on this demo site are sample data). Get the full e-book at thenexbooks on Gumroad.

Turn web pages into clean, trustworthy data — without being the scraper every site owner dreads.

Sooner or later almost every analyst and developer needs data that lives on a web page: prices to track, a catalog to monitor, public records nobody has packaged as a download. Copy and paste works for ten rows; it does not work for ten thousand rows every Monday. Python has superb tools for the job, but good scraping is more than tools. A scraper that ignores robots.txt, hammers a server, collects personal data carelessly, or silently breaks when a page changes is a liability.

This book teaches both halves: the techniques and the responsibility. You learn how HTTP really behaves, how to fetch and parse pages robustly, how to prefer APIs and structured data when they exist, and how to turn a one-off script into a tested, scheduled pipeline that knows exactly what changed since last week.

What's inside

• When to scrape and when not to: APIs, exports, feeds, and the decision ladder
• Responsible collection: robots.txt (RFC 9309) with Python's robotparser, terms of use, rate limits, identifying your client, and handling personal data (general guidance, not legal advice)
• HTTP for scrapers: status codes, headers, cookies, redirects, timeouts, and caching
• requests and httpx: sessions, headers, timeouts, retries with exponential backoff and Retry-After, and conditional requests with ETags
• Beautiful Soup, CSS selectors, XPath and lxml — with robust selector strategies that survive small layout changes
• Crawling: pagination, sitemaps, link following, URL normalization, and avoiding crawl traps
• Structured data in pages: JSON-LD, embedded JSON, and HTML tables
• Forms and logins with CSRF tokens on accounts you own, and JSON APIs as the better alternative
• JavaScript-rendered pages: how to detect them, find the data behind them, and when browser automation (Playwright) is worth it
• Async scraping with httpx and aiohttp, semaphores, token buckets, and polite request budgets
• Cleaning prices, dates, units, whitespace, and encodings; validating records with dataclasses and quality checks
• Storing results in CSV, JSON Lines, and SQLite with upserts and price history; a first look with pandas and DuckDB
• Production: offline tests with saved fixtures, logging, scheduling, monitoring, change detection, and a complete pipeline

Practice safely on a demo shop that runs on your own computer. The book includes Cobblestone Market, a small Python program that serves a realistic online shop at 127.0.0.1: thirty products across five categories, pagination, reviews, a login area with a CSRF-protected form, a JSON API, a JavaScript-rendered deals page, robots.txt, a sitemap, a real rate limit that answers 429, an occasionally failing endpoint for retry practice, and a "one week later" mode for change detection. Every example in the book runs against it — never against real websites — and every output shown is real.

One running example, start to finish. You work for Hollowbrook Analytics, a fictional research firm with permission to collect Cobblestone Market's catalog for a weekly price-and-stock report. Chapter by chapter you build cobblescrape, a polite, tested scraper package, and finish with a scheduled pipeline that crawls the shop, cleans and validates the data, stores it in SQLite with price history, and reports exactly what changed. The complete reference package and its tests are included.

Current and honest. Examples use Python 3.12+ and recent versions of requests, httpx, aiohttp, Beautiful Soup, lxml, pandas, DuckDB, and pytest. The book contains no invented benchmarks or statistics, and it never teaches evasion: no CAPTCHA solving, no fingerprint spoofing, no proxy rotation to dodge blocks. Every chapter includes runnable code, real output, Tips, Notes, Warnings, exercises, and Key Takeaways, and the appendices give you a cheat sheet, templates, a production checklist, and a glossary.

Format: PDF, 518 pages (A4), linked table of contents and bookmarks, plus the Cobblestone Market demo site and the cobblescrape reference package (Python source, tests, and README) as a separate download. English. First Edition, 2026. Published by thenexbooks. This book is independent and is not affiliated with, sponsored by, or endorsed by the Python Software Foundation or the maintainers of any library mentioned. Nothing in it is legal advice.

Table of Contents

Part I — Foundations

• Ch 1 · Web Scraping Today: What It Is, When to Use It, and the Cobblestone Project
• Ch 2 · Scraping Responsibly: robots.txt, Terms, Rate Limits, and Personal Data
• Ch 3 · Setting Up: Python, Packages, and the Cobblestone Market Demo Site
• Ch 4 · HTTP for Scrapers: Requests, Responses, Status Codes, and Headers
• Ch 5 · HTML and the DOM: Reading Pages the Way a Scraper Does

Part II — Fetching Pages

• Ch 6 · Your First Scraper with requests
• Ch 7 · Sessions, Headers, Cookies, and Redirects
• Ch 8 · httpx: A Modern HTTP Client, Timeouts, and Connection Reuse
• Ch 9 · Errors, Retries, and Backoff: Handling 429s, 5xx, and Flaky Networks
• Ch 10 · Caching and Conditional Requests: ETags, Last-Modified, and Local Caches

Part III — Parsing and Extracting

• Ch 11 · Beautiful Soup Fundamentals: Navigating and Searching the Tree
• Ch 12 · CSS Selectors in Depth: Extracting Product Pages Reliably
• Ch 13 · XPath and lxml: Precise Extraction at Speed
• Ch 14 · Pagination and Crawling: Following Links, Sitemaps, and Avoiding Traps
• Ch 15 · Structured Data in Pages: JSON-LD, Embedded JSON, and HTML Tables

Part IV — Beyond Static Pages

• Ch 16 · Forms and Logins: CSRF Tokens, Sessions, and Accounts You Own
• Ch 17 · JSON APIs vs Scraping: Finding and Using the API First
• Ch 18 · JavaScript-Rendered Pages: Detecting Them, Finding the Data, and Browser Automation
• Ch 19 · Async Scraping with httpx and asyncio: Concurrency with Politeness
• Ch 20 · aiohttp and Rate-Limiting Patterns: Semaphores, Token Buckets, and Budgets

Part V — From Raw Data to Clean Datasets

• Ch 21 · Cleaning Scraped Data: Text, Prices, Dates, Units, and Encodings
• Ch 22 · Modeling and Validating Records: Dataclasses, Types, and Quality Checks
• Ch 23 · Storing Results: CSV, JSON Lines, and SQLite
• Ch 24 · Exploring Scraped Data with pandas and DuckDB

Part VI — Production Pipelines

• Ch 25 · Testing Scrapers: Fixtures, Offline Tests, and Catching Layout Changes
• Ch 26 · Scheduling, Logging, and Monitoring Scrapers
• Ch 27 · Change Detection and Price History
• Ch 28 · Project: The Complete Cobblestone Data Pipeline

Appendices
A · Web Scraping Cheat Sheet
B · Templates and Reference
C · Production Checklist
D · Glossary

Book details