Map how any website is built, from one URL

The problem

Before you redesign a website, or learn from someone else's, you need to know how it works: what it is built on, how its pages connect, and which paths a visitor takes to buy, sign up or get in touch. Finding that out by hand means clicking through dozens of pages, viewing source and taking notes, and the result is hard to share. I wanted a tool that takes one URL and returns a clear blueprint, with diagrams, that a technical or non-technical reader can follow and check.

Constraints

• It had to be polite to other people's sites: respect robots.txt and crawl delays, stay on the same site, and never log in or submit a form.
• It had to work without AI, so the results are repeatable, with an agent on top to explain them.
• Many modern sites render in JavaScript, so plain HTML fetching would miss content and API calls.
• Tests had to run offline, with no network.

What I did

Engineering

I built a Python pipeline in five stages: crawl, extract, fingerprint, analyze and report.

• Crawler. A breadth-first crawler reads robots.txt and the sitemap, keeps to the site's scope (a different port counts as out of scope) and waits between requests. An optional Playwright mode renders JavaScript pages and records the live API calls they make.
• Extractor. Each page is classified by type, with its forms, calls to action, navigation and API endpoints pulled out. Shared header and footer text is ignored so it doesn't skew the page type.
• Tech fingerprinting. 105 signatures match headers, cookies, scripts and generator tags to name frameworks, hosting, CMS, analytics, payments and more, each placed on an architecture layer.
• Analyzer. It builds the link graph and sections, detects journeys such as purchase, sign-up, lead capture and search, scores how complete each one is, and lists findings like broken pages, missing security headers, insecure forms and SEO or accessibility gaps. It skips header findings when a site couldn't be reached.
• Reports. Every run writes blueprint.json, a Markdown report, an interactive HTML report and separate Mermaid diagrams for architecture, site map, navigation and each workflow. Steps the crawl didn't reach are drawn dashed.

A fixture site and 13 tests cover the pipeline offline.

AI

I wrote a resident Claude Code agent and a /blueprint <url> command. The agent picks crawl limits, switches to JavaScript rendering when a site looks like a single-page app, then reads the output and explains it: what the site is for, how data flows, each journey step by step, and the top recommendations. Its rules ask it to keep what it observed apart from what it inferred, and to label a result as partial when the network blocks the crawl.

Outcome

• For this website's own research, a script ran the agent across 43 benchmark sites in five groups, at most 15 pages each with a one-second delay. All 43 crawls succeeded, producing 548 crawled pages and 182 Mermaid diagrams. Those blueprints fed the site plan you are reading now.
• I also used it on pitch-stage client work for ZAD ZEAL. The agent crawled 113 public pages of their current website (from 173 sitemap URLs), and its blueprint, together with real-browser tests on desktop and mobile, became the audit behind a redesign proposal. It found, for example, that 103 of the 113 pages had no meta description. The pitch has not been signed.
• The 13 offline tests pass.
• It is an internal tool. It has not been packaged for other users, and accuracy against a hand audit has not been measured.

What I'd change next time

I'd write a small hand-checked answer key for a few sites early on, so I could measure how often the tech fingerprints and journey detection are right. I'd also make the crawl resumable, since large sites with JavaScript rendering take a long time and a dropped connection means starting again.