How it works

1.0Run every site at once

Sites are independent: different hosts, different sitemaps, different proxies. Each gets its own transport context, so a four-site sweep discovers all four together and pays roughly one site's wall clock instead of four. Each site keeps its own block in the log so concurrent output still reads one site at a time.

Up to 4 sites at a time when proxies are in play, 6 when every site allows direct access.

1.1Negotiate a transport, once per site

A browser cannot read another site's robots.txt unless that site sends CORS headers, and most do not. Rather than pay a fallback chain on every request, the app probes once per site and remembers the winner.

All six transports are raced against /robots.txt at once rather than tried in turn, so a site behind a slow proxy no longer pays six sequential timeouts before discovery starts. Direct fetch gets a 1.2 second head start and, if a proxy answers first, another 0.8 seconds of grace: direct is faster and it is the only transport that does not hand every audited URL to a third party, so it wins whenever it works at all.

The probe keeps the body. robots.txt is both the probe target and the first thing discovery needs, so fetching it twice was a wasted round trip per site.

1.1bHedge, and let proxies earn their place

Sequential fallback means the slowest proxy sets the wall clock: you wait a full timeout to learn nothing, then start over somewhere else. Instead a second transport is started while the first is still outstanding and whichever answers first wins, so a stalled proxy costs the hedge delay rather than the timeout.

The probe winner starts as the only proxy in the rotation. An alternate joins it by actually answering, and three failures drops it for the rest of that site. Sitemap concurrency then follows how many proxies are genuinely working, at 3 requests per working proxy: spreading load helps only while they all work, and rotating onto one that is stalling just moves the stall around.

Proxy latency and failure rates are scored, shared across every site in the run, and kept in localStorage between runs, so the ordering starts from whatever was actually fast last time instead of relearning it each run.

1.1cYour own proxy

The public proxies share datacentre address ranges, and a site behind a WAF blocks those ranges wholesale. The proxy connects, the site answers 403, and no amount of retrying changes it. This is the one discovery failure a browser-only tool cannot engineer around, because the block is on the address, not on the request.

A proxy on an address you control is the way out, and it is also better for privacy: the audited URLs go to your server rather than to a stranger’s. Paste its template into Your own CORS proxy under “Sampling and run options”, using {url} where the encoded target goes. It is tried straight after direct fetch, ahead of the public pool, and it works even with public proxies switched off.

A Cloudflare Worker is the quickest one to stand up. Create a Worker, paste this, deploy, and use https://<your-worker>.workers.dev/?url={url}:

export default {
  async fetch(request) {
    const target = new URL(request.url).searchParams.get('url');
    if (!target) return new Response('missing url', { status: 400 });

    // Only proxy for your own page, and only plain GETs.
    const cors = {
      'Access-Control-Allow-Origin': 'https://pagespeed.jakelabate.com',
      'Access-Control-Allow-Methods': 'GET, OPTIONS'
    };
    if (request.method === 'OPTIONS') return new Response(null, { headers: cors });
    if (request.method !== 'GET') return new Response('GET only', { status: 405 });

    const upstream = await fetch(target, {
      headers: { 'User-Agent': 'PageSpeedSampler/1.0 (+sitemap reader)' },
      redirect: 'follow'
    });
    return new Response(upstream.body, {
      status: upstream.status,
      headers: { ...cors, 'Content-Type': upstream.headers.get('content-type') || 'text/plain' }
    });
  }
};

1.2Find the sitemap

robots.txt is checked first for a Sitemap: directive, which is authoritative when present. If there is none, 22 common paths are probed in parallel, 10 at a time through a proxy and 14 direct:

/sitemap.xml /sitemap_index.xml /sitemap-index.xml /sitemap/sitemap.xml /wp-sitemap.xml /sitemap-0.xml /sitemap1.xml /sitemaps/sitemap.xml /sitemap/index.xml /sitemap.php /sitemap.txt /sitemapindex.xml /page-sitemap.xml /post-sitemap.xml /product-sitemap.xml /sitemap/ /sitemap-pages.xml /sitemap_pages.xml /xmlsitemap.xml /sitemap/sitemap-index.xml /api/sitemap.xml /server-sitemap.xml

Where several hit, an index beats a leaf urlset, then the shallowest path wins.

1.3Walk it

A <sitemapindex> is walked breadth-first into its children, to a depth of five and a default ceiling of 30 child sitemaps. Plain-text sitemaps are parsed too. Gzipped sitemaps are skipped, because a browser cannot decompress them from fetch.

The parallel pass fails fast at 16 seconds. Any child that timed out is then swept again through different proxies, four at a time, because a batch of timeouts usually means the proxy was saturated rather than the files being absent. Recovered children rejoin the inventory.

A wall-clock budget (90 seconds by default, adjustable) bounds the whole walk, checked per sitemap rather than per depth level so a single index handing back twelve children cannot run past it. Whatever was collected when the budget expires is used.

1.4Detect the stack

This runs inside the site's own lane rather than after every site has finished, since nothing downstream of discovery waits on it. The homepage is fetched on the same transport and fingerprinted: 20 platforms, 8 image CDNs, the generator meta tag, and on WordPress the installed plugin and theme slugs read out of asset paths. This is a side input; it changes what phase 5 says, not what gets measured.

2.1Normalise

Every URL is resolved absolute, filtered to the same host ignoring www, stripped of query and fragment, de-trailing-slashed, de-duplicated, and filtered against asset extensions. A 20,000 URL ceiling applies.

2.2Group into collections

Collection is the first path segment. /blog/x and /blog/y become blog. Root-level one-off pages such as /about and /contact collapse into a single top-level pages group rather than becoming twenty groups of one. The homepage gets its own group.

2.3Tag page roles

17 patterns tag pages by role: home, about, contact, services, pricing, products, blog, faq, team, careers, locations, work, reviews, privacy, terms, resources. Roles are what make like-for-like comparison possible across sites that share no URL structure.

2.4Pair collections across sites

Only when more than one site is measured. Two sites rarely name the same thing the same way, so pairings are seeded from exact names plus a 13-cluster synonym table with plural stripping: blog / news / articles / insights, products / shop / store / collections, case-studies / work / portfolio / projects, and so on. A pairing is proposed only when at least two sites match.

Measuring every page of a 4,000-page site is neither affordable nor necessary. The sample is built in a fixed order so that the things a comparison depends on can never be squeezed out by the budget:

Within a collection the pick is an even, deterministic spread across the sorted list rather than the first N, so a blog sample is not twenty consecutive posts from one week. Caps are 10 per collection and 100 per site, both adjustable. The homepage is always included.

4.1Build the job list

One job per selected URL per strategy. 100 URLs across mobile and desktop is 200 jobs.

4.2Run them concurrently, and discover the real ceiling

The 240-queries-per-minute quota is not the binding constraint. Each call holds a Lighthouse run on Google's side for 10 to 30 seconds, and one project's share of that pool is far narrower than 240. Firing 200 at once saturates it and the overflow returns 500 ERRORED_DOCUMENT_REQUEST, not 429.

So concurrency is discovered rather than declared:

• The right in-flight number is not a constant, it is whatever holds the admission rate at the quota given how long a call currently takes: concurrency = rate × latency. At 230 a minute and 20 seconds a call that is 76; if Google speeds up to 12 seconds it is 46, and holding 76 would just buy 429s. A rolling median of the last 40 successful calls sets the target, and the ceiling climbs toward it in steps of at most 1.5×.
• Opens at the ceiling the last run settled on, carried in localStorage, rather than relearning the same number from 40 every time.
• Halves the ceiling after 3 clustered errors, floor of 5, then climbs again. Additive increase, multiplicative decrease.
• A 429 pauses the whole batch 7 seconds; a cluster of 5xx pauses it 3.
• A rolling 230-per-minute admission guard sits underneath all of it.

5.1Extract

Each response yields far more than seven numbers. A run whose runtimeError is set is discarded here rather than averaged in, because Lighthouse documents that field as meaning the result may need to be thrown away:

5.2Aggregate across the sample

Findings are rolled up so that one slow asset on forty pages is one finding, not forty. Figures are reported per page throughout: Lighthouse savings summed across a sample produce numbers like "90 seconds wasted", which is not time anyone can save. A resource's size is its own weight, not that weight multiplied by the page count. The across-sample totals survive only as the ranking signal.

5.3Apply the rules

17 rules decide what counts as a finding, which is a different job from listing every audit that returned data. Each declares a condition, a severity, a fix risk, an effort level, an evidence set and a fix. Two of them combine signals a spreadsheet row cannot: a collection performing well below the site mean, and a competitor beating the subject.

Rules fire in declaration order and sort by severity. Nothing is written by a language model. Every sentence is a template filled from the same numbers its evidence table shows.

5.4Route the fix to the platform

Per finding, per site, resolution runs in a fixed order: the curated playbook first, then Lighthouse's own stack pack, then nothing. It never invents advice for a platform it has no entry for, and Lighthouse-sourced text is labelled.

The playbook covers 14 platforms by audit id, and 18 plugin slugs carry their own note. If ShortPixel is already installed, the image finding is a bulk run in a plugin they own rather than a purchase.

Six outputs, all built from the same model so none of them can drift from the others:

• Lab data still leads the findings. Field data is now reported at both page and origin level and the rules rank it above the lab numbers, but the diagnosis underneath is still synthetic and varies between runs. Treat a lab difference under about five performance points as noise.
• The sample ignores traffic. An even spread weights a dead 2019 post the same as the page carrying revenue. Search Console impressions would fix the weighting.
• No history. Every run is independent. There is no before-and-after and no trend.
• Fingerprinting reads the homepage only. A platform whose signature appears only on interior templates comes back unidentified, and headless setups often will. It degrades to the generic fix rather than guessing.
• Gzipped sitemaps are skipped. Browsers cannot decompress them from fetch.