Comparison · Official GSC API
Screaming Frog's URL Inspection API: when a sample of your site isn't enough
Screaming Frog is an excellent crawler — and its URL Inspection integration is honest about its ceiling: 2,000 URLs per day per property, populated in crawl order, with a manual path beyond. IndexProbe is the other half of the job: the official index verdict for your whole list, day after day, automatically.
Their integration
What their integration does — and where it stops
The SEO Spider pulls URL Inspection API data for up to 2,000 URLs per property per day, alongside crawl data. Their tutorial is precise about the mechanics: data lands on “the first 2k URLs found in the crawl, which is breadth-first … from the start page”, and past the limit, “URLs just won't continue to be populated with URL Inspection data.” Your deep money pages are exactly the ones crawl order reaches last.
Their documented path beyond 2,000 is manual: “wait for 24hrs, re-open the crawl … bulk highlight and ‘re-spider’ the next 2k URLs”, then export and combine “with the previous days crawl data” — by hand, every day. Credit where due: their “Use Multiple Properties” option is real and automated within a single crawl, the same per-property lever IndexProbe uses.
And their philosophy is stated openly: “Google didn't build the API to allow webmasters to check if every single URL on their website is indexed … The purpose of the API is to … get a better sample across templates.” Their own automation workflow follows suit: schedule a list of “the top 10, 20 or 100 URLs”. Sampling, by design.
The gap
A sample tells you rates. It can't name the page that dropped.
Sampling across templates answers “roughly how much of this template is indexed?”. It cannot answer the question that costs money: which of your pages left the index yesterday, and why. On a 100,000-page site, a 100-URL sample will miss almost every individual regression, by construction.
On the underlying point, Google — and Screaming Frog relaying it — are right: not every URL needs to be indexed, and some exclusions are perfectly healthy. IndexProbe's stance is not “index everything”. It is: know which pages are out, and why — the legitimate canonical consolidations on one side, the real losses on the other.
That question only stays answerable if the quota stops being a reason to shrink it. IndexProbe engineered around it — multi-property aggregation, the accelerated inspection mode, automatic day-after-day resume — so you can send the full list and let the tool work through it.
The product
The inspection layer as a product, not a passenger
In Screaming Frog, URL Inspection is an add-on riding along on a crawl. In IndexProbe, it is the entire product: list in, verdicts out, monitoring on — at a scale their own tutorial advises giving up on.
Your list, not the first 2,000
The crawl decides nothing here. Import the pages that earn clicks or impressions straight from Search Console, match a regex pattern, paste a CSV, load a sitemap — and suspected regressions are inspected first, so the verdicts that matter arrive within hours.
Press start once
Their documented path past 2,000: re-open, highlight, re-spider, export, merge — by hand, every day. IndexProbe's path: press start once. The analysis resumes automatically day after day until the whole list is covered, with your machine off.
The quota, engineered instead of endured
The accelerated inspection mode counts pages with recent impressions without touching the quota. Multi-property aggregation and the domain-aggregated view (consolidated, deduplicated) raise throughput to 60,000 inspections a day on the Agency plan — 100,000 URLs in 2 days, up to 1.8 million distinct URLs a month.
Monitoring built in, not assembled
Their documented monitoring pipeline has five parts: Scheduling, a saved config file, headless mode, a Google Sheets export on your Drive, and their Looker Studio template to copy and connect — for a top-100 list. IndexProbe ships the result: recurring analyses, email alerts, the Comparison view, segments and history — built in, for 100,000 URLs, reasons included.
Side by side
Side by side
Two different objects: a desktop crawler with an inspection add-on, and a cloud product built around inspection. The table is about the half of the job they share.
| Screaming Frog + URL Inspection API | IndexProbe | |
|---|---|---|
| Built as | A crawler with an API add-on | The inspection layer as the whole product |
| Which URLs get data | The first 2,000 found in crawl order (their doc) | The list you choose — clicks, impressions, regex, CSV, sitemap |
| Past 2,000 | “Wait for 24hrs, re-open … re-spider”, merge by hand (their doc) | Automatic resume, day after day, until the list is done |
| Multi-property | Real: “Use Multiple Properties”, single crawl | Real too — plus the domain-aggregated view, consolidated and deduplicated |
| Speed levers | — | Accelerated mode: impression-backed pages counted without quota; regressions inspected first |
| Their advice at scale | “A better sample across templates” — top 10-100 URLs | Full coverage: 60,000/day on Agency, 100,000 URLs in 2 days, 1.8M distinct a month |
| Monitoring | Assemble it: Scheduling + config + headless + Sheets + Looker Studio template | Built in: recurring analyses, email alerts, Comparison view, history |
| Canonical data | “User-Declared Canonical Not Selected” filter | Declared vs Google-selected in filterable columns, difference badge, at 100K scale |
| Runs on | Your desktop, machine on | Cloud — nothing installed, nothing left running |
| Price | Licence £199/year, per user (integrations require the paid version) | From $19/month · free 10-day trial, no card |
Pricing
Two tools, one budget line each
This is not a rip-and-replace comparison: Screaming Frog is a crawler, and IndexProbe is not one — by design. Keep the crawler. The question is what runs your index monitoring.
The deal, spelled out: keep your Screaming Frog licence for crawling — it's excellent at it. For the indexing layer, $19/month buys what the DIY pipeline can't: your whole list inspected with reasons, automatic daily resume, alerts and history — and the Agency plan at $99 carries 30 projects, 60,000 inspections a day, up to 1.8 million distinct URLs a month. No spreadsheets to merge, no dashboard to assemble.
Screaming Frog SEO Spider
£199 per year, per user, as published; the free version stops at 500 crawled URLs and the integrations ship with the paid licence. URL Inspection data arrives alongside crawls: 2,000 URLs per property per day, first-found order, a manual path beyond, monitoring via the Looker Studio pipeline.
IndexProbe
Starter $19/month, 3 projects, 30,000 URLs per analysis · Pro $49/month, 10 projects, 60,000 URLs, AI segmentation · Agency $99/month, 3 Search Console accounts, 30 projects, 100,000 URLs per analysis, add-ons (+10 projects, +1 account). Free 10-day trial, no credit card.
Honest framing
What IndexProbe does, and does not do
IndexProbe is not a crawler, and does not want to be one. Screaming Frog crawls what your site exposes; IndexProbe reads what Google decided. Two different jobs — one of them is our whole product.
100% official data
The same verdicts as Search Console's URL inspector, non-indexation reason included, with no scraping and no estimates.
Your lists, not a crawl
Search Console export, CSV, sitemap, or paste, up to 100,000 URLs per analysis. IndexProbe does not discover URLs by following links.
Google's quota, handled for you
Official inspection is capped at 2,000 URLs per day per property, but IndexProbe gets you past that limit. First through the accelerated inspection mode: pages with recent impressions or whose indexing was recently confirmed are counted without consuming that quota, through an API with a separate quota. Then through multi-property aggregation: the quota applies per property, and IndexProbe aggregates the projects of a single domain — with 10 prefix properties created in your Search Console, at least 20,000 URLs are processed every day — more than 140,000 distinct URLs in a week — and often far more thanks to the accelerated mode. At Agency-plan scale, 30 projects × 2,000 URLs per day open up to 60,000 daily inspections — potentially 1.8 million distinct URLs a month, before the accelerated mode even kicks in. Whatever remains spreads automatically over the following days.
Not every exclusion is a problem
Google's point stands: some URLs are legitimately out of the index. The job is knowing which ones, and why — a variant properly canonicalized is healthy; a money page in Soft 404 is not. Reasons make the difference, samples don't.
FAQ
Frequently asked questions
Does Screaming Frog's URL Inspection integration show the same data as IndexProbe?
Per inspected URL, yes — both read the official URL Inspection API. The differences are which URLs get inspected (the first 2,000 in crawl order versus the list you choose), what happens past the quota (a manual re-spider path versus automatic day-after-day resume), and the monitoring layer around the data.
Can Screaming Frog check more than 2,000 URLs a day?
Yes, through two documented paths: waiting 24 hours and manually re-spidering the next 2,000, or verifying multiple properties so a single crawl draws on several quotas. What the integration does not offer is automatic multi-day resume over a large list, impressions-based acceleration, or built-in recurring monitoring at that scale.
Should I stop using Screaming Frog?
No. It is an excellent crawler, and IndexProbe is not one, by design. Keep it to audit what your site exposes — structure, tags, links. Use IndexProbe for the other half: what Google decided about your pages, at scale, continuously, with the reason for every non-indexed URL.
Why not just monitor a sample of top pages?
A sample estimates rates per template; it cannot name the individual pages that dropped out yesterday. Regression detection needs the full list — which is exactly what per-property quotas, the accelerated mode and automatic resume make affordable in IndexProbe.
How do the prices compare?
Different objects: Screaming Frog is a desktop licence at £199 per year per user, covering the crawler and its integrations. IndexProbe is a cloud subscription from $19/month for the indexing layer. Many teams run both; the overlap is only the 2,000-a-day inspection window inside a crawl.
How fast can IndexProbe cover 100,000 URLs?
Two days with 30 prefix properties (60,000 inspections a day), about five days with 10, and faster still with the accelerated mode. Then it keeps going: recurring analyses re-cover the list without limit, and email alerts fire when a page leaves the index.
Send the whole list
Your crawler audits what the site exposes. IndexProbe reads what Google decided — for every page on your list, day after day. From $19/month.
Start the free 10-day trial