Before a page can rank, Google has to clear three steps with it: discover it, crawl it, then decide to index it. Technical SEO is the craft of smoothing that path, spotting where it stalls, and fixing it.
This section gathers IndexProbe's cross-cutting guides on crawling and indexing. Where the Google Search Console indexing statuses section breaks down each state in the report one line at a time, these guides take the topic by its mechanics: how Google discovers your URLs, how fast it crawls them, and why some never reach the index.
Discovery and Submission
How to Submit Your Website to Google
From the indexing request to verification. Why submitting guarantees nothing, how to declare a sitemap properly, the real limits of the URL Inspection tool, and how to confirm what Google actually indexed.
Crawl Budget and Crawl Frequency
Crawl Budget: Is It Actually Your Problem?
A real SEO lever, but not for every site. Who crawl budget truly matters for, why the threshold sits far lower than the million pages Google cites, and how to pinpoint, URL by URL, where the budget drains away.
Crawl Errors and HTTP Codes
How to Fix 404 Errors in Google Search Console
Not all 404s are equal, and not all of them need fixing. How to tell legitimate 404s from broken links worth repairing, and how to handle the volume without losing entire days to it. (This guide also belongs to the GSC indexing statuses section.)
Coming to This Section
Technical SEO for crawling and indexing doesn't stop there. Several guides are in the works: faceted navigation and the URL explosion it triggers, pagination and the self-referencing canonical, JavaScript rendering and single-page apps against indexing, duplicate content on e-commerce sites, and a decision guide for choosing between canonical, noindex, robots.txt, or 404 depending on the case.
The indexing pitfalls specific to each CMS have their own section: CMS Indexing Pitfalls.
Why These Guides Matter
A crawling or indexing problem isn't visible to the naked eye: a page can look perfect and stay out of the index for weeks. The only reliable source is Google's official verdict, page by page. Checking it by hand in Search Console, one URL at a time, doesn't hold up at scale. That's exactly what IndexProbe does: the bulk version of the URL Inspection tool, querying the official Search Console API for the whole list of URLs you provide or build from GSC, with the indexing status, the non-indexation reason, and the last crawl date for each one.
Frequently Asked Questions
What's the difference between crawling and indexing?
Crawling is Googlebot's visit to read the page. Indexing is Google's decision to keep it in its index. A page can be crawled without being indexed: Google read it but didn't judge it worth keeping. The two steps are distinct, and a problem can arise at either one.
Where should a technical SEO audit start?
With the "Page indexing" report in Google Search Console, which shows how many of your URLs are indexed and why the rest aren't. That's the starting point: it steers you to the right guide, whether the issue is discovery, crawl budget, HTTP errors, or duplicates.
Are these guides meant for small sites?
Partly. Some topics, like submission or 404 errors, concern every site. Others, like crawl budget or faceted navigation, only become real issues past a certain URL volume. Each guide states clearly who the topic is relevant for, without needless drama.