A page that Google will not index cannot rank, cannot be found, and cannot earn a single lead. You paid to build it and it is invisible. Most companies your size, roughly ten to a hundred staff with one marketing generalist or a small team, have somewhere between ten and forty percent of their published pages sitting unindexed and nobody has noticed, because nobody owns the check. The good news is that the causes are boring and repeatable. Eight of them cover almost everything, six are fixable in under an hour, and the whole diagnosis fits into a ten-minute weekly habit that one person on your team can run. This is that process, written so you can hand it to whoever owns marketing and have them run it the same way every week.
Why this is a process problem, not a tool problem
At your size the temptation is to buy your way out. A sales rep for an enterprise crawler will tell you their platform catches indexing issues automatically, and it does, at four figures a month you do not need to spend yet. The thing that actually catches these issues is a person looking at the same free report on the same day every week. The tool is Google Search Console, which is free and is the source of truth Google itself hands you. The discipline is what you are buying, and you buy it with fifteen minutes of one person's time, not with a subscription. Get the habit working first. If you outgrow it, you will know, because the check will start taking too long. Most companies your size never reach that point.
Step 0: confirm the page actually is not indexed
Before you diagnose anything, confirm the problem is real. There are two fast ways. Search Google for "site:example.com/your-path" with your real domain and path. If the page appears, it is indexed and your problem is ranking, not indexing, which is a different piece of work. If it does not appear, paste the URL into the Inspect URL tool inside Search Console. That gives you Google's own answer, indexed or not, plus the exact reason category when it is not. That reason category is what drives everything below. Do not guess at causes. Read the category off the report and act on it.
The eight causes, and the fix for each
Almost every not-indexed page falls into one of eight buckets. Your generalist does not need to memorize these. They need this list open next to the Inspect URL result so they can match the reason Google gives to the fix. Here they are in the order you will meet them.
1. Google has not crawled it yet
Search Console status: "Discovered, currently not indexed." Google knows the URL exists, probably from your sitemap, but has not prioritized crawling it. This is normal for new pages on a site without much authority yet. The fix is to build internal links to the page from pages that are already indexed, which is also just good site structure, then use Inspect URL and Request Indexing to nudge a crawl. Do not panic on this one. It often resolves itself within a week or two.
2. Google crawled it and chose not to index it
Status: "Crawled, currently not indexed." This is the one that stings, because Google looked at the page and decided it did not add value. Usually the page is thin, duplicates something else on your site, or reads as low quality. The fix is content, not technical: make it longer, more specific, and genuinely useful, add internal links, update the publish date, and re-request indexing. If a page keeps landing here after you improve it, that is Google telling you the page may not need to exist. That is a real signal worth listening to.
3. A noindex directive is on the page
Status: "Excluded by noindex tag." The page carries a robots meta tag or an X-Robots-Tag header telling Google not to index it. The fix is to remove the directive if you want the page found. At companies your size the usual culprits are a layout-level tag that was never meant to apply to this route, a setting flipped in your CMS, or a developer's work-in-progress state that got shipped by accident. This one is worth catching fast, because it is invisible on the page itself and can silently sit on a whole section of your site.
4. Blocked by robots.txt
Status: "Blocked by robots.txt." A rule in your robots.txt file is disallowing the URL. The fix is to edit that file to allow the path. The classic version of this at a growing company is an over-broad rule someone added to block admin or staging pages that accidentally also catches real marketing pages. Have whoever owns the site read the robots.txt file line by line once. It takes five minutes and prevents a costly surprise.
5. The canonical points somewhere else
Status: "Alternate page with proper canonical tag." Your page declares a different URL as the canonical, so Google indexes that other URL instead. Most of the time this is healthy and you should leave it alone, because it is exactly what prevents duplicate-content problems from query strings and pagination. Only fix it if the canonical is genuinely pointing at the wrong page. This is the cause your team should learn to recognize and then not touch, which saves hours of chasing a non-problem.
6. The page returns a 404 or a 5xx error
Status: "Not found (404)" or "Server error (5xx)." Google tried to crawl the URL and got an error. The fix is to make the URL respond correctly. If the page genuinely should not exist, that is fine, but then remove the URL from your sitemap so you stop telling Google to look for it. Intermittent 5xx errors are the sneaky version here: the page works when you check it and fails when Googlebot hits it, so trust the report over your own browser.
7. A redirect chain or loop
Status: "Page with redirect" or "Redirect error." The URL redirects to another URL, and Google may not follow deep chains or may hit a loop. The fix is to collapse chains to a single hop. If page A redirects to B which redirects to C, rewrite it so A goes straight to C. Redirect chains accumulate quietly over years of site changes, so a company that has rebranded or restructured its URLs a couple of times almost always has some.
8. Duplicate content
Status: "Duplicate without user-selected canonical" or "Duplicate, Google chose different canonical." The page is too similar to another page on your site and Google collapsed them. The fix is to make the pages substantively different or to canonicalize deliberately to one of them. This shows up a lot on template-driven pages, like near-identical location or service pages that only swap a city name, which is a common pattern at companies expanding into new markets.
The investigation order your team follows every time
The point of a repeatable process is that the person running it does not have to think about where to start. Give them this exact order and they will diagnose any single unindexed page the same way every time.
- Inspect the affected URL in Search Console and read the reason category.
- If the status is "Crawled, currently not indexed," the problem is the content. Thin, duplicate, or low value. Improve it.
- If the status is technical (noindex, robots.txt, redirect, server error), fix the technical cause. This usually needs whoever owns the code or the CMS.
- If the status is "Discovered," the page needs more internal links pointing at it. Add them from your strongest existing pages.
- After any fix, re-inspect the URL and click Request Indexing. Wait 24 to 48 hours, then re-check that it moved to indexed.
The weekly check, as a ten-minute standing habit
The individual diagnoses above are what you do when a specific page is broken. The weekly check is what surfaces the broken pages before they cost you a quarter of traffic. Put it on one person's calendar, same slot every week, and keep it short so it actually happens. Ten minutes is the target.
- Open the Search Console Pages report and look at the "Not indexed" total. Is it higher than last week? By how many?
- Click into any reason category that grew. See which new URLs landed there in the last seven days.
- For each new not-indexed URL, run it through the investigation order above, or log it for whoever owns that fix.
- Confirm nothing that was indexed last week has silently dropped out. A sudden fall is your early warning that something broke site-wide.
- Write one line in a shared doc: what you found, what you fixed, what you handed off. That log is your ROI evidence.
Most weeks the honest answer is "nothing changed," and that is a win, not wasted time. The value is entirely in the week something does change, because you catch it in the same cycle instead of three weeks later when a whole section has quietly fallen out of the index and your organic leads are down and nobody can say why.
When a page gets deindexed after it was indexed
This is a different and more urgent problem than a page that was never indexed. A page Google indexed and then dropped means something changed. Walk through these before anything else.
- Did the page's content change a lot? Google may have re-evaluated it and decided it no longer qualifies.
- Did a Google algorithm update land recently on your topic? A quick industry-tracker check confirms whether this was you or the whole SERP.
- Was there a stretch of server errors that stopped Google crawling? Pages with persistent crawl failures get dropped.
- Did the page get a new canonical tag pointing somewhere else, maybe from a template change nobody reviewed?
Fixing indexing problems in bulk without buying a bulk tool
When you have not just one bad page but dozens, the instinct is to fix them one by one, or to buy a crawler that promises to do it for you. Neither is right at your size. The move is to fix the underlying pattern, because problems at scale almost always come from a shared cause. Look for the pattern first.
- Every blog tag or category page excluded? Those pages are probably thin auto-generated lists. Either give them real intro content or accept that they should not be indexed.
- Every paginated page excluded? That is usually healthy. Leave it alone.
- Many location or service pages sitting in "Crawled, currently not indexed"? They are too similar to each other. Invest in real, specific detail per page.
- A batch of old blog posts dropping out together? Update the ones worth keeping, consolidate the rest into stronger single pages.
Fixing the pattern once fixes fifty pages at a stroke, and it costs you a person's afternoon rather than a monthly subscription. This is exactly the kind of judgment a person makes and a tool does not, which is why the process beats the platform at your scale.
The first tool, vendor, and hire decisions
You will eventually ask whether to buy software, hire someone, or bring in an outside vendor. Here is the honest ordering for a company your size.
Tools
Start and stay with Search Console for as long as it holds. The next tool worth paying for is usually a mid-tier crawler like Screaming Frog, which is affordable and lets one person audit the whole site's technical health in an afternoon. Do not buy an enterprise SEO suite priced for a marketing department of twenty until you actually have a marketing department of twenty. Over-buying tooling early is one of the most common ways companies your size waste budget, because the seats sit unused and the owner rightly asks what the line item is doing.
In-house versus outsource
The weekly check is a keep-it-in-house job. It is ten minutes, it needs continuity, and the person who runs it every week builds the pattern recognition that makes it valuable. The one-time deep indexing audit, the kind that untangles years of accumulated redirect chains and canonical mistakes, is a good thing to outsource, because it is a bounded project that rewards specialist experience and does not need to live on your payroll. Keep the ongoing habit; buy the occasional deep clean. Compare the three delivery models, in-house, sprint, and thin retainer, on our solutions overview.
When to hire
You are ready to hire dedicated SEO help when the weekly check consistently surfaces more work than your generalist can clear alongside their other work, and when the leads that work protects are worth clearly more than the salary. Until then, a generalist running a good process beats a specialist running no process.
Proving the spend to an owner or finance
At your size someone has to justify the time and money, usually to an owner who reasonably wants to know what marketing is producing. Indexing work is unusually easy to defend, because the before-and-after is concrete. Report it like this.
- The gap number. Pages in your sitemap versus pages Google has indexed. "We published 140 pages, Google was showing 96, we closed the gap to 138." That is a number an owner understands in one sentence.
- Issues caught per month. The weekly log turns into a monthly count of problems found and fixed before they cost traffic. Prevention is hard to see, so make it visible.
- Traffic on recovered pages. When a previously unindexed page starts earning clicks, that click line is directly attributable to the indexing work. Tie it back to leads where you can.
- Time spent. Roughly fifteen minutes a week plus the occasional fix. Against the traffic it protects, the ratio makes the case for itself.
How other team sizes run the same check
The eight causes are universal, but the way you operate the check scales with your team. If you are a one-to-nine-person owner-operator doing this yourself with no marketing staff, the leaner do-it-yourself version is in the micro-business guide. If you deliver this across many client sites as an agency, the templatized, repeatable-across-clients version is in the agency guide. If you are a hundred-plus-person organization with governance, procurement, and an existing martech stack, monitoring index health at scale is covered in the mid-market guide. Pick the one that matches how you actually operate.
Where this fits in a real SEO program
Indexing is the foundation, not the whole house. There is no point spending on rankings, content, or links for pages Google will not even index, which is why a serious engagement starts by closing the indexing gap before anything else. Once the gap is closed and the weekly check is holding it there, you have earned the right to work on ranking and conversion. For a professional services firm in particular, where a single indexed and ranking service page can be worth a lot of pipeline, this foundation is where the return starts. See how we frame the work for professional services specifically.
Set up the check this week
You do not need a budget approval or a new hire to start. You need Search Console verified, one person assigned, and a ten-minute weekly slot on their calendar. Run the investigation order when a page breaks, keep the log so you can prove the value, and buy tooling or bring in help only when the process itself tells you it is time. The two official references worth bookmarking for your team are Google's own Search Central documentation and the Search Console help center, both of which are the primary sources for every status category above.
If you would rather have the indexing gap found and closed for you, and the weekly cadence set up so your generalist can run it going forward, run the estimator to see the scope, or talk to us. The indexing audit is a bounded, defensible first project that almost always finds quick wins a small team can then maintain on its own.
