The short answer: indexation logic is the set of rules a programmatic site uses to decide, for every possible combination of data, whether a page should be generated, indexed, kept out of search, consolidated into another page or not created at all. Good rules score each combination on search demand, data sufficiency, uniqueness and business value, apply thresholds automatically, and re evaluate pages as data changes, so that only pages worth indexing are offered to Google. Google itself says not to expect every URL to be indexed, and that it is fine for a URL not to be indexed for the right reasons.
Programmatic sites can generate far more URLs than they should. A directory with 400 locations, 60 categories and five filters can produce millions of combinations, most with no searches and little data. Without explicit indexation rules, the site leaves the decision to Google, which shows up as large numbers of pages “crawled, currently not indexed” or “discovered, currently not indexed” in Search Console (Search Console Help). This guide sets out a practical framework for those rules, with scoring criteria, thresholds, the technical controls for each outcome, how to keep rules current as data changes, and a worked example. It builds on our hub on programmatic SEO.
What is indexation logic in programmatic SEO?
Indexation logic is a decision layer between your dataset and your website: for every combination of attributes, such as location, category, brand or filter, it decides which of five outcomes applies, and the site’s templates, sitemaps, canonicals and robots rules then implement that decision automatically.
| Outcome | What it means | Technical control |
|---|---|---|
| Index | Page is generated, linked, in the sitemap and indexable | 200 status, self canonical, in XML sitemap |
| Generate but noindex | Useful to visitors who navigate there, not worth a search result | noindex meta tag; excluded from sitemap |
| Consolidate | Near duplicate of a stronger page | Canonical or 301 to the stronger page |
| Block from crawling | Pattern that creates crawl waste, such as sort orders | robots.txt disallow for the parameter |
| Do not generate | Combination with no data or no meaning | No link to it; 404 if requested |
Why do programmatic sites need explicit indexation rules?
Programmatic sites need explicit rules because the number of possible pages grows multiplicatively with each attribute, while search demand and useful data are concentrated in a small share of combinations; without rules, low value pages dilute crawling, slow discovery of good pages and risk being treated as scaled content.
- Google’s crawl budget guidance is aimed at very large sites, those with around a million or more pages, sites with 10,000 or more pages changing daily, and sites with many URLs discovered but not indexed (Google Search Central). Programmatic sites often fall into the last group long before they reach a million pages.
- Google lists generating many pages without adding value for users, including through automated transformations, as an example of scaled content abuse (Google Search Central).
- For faceted and filtered URLs, Google’s guidance warns of overcrawling of duplicate content and slower discovery of new pages (PPC Land, summarising Google).
Which criteria should decide whether a programmatic page is indexed?
Score every combination on four criteria, search demand, data sufficiency, uniqueness and business value, and require a minimum on each rather than a high total, because a page with strong demand but no data still fails its visitors.
| Criterion | Question | Typical measure | Example threshold (illustrative) |
|---|---|---|---|
| Search demand | Do people search for this combination? | Keyword volume, Search Console impressions for similar pages | At least some measurable demand, or the parent pattern earns impressions |
| Data sufficiency | Is there enough data to answer the search? | Listings, products, reviews, data points on the page | At least 5 listings, or 3 data blocks populated |
| Uniqueness | Is the page meaningfully different from its siblings? | Share of content that differs from the nearest page | Results overlap with sibling page under 60% |
| Business value | Can visitors convert or does it support pages that do? | Conversions from similar pages, commercial intent | Commercial or local intent, or supports a commercial page |
Thresholds must be set per site from your own data. The values above are starting points to test, not rules from Google.
How do you turn the criteria into decision rules?
Turn the criteria into an ordered set of rules that run for every combination, checking the cheapest disqualifiers first, then duplicates, then demand, and indexing only what passes every test.
for each combination in dataset:
if combination is meaningless or data_points == 0:
outcome = DO_NOT_GENERATE
elif pattern is crawl_waste (sort order, session, view mode):
outcome = BLOCK_CRAWL
elif overlap_with_stronger_sibling >= 0.6:
outcome = CONSOLIDATE (canonical or 301 to sibling)
elif data_points < MIN_DATA:
outcome = NOINDEX (or DO_NOT_GENERATE if no visitor value)
elif search_demand == 0 and not supports_commercial_page:
outcome = NOINDEX
else:
outcome = INDEX
store outcome, scores and date for the pageStoring the outcome and the scores for every page makes the system auditable: when Search Console reports a page as not indexed, you can see whether your own rules expected it to be indexed. For building and testing rules like this on real data, see our guide to Claude Code for SEO.
How do you measure search demand for combinations that have no keyword data?
Estimate demand for long tail combinations from the pattern rather than the single page: use keyword data for the head terms, Search Console impressions for similar pages already live, internal site search, and the performance of a pilot batch.
| Signal | How to use it |
|---|---|
| Keyword tools for head patterns | Confirm that “[service] in [city]” style searches exist at all |
| Search Console impressions for live siblings | Pages in the same pattern earning impressions suggest demand for the rest |
| Internal site search | What visitors type shows combinations they want |
| Pilot batch | Publish a sample, measure impressions and indexing after several weeks |
| Sales and enquiry data | Locations and categories that generate leads deserve pages first |
Demand alone is not enough. Use your revenue keyword set to weight combinations that lead to business above those that only bring traffic.
How do you measure data sufficiency and uniqueness?
Measure sufficiency by counting the populated data blocks and items on each page, and uniqueness by comparing each page’s results and content with its nearest sibling; a page that shows the same items as its parent with a different heading is a duplicate, whatever its URL says.
- Item counts: listings, products, providers or data points per page; set a minimum per template.
- Block coverage: how many of the template’s data blocks are populated; empty blocks should hide, and pages with too few should not be indexed.
- Result overlap: share of items shared with the parent or sibling page; high overlap means consolidate.
- Template quality: our guide to programmatic page templates explains how answer blocks and conditional sections create unique value per page.
What should happen when the data behind a page changes?
Re evaluate pages on a schedule and when data changes, with hysteresis: a page drops out of the index only after staying below threshold for a set period, and returns only after staying above it, so pages do not flip between indexed and noindexed every week.
| Situation | Rule (illustrative) |
|---|---|
| Listings fall below minimum | Keep indexed for 30 days; then noindex if still below |
| Listings recover | Return to index after 14 days above minimum |
| Combination permanently empty | 301 to the parent page, or 404 if no relevant parent |
| Seasonal combination | Keep one permanent URL; update content each season |
| New combination appears | Enter pilot status; index once it passes all criteria |
Frequent flipping wastes crawling and confuses both Google and visitors. Stable URLs with clear rules perform better than aggressive automation.
Which technical controls implement each indexation decision?
Implement decisions with controls that match their purpose: noindex for pages visitors may use but search should not show, canonical or 301 for duplicates, robots.txt for crawl waste patterns, a 404 for combinations that should not exist, and sitemaps containing only pages meant to be indexed; Google recommends returning a 404 for truly not found pages rather than a soft 404 (Search Console Help).
| Decision | Control | Common mistake |
|---|---|---|
| Index | Self canonical, in sitemap, linked from hub pages | Indexable page with no internal links |
| Noindex | Meta robots noindex; remove from sitemap | Blocking in robots.txt, which stops Google seeing the noindex |
| Consolidate | Canonical, or 301 if the page has no separate purpose | Canonical to a page with different results |
| Block crawl | robots.txt disallow on parameters | Blocking pages that should be indexed |
| Do not generate | No links; 404 if requested | Empty pages returning 200 (soft 404) |
How do you monitor whether your indexation logic is working?
Compare your intended outcomes with Google’s actual indexing by pattern: for each template and tier, track how many pages you intend to index, how many Google indexes, and how many earn impressions and conversions.
| Metric | Healthy sign | Warning sign |
|---|---|---|
| Index rate of intended pages | Most intended pages indexed within weeks | Many crawled or discovered but not indexed |
| Impressions per indexed page | Most indexed pages earn impressions | Many indexed pages with none |
| Conversions by pattern | Patterns convert in line with expectations | Traffic without business |
| Crawl stats | Crawling focused on indexable patterns | Crawling dominated by parameters |
Google says pages reported as crawled but not indexed may or may not be indexed later, and resubmitting them will not help (Search Console Help); a high share in one pattern usually means the rules let through pages that do not meet Google’s bar. Prune them using the approach in our guide to content pruning with conversion data.
What does indexation logic look like in a worked example?
In a worked example for a property portal, applying demand, data, uniqueness and value rules reduced tens of thousands of possible combinations to a few thousand indexable pages, with most of the rest either not generated or kept out of the index.
Illustrative scenario: “A UAE property portal covers 160 communities, 6 property types, 6 bedroom counts, rent or sale, and 8 amenity filters. Before rules, the site exposed every combination. After rules:”
| Stage | Pages | Rule applied |
|---|---|---|
| Possible combinations (without amenities) | About 11,500 | 160 communities x 6 types x 6 bedrooms x 2 |
| With one amenity filter applied | About 104,000 | 11,520 x 9 (no amenity or one of 8) |
| Generated | About 18,500 | Only combinations with at least one listing |
| Indexable | About 6,200 | At least 5 listings, overlap with parent under 60%, demand in pattern |
| Noindex | About 9,100 | Useful for browsing but thin or no demand |
| Consolidated | About 3,200 | Near duplicates canonicalised to parent |
| Amenity filter URLs | Blocked | Crawl waste; three high demand amenities built as pages instead |
The portal and figures are illustrative, not a real client. Expected outcome: crawling concentrated on about 6,200 pages, faster discovery of new listings, and fewer pages in the not indexed buckets of Search Console.
What are the common indexation logic mistakes?
| Mistake | Better approach |
|---|---|
| Indexing every combination and letting Google decide | Decide explicitly; offer Google only pages worth indexing |
| Using robots.txt to hide pages that carry noindex | Let Google crawl them to see the noindex, or do not generate them |
| Flipping pages in and out of the index weekly | Use hysteresis: time based thresholds |
| Thresholds copied from another site | Set from your own data and pilot results |
| Sitemaps listing noindex or canonicalised URLs | Sitemaps only for indexable pages |
| Empty result pages returning 200 | Return 404 or redirect to the parent |
| No record of why each page was indexed | Store scores and decisions per page |
How Crawl To Click designs indexation logic
Crawl To Click is an SEO and AI visibility consultancy, and for programmatic sites we design indexation logic before templates go live: we model every attribute combination, measure demand by pattern, define data and uniqueness thresholds with the product team, map each outcome to technical controls and sitemaps, and set up monitoring by template and tier. We have worked on property portals, job and travel listings, software integration directories, healthcare and education directories and ecommerce catalogues. Google’s guidance on helpful, people first content is the test every rule set is checked against.
FAQs on indexation logic for programmatic SEO
What is indexation logic?
Indexation logic is the set of rules a programmatic site uses to decide, for every possible combination of data, whether a page is indexed, generated but noindexed, consolidated into another page, blocked from crawling or not created at all.
Should every programmatic page be indexed?
No. Google says not to expect every URL to be indexed and that it is fine for URLs not to be indexed for the right reasons. Only combinations with search demand, enough unique data and business value should be offered for indexing.
Should I use noindex or robots.txt for thin programmatic pages?
Use noindex for pages that visitors may use but that should not appear in search, and keep them crawlable so Google can see the tag. Use robots.txt for patterns that waste crawling, such as sort orders and session parameters, that should never be crawled.
What should happen to a programmatic page with no results?
A page with no results should not return a normal 200 status with an empty template. Redirect it to the closest parent page or return a 404, because Google treats empty pages returning 200 as soft 404s.
How many listings does a programmatic page need to be indexed?
There is no Google rule. Set a minimum from your own data, such as the number of items at which pages start earning impressions and engagement in a pilot, and apply it consistently with a grace period before pages drop out of the index.
Why are my programmatic pages crawled but not indexed?
Programmatic pages are usually crawled but not indexed because Google sees too little unique value in them, they duplicate sibling pages, or there are so many low value URLs that Google limits indexing. Tighten indexation rules and improve the data on pages you keep.
Building or fixing a programmatic site?
If Search Console is full of crawled but not indexed pages, or you are about to launch thousands of URLs, get a second opinion on your indexation rules before Google decides for you.
Disclaimer: This article reflects Google documentation as of the dates stated. Thresholds and the worked example are illustrative and must be tested on your own data. The example does not describe a real business.
