Crawl Budget and Index Bloat Review
Find low-value URLs that waste crawling and dilute quality.
The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.
Review my site for index bloat and crawl waste. Site type and size: [SITE_TYPE_AND_SIZE]. Data: indexed page counts, crawl stats, URL patterns from a crawl or logs, sitemap counts, parameter URLs, search performance by URL group: [CRAWL_AND_INDEX_DATA]. Work through this tree for each URL pattern group: 1. Is the page useful to a searcher and unique (product, category, article, location)? IF yes THEN keep indexable and ensure it is linked and in the sitemap. 2. Is it a duplicate or near-duplicate (sorting, session, tracking parameters, print versions)? IF yes THEN use canonical tags or parameter handling and avoid linking to the variants. 3. Is it thin or auto-generated with little value (empty categories, tag pages, internal search results)? IF yes THEN noindex or remove, and fix the generator. 4. Is it needed for users but not for search (account pages, cart, filters)? IF yes THEN block crawling where appropriate or noindex, understanding that blocking prevents search engines from seeing noindex. 5. Are there redirect chains, soft 404s, or pages that return errors? Fix. Return a table: Pattern | Count | Indexed? | Value | Problem | Action | Risk | Verification. Add a sitemap hygiene checklist, a rollout plan in stages with monitoring (indexed counts, crawl stats, impressions), and warnings: big sites only need to worry about crawl budget at scale, and changes should be staged and reversible. No ranking promises.
Fill in before use
Optional: type your details and the prompt above updates. Nothing is saved or sent.
Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.