Develop an internal linking QA checklist for large sites: crawl path validation, anchor…
- User
Develop an internal linking QA checklist for large sites: crawl path validation, anchor consistency, nofollow audits, duplicate hub detection, indexation monitoring, and dashboards for ongoing reporting.
Assistant
The user wants an internal linking QA checklist specifically for large sites. The request lists specific components to include:
- Crawl path validation
- Anchor consistency
- Nofollow audits
- Duplicate hub detection
- Indexation monitoring
- Dashboards for ongoing reporting
The target audience is likely SEO professionals, technical SEOs, or site owners managing complex architectures. The tone should be professional, actionable, and structured.
Deconstruction of the Request:
- Large Sites: This implies challenges with scale, automation, depth of clicks, dynamic generation of links, and legacy issues. Manual checking is impossible; tools and logic are needed.
- Checklist Format: Needs to be a series of steps or tasks, organized logically.
- Components:
- Crawl Path Validation: Ensuring important pages aren't orphaned and are accessible within X clicks from the home page.
- Anchor Consistency: Ensuring the same target URL doesn't have wildly different anchors, and anchors are descriptive.
- Nofollow Audits: Checking for internal nofollows (which usually shouldn't exist) or pagerank sculpting gone wrong.
- Duplicate Hub Detection: Identifying pages that compete to be the "hub" for a topic (keyword cannibalization via structure).
- Indexation Monitoring: Tracking how changes in linking affect what gets indexed.
- Dashboards: Visualization and alerts.
Drafting the Content - Section by Section:
1. Crawl Path Validation (Architecture & Depth):
- Goal: Ensure no page is an orphan and deep pages are reachable.
- Key Metrics: Click depth (distance from Home), PageRank/In-link count.
- Checklist items:
- Identify orphan pages (pages with zero internal links).
- Check depth (e.g., >3 clicks for important pages is bad).
- Verify XML Sitemap vs. Internal Link alignment.
2. Anchor Consistency (Context & Relevance):
- Goal: Clear topical signals to Google.
- Key Metrics: Unique anchors per URL, exact match vs. partial match.
- Checklist items:
- Identify "here" or "click here" anchors for critical pages.
- Check for multiple unique anchors pointing to one URL (is it confusing?).
- Ensure self-referencing canonicals (often overlooked in linking checks, but relevant to indexation).
3. Nofollow Audits (Link Equity Flow):
- Goal: Ensure equity flows freely. Internal nofollow is generally deprecated practice (PageRank sculpting).
- Key Metrics: Count of internal
rel="nofollow". - Checklist items:
- Find internal links with nofollow.
- Check for
robots="noindex, follow"vsnoindex, nofollowlogic (if a page is noindexed, links on it might still pass value, but if nofollowed, they don't). - Review user-generated content (UGC) links (e.g., comments) to ensure they aren't creating a spider trap or equity sink if not handled.
4. Duplicate Hub Detection (Cannibalization):
- Goal: One clear page per topic cluster.
- Key Metrics: Topical clustering, internal link distribution among similar URLs.
- Checklist items:
- Identify pages targeting the same primary keyword.
- Check which page has the highest internal link volume (is it the right one?).
- Consolidate or differentiate.
5. Indexation Monitoring (The Result):
- Goal: Tracking if the links work to keep pages in Google's index.
- Key Metrics: Coverage report status, Indexed vs. Submitted.
- Checklist items:
- Monitor crawl budget waste (linked but non-canonical pages).
- Track "Discovered - currently not indexed" vs. linked pages.
6. Dashboards for Ongoing Reporting (Visibility):
- Goal: Automating the QA.
- Tools: Looker Studio, Screaming Frog, Python, Site: operators (hard for dashboards), GSC API.
- KPIs to Visualize:
- Orphan page count trend.
- Average click depth.
- 404 error rate on internal links.
- Internal link count distribution (does the head match the tail?).
Refining the Structure for "Large Sites":
- Need to mention tools like Screaming Frog, Ahrefs/SEMrush, Python/BigQuery, and Botify/Oncrawl (enterprise level).
- Emphasize automation. On a million-page site, you can't manually check anchors. You need regex or scripts.
Self-Correction/Review during drafting:
- Initial thought: Just list the steps.
- Correction: Need to specify how to do it for a large site. Mentioning "Site:" queries is useless for large sites. Mentioning Log file analysis is crucial for "Crawl Path" validation (real world vs theoretical).
- Nuance: Internal
nofollowisn't always bad (e.g., login pages, admin areas), but it is bad on main content.
Final Polish: Organize into phases: Setup/Discovery, Technical Audit, Strategic/Content Audit, Monitoring. Add a "Tools Stack" section to make it practical.
Let's build the output based on this structure.
- Introduction: Define the scope.
- Section 1: Crawl Path & Architecture (The Skeleton)
- Section 2: Anchor & Semantic Consistency (The Context)
- Section 3: Link Equity & Technical Attributes (The Flow)
- Section 4: Cannibalization & Hub Logic (The Strategy)
- Section 5: Indexation & Crawl Budget (The Result)
- Section 6: Dashboards & Reporting (The Maintenance)
Specific detail to add: "The 3-Click Rule" is a myth for large sites, but "Click Depth relative to Authority" is the reality. I should mention that.
Duplicate Hub Detection: This is effectively "Internal Link Cannibalization." If Page A and Page B both target "Blue Widgets," and the blog links to Page B 100 times and Page A 5 times, but Page A is the money page, that's a failure.
Dashboarding: Google Looker Studio is the standard. Connect GSC + Crawl Data.
Okay, ready to write. For large enterprise sites (100k+ pages), internal linking is not just an SEO tactic; it is the infrastructure that dictates crawl efficiency and authority flow. Manual checks are impossible at scale.
Here is a comprehensive QA checklist designed for programmatic auditing and ongoing maintenance of large-scale websites.
Phase 1: Infrastructure & Crawl Path Validation
Goal: Ensure every valuable page is reachable by bots and users within a reasonable number of clicks, and that no "islands" exist.
1. Orphan Page Detection
- Logic: Pages that exist in the sitemap or have external links but possess zero internal links pointing to them.
- Check: Compare the full crawl list (Screaming Frog/Sitebulb) against XML Sitemaps and Log Files.
- Pass Criteria: No high-value or indexed pages appear as orphans.
- Action: Integrate orphans into the main site architecture via hubs or footer links.
2. Click Depth Analysis
- Logic: Authority dilutes and crawl priority drops as depth increases.
- Check: Audit the "Crawl Depth" metric. For large sites, "Deep" might be 10+ clicks.
- Pass Criteria: High-traffic/priority pages are max 3–4 clicks from the Home page.
- Action: Flatten the architecture using intermediary hub pages.
3. Pagination & Infinite Scroll Validation
- Logic: Bots often fail to "click" to page 50 of a category.
- Check: Verify that
rel="next"andrel="prev"are implemented correctly (if still used) or that a "View All" page exists. - Pass Criteria: Google is discovering products on deep paginated pages (verify via Index Coverage report).
Phase 2: Anchor Consistency & Integrity
Goal: Ensure link signals are clear and the user experience is reliable.
1. First Link Priority & Consistency
- Logic: If a page links to a URL multiple times (e.g., via an image and a text link), the anchor text of the first link in the code usually carries the most weight.
- Check: Audit pages with multiple internal links to the same target.
- Pass Criteria: The first anchor text is descriptive (e.g., "Men's Running Shoes") rather than generic (e.g., "Buy Now").
2. Descriptive vs. Generic Anchors
- Logic: "Click Here" and "Read More" provide no semantic context to search engines.
- Check: Regex search for common generic terms in the HTML.
- Pass Criteria: Generic anchors account for <5% of total internal link equity.
- Action: Implement variable replacement in CMS to pull Article Titles instead of static "Read More" text.
3. Broken Links (404s) & Soft 404s
- Logic: Dead ends waste crawl budget and trap link equity.
- Check: Run a headless browser crawl to detect HTTP 404, 500, and timeout codes.
- Pass Criteria: Zero internal 404s. All outdated URLs 301 redirect to the nearest live equivalent.
Phase 3: Link Equity Flow & Nofollow Audits
Goal: Prevent "PageRank evaporation" and ensure open pathways for crawlers.
1. Internal Nofollow Audit
- Logic: Using
rel="nofollow"internally (often done to "sculpt" PageRank) is an outdated and harmful practice. It effectively burns link equity. - Check: Search code base for
rel="nofollow"pointing to internal URLs. - Pass Criteria: Zero internal
nofollowattributes. - Exception: Login pages, register links, or user-generated content (to prevent spam).
2. Canonical Cross-Linking
- Logic: Linking to a URL that redirects or canonicalizes to another URL slows crawling and confuses signals.
- Check: Do internal links point to the "Canonical" version of the URL?
- Pass Criteria: 100% of internal links point to the self-referencing canonical URL, not a parameter-based or redirecting URL.
3. Hreflang Return Tags
- Logic: While technical, these are links. If page A links to page B (EN to ES), page B must link back to page A.
- Check: Validate reciprocal hreflang tags.
- Pass Criteria: 1:1 return tag ratio across all localized pages.
Phase 4: Hub & Cannibalization Logic
Goal: Prevent pages from competing against each other and ensure hubs actually hold authority.
1. Duplicate Hub Detection (Keyword Cannibalization)
- Logic: If two "Hubs" target the same keyword, they split authority.
- Check: Map primary keywords to hub pages. Look for overlap.
- Pass Criteria: One unique primary keyword per Hub page. Use canonical tags or 301s to merge duplicates.
2. Silo Integrity
- Logic: Does a page about "Laptops" link to "Desktops" (broad topic) or does it stay within the "Portable Computers" silo?
- Check: Visualize site structure (treemap).
- Pass Criteria: Hubs link primarily to children and grandchildren, not sideways to unrelated categories.
Phase 5: Indexation Monitoring
Goal: Ensure the linking strategy results in actual indexed pages.
1. Crawl Budget Efficiency
- Logic: If Google crawls 10,000 pages but only indexes 2,000, your internal linking might be pointing to low-quality (thin) pages.
- Check: Compare Crawl Stats (GSC) vs. Index Count.
- Pass Criteria: High ratio of Indexed / Crawled pages for money keywords.
2. Parameter Linking
- Logic: Internal links pointing to URLs with session IDs (
?sessionid=123) or sorting parameters (?sort=price_asc) create infinite crawl loops. - Check: Audit internal link hrefs for
?or&. - Pass Criteria: Internal links are clean. Parameters are handled via Search Console parameter handling settings.
Phase 6: Dashboards for Ongoing Reporting
Goal: Move from "audit" to "monitoring." Use Looker Studio (GDS) or a similar BI tool.
Data Sources:
- Google Search Console: Index coverage, Internal Links report.
- Screaming Frog / Sitebulb (via API export): Status codes, anchor text, depth.
- Log File Analysis (e.g., Splunk, Oncrawl, Lumar): Actual bot hits.
- Google Analytics: User flow.
Dashboard 1: The "Health" Widget (Weekly)
- Internal 404 Count: Line chart (should be 0).
- Orphan Page Count: Line chart (should trend down).
- Avg. Click Depth: Gauge (e.g., avg 2.4 clicks).
Dashboard 2: The "Equity" Widget (Monthly)
- Internal Link Distribution: Bar chart showing Top 50 linked pages vs. Bottom 50.
- Anchor Cloud: Word cloud of top 100 anchor texts (watch for "click here" dominance).
- Nofollow Ratio: % of internal links currently nofollowed.
Dashboard 3: The "Indexation" Widget (Monthly)
- Crawled vs. Indexed: Comparison trend.
- Excluded by Noindex: Count of pages accidentally noindexed (often caused by template errors).
- Duplicate Hub Warnings: Alert if two pages begin ranking for the same exact-match term.