Skip to content
TheChatArchiveThe Chat Archive
GLMGLM-5 / Pony Alpha· unknownHugging Face Frontier Reasoning Traces

Build a cannibalization detector: query mapping and resolution.

  1. User

    Build a cannibalization detector: query mapping and resolution.

  2. GLM Assistant

    The user wants me to build a "cannibalization detector" specifically for "query mapping and resolution."

    1. Deconstruct the Request:

    • Core Concept: Keyword Cannibalization. This happens when multiple pages on a website rank for the same (or very similar) keywords, causing them to compete against each other. This can lead to lower overall rankings, confusing search engines about which page is the "authoritative" one, and lower conversion rates.
    • Specific Tasks:
      • Query Mapping: Identifying which search queries are sending traffic to which URLs.
      • Resolution: Determining if there is a conflict and suggesting how to fix it (merge, canonicalize, differentiate).
    • Goal: Build a system (or a conceptual framework/code logic) to detect this.

    2. Determine the Approach:

    Since I am an AI text model, I cannot physically crawl a live website in real-time or access a user's private Google Search Console (GSC) data directly unless they provide it.

    Therefore, I need to provide a solution in the form of:

    • Conceptual Framework: Explain how it works.
    • Python Script (The "Builder" part): A robust Python script that takes input data (like a CSV export from GSC or an SEO tool) and performs the detection logic.
    • Heuristics/Logic: Define the rules that flag cannibalization.

    3. Developing the Logic (The "Brain"):

    • Input: We need a dataset containing: Query, Page (URL), Clicks, Impressions, Position, CTR.
    • Step 1: Grouping: Group data by Query.
    • Step 2: Detection:
      • If a single query maps to > 1 URL -> Potential Cannibalization.
      • Refinement: Just appearing isn't enough. Are both pages actually competing?
        • Thresholds: Is the second page close to the first in position? (e.g., Page A is #4, Page B is #12. This is less critical than A at #4 and B at #6).
        • Volume: Does the query have enough impressions to matter?
    • Step 3: Resolution Strategy (The "Advice"):
      • If Page A is clearly superior (better content, higher conversion), keep it.
      • If Page B is old/thin, 301 redirect to A or set canonical.
      • If Page A and B cover slightly different intent, differentiate the keyword targeting.

    4. Drafting the Solution Structure:

    • Introduction: Briefly define cannibalization.
    • Methodology: The 3-step process (Data Collection -> Clustering -> Analysis).
    • The Python Tool: This is the core. I'll use pandas.
      • Mock Data: Create a small dummy dataset so the user can run the code immediately.
      • Code Logic:
        1. Load data.
        2. Group by query.
        3. Filter for queries with > 1 URL.
        4. Calculate "Cannibalization Score" (optional but cool) or just sort by URL count.
        5. Determine the "Dominant URL" (highest clicks/impressions).
      • Output: A report showing the conflict.
    • Resolution Framework: A guide on how to actually fix the issues found (301s, Canonicals, Content differentiation).
    • Advanced/Enterprise approach: Mention embeddings (Vector databases) for semantic cannibalization (detecting "cheap shoes" vs "shoes low cost").

    5. Refining the Python Code (Mental Sandbox):

    • Input: gsc_data.csv
    • DataFrame: df
    • Operation: groupby('query')
    • Filter: nunique('url') > 1
    • Logic:
      • Identify the "Winner" (Best avg position or most clicks).
      • Identify the "Cannibal" (The competing page).
      • Calculate Position Delta. If delta is small (< 10), it's high urgency.

    6. Adding "Semantic" Detection (The "Advanced" part):

    Standard GSC data only shows exact string matches. True cannibalization is often semantic (e.g., "buy laptop" and "laptop deals"). I should mention using TF-IDF or Embeddings (OpenAI/HuggingFace) to map similar queries together, but for a "builder" script, stick to the structured data approach first to keep it runnable and understandable.

    7. Final Review of the Response:

    • Does it answer the specific prompt? Yes.
    • Is the code copy-pasteable? Yes.
    • Is the business logic sound? Yes.

    Self-Correction during drafting:

    • Initial thought: Scrape Google search results.
    • Correction: Too unstable, liable to get blocked, and slow. Better to use internal data (GSC) which reflects actual cannibalization happening to the user's site, not just theoretical ranking.

    8. Constructing the Output: (This aligns with the structured response provided previously).

    • Phase 1: The Logic (How we define the problem).
    • Phase 2: The Python Script (The tool).
    • Phase 3: The Resolution (How to fix it).

    Let's write the code snippet now to ensure it works mentally. df.groupby('query').filter(lambda x: x['url'].nunique() > 1) -> This gets the right rows. Then aggregate to show the conflicting URLs.

    Let's refine the output report. It should list:

    1. Query
    2. URL A (Position, Clicks)
    3. URL B (Position, Clicks)
    4. Recommended Action.

    Action Plan:

    1. Define the Strategy: Exact match (GSC data) vs Semantic match (Vector search). I will focus on Exact Match from GSC as the MVP.
    2. Write the Python Script: Use Pandas.
    3. Write the Resolution Guide: 301, Canonical, Internal Linking, Content differentiation.

    This looks solid. I will structure the answer clearly with these sections. A Keyword Cannibalization Detector identifies when multiple pages on your website compete for the same search queries, effectively eating into each other's traffic and confusing search engines.

    Below is a complete guide and a Python solution to build this detector using Google Search Console (GSC) data.


    Phase 1: The Logic (Query Mapping)

    To detect cannibalization, we must map Queries →\rightarrow URLs.

    Cannibalization typically occurs in two forms:

    1. Exact Match: Two URLs rank for the exact same keyword (e.g., both /blog/seo-tips and /services/seo rank for "seo tips").
    2. Semantic Overlap: Two URLs rank for slightly different variations of the same intent (e.g., "how to fix shoe" and "shoe repair guide").

    The Detection Heuristic: We will look for instances where:

    1. A single Query triggers more than 1 URL from your domain.
    2. The URLs are within a close Position range (e.g., both are on Page 1 or top of Page 2).
    3. The Click distribution is fragmented (neither gets the full potential traffic).

    Phase 2: The Python Detector

    This script assumes you have exported your data from Google Search Console (or are using the Google Search Console API).

    Prerequisites:

    pip install pandas
    

    The Script (cannibalization_detector.py):

    import pandas as pd
    
    class CannibalizationDetector:
        def __init__(self, csv_file_path):
            """
            Initialize with path to GSC export CSV.
            Expected columns: 'Query', 'Page', 'Clicks', 'Impressions', 'Position', 'CTR'
            """
            try:
                self.df = pd.read_csv(csv_file_path)
                # Standardize column names just in case
                self.df.columns = [c.lower() for c in self.df.columns]
                self.rename_map = {
                    'page': 'url',
                    'landing page': 'url',
                    'top queries': 'query',
                    'search query': 'query'
                }
                self.df.rename(columns=self.rename_map, inplace=True)
                
                print(f"Loaded {len(self.df)} rows of GSC data.")
            except Exception as e:
                print(f"Error loading data: {e}")
    
        def detect_cannibalization(self, position_threshold=20, min_clicks=5):
            """
            Identifies cannibalization issues.
            
            :param position_threshold: Max position to consider (e.g., 20 means only look at top 20 results)
            :param min_clicks: Minimum total clicks for a query to be considered significant.
            :return: DataFrame containing conflicting URL pairs.
            """
            
            # 1. Filter out poor performing keywords first
            df_filtered = self.df[self.df['position'] <= position_threshold].copy()
            
            # 2. Group by Query to find how many URLs rank for it
            query_group = df_filtered.groupby('query').agg({
                'url': 'nunique',  # Count unique URLs
                'clicks': 'sum'
            }).reset_index()
            
            # 3. Filter queries that have more than 1 URL (Potential Cannibalization)
            cannibal_queries = query_group[
                (query_group['url'] > 1) & 
                (query_group['clicks'] >= min_clicks)
            ]['query'].tolist()
            
            # 4. Isolate the rows causing the conflict
            conflicts = df_filtered[df_filtered['query'].isin(cannibal_queries)]
            
            # 5. Format the output for analysis
            # We want to see the Query, the URLs involved, and their individual metrics
            conflict_report = conflicts.sort_values(['query', 'clicks'], ascending=[True, False])
            
            # Pivot to see side-by-side comparison (Optional advanced view)
            # For now, we return a clean sorted list
            
            return conflict_report[['query', 'url', 'clicks', 'impressions', 'position', 'ctr']]
    
        def resolve_strategy(self, query, url_list):
            """
            Simple logic to suggest resolution strategy.
            """
            # Sort by Clicks (Descending) to find the 'Winner'
            sorted_urls = url_list.sort_values(by='clicks', ascending=False)
            winner = sorted_urls.iloc[0]
            losers = sorted_urls.iloc[1:]
            
            print(f"\n--- Strategy for Query: '{query}' ---")
            print(f"WINNER: {winner['url']} ({winner['clicks']} clicks)")
            print("COMPETITORS (Candidates for merging/canonicalizing):")
            for _, row in losers.iterrows():
                print(f"- {row['url']} ({row['clicks']} clicks)")
    
    # ==========================================
    # Example Usage
    # ==========================================
    
    # 1. Create dummy data (In real use, load your GSC Export)
    data = {
        'query': ['running shoes', 'running shoes', 'running shoes', 'marathon training', 'marathon training', 'buy socks'],
        'page': ['https://site.com/running-shoes', 'https://site.com/blog/best-running-shoes', 'https://site.com/shoe-guide', 'https://site.com/marathon', 'https://site.com/training', 'https://site.com/socks'],
        'clicks': [100, 20, 5, 50, 40, 10],
        'impressions': [1000, 500, 200, 800, 750, 100],
        'position': [1.5, 8.2, 15.0, 5.0, 5.5, 3.0],
        'ctr': [0.10, 0.04, 0.025, 0.06, 0.05, 0.10]
    }
    df = pd.DataFrame(data)
    df.to_csv('gsc_data.csv', index=False) # Simulating a file load
    
    # 2. Run Detector
    detector = CannibalizationDetector('gsc_data.csv')
    report = detector.detect_cannibalization(position_threshold=20, min_clicks=1)
    
    print("\n=== CANNIBALIZATION REPORT ===")
    print(report)
    
    # 3. Run Resolution Strategy on a specific keyword
    # Extract data for specific query
    query_data = df[df['query'] == 'running shoes']
    detector.resolve_strategy('running shoes', query_data)
    

    Phase 3: Resolution Framework

    Once the script identifies the conflicts, you must resolve them. Here is the decision matrix:

    1. The "Intent" Check

    Before merging, check Google manually.

    • Scenario A: You type the query and see both your pages on Page 1.
      • Verdict: True Cannibalization. You are splitting your own CTR.
    • Scenario B: You type the query and see one page on Page 1, the other on Page 3.
      • Verdict: False Alarm (usually). Google prefers one page. The other is irrelevant noise.

    2. Resolution Actions

    ScenarioStrategyAction Item
    Same IntentCanonicalizationIf Page A and Page B are about "SEO Tools" but A is better, add <link rel="canonical" href="Page A"> to the <head> of Page B.
    Subset ContentContent Pruning / 301 RedirectIf Page B is a short blog post about a topic fully covered in Page A's comprehensive guide, 301 Redirect Page B to Page A. This transfers link equity.
    Different IntentKeyword DifferentiationIf Page A sells "Blue Widgets" and Page B is "How to paint widgets blue", ensure the on-page SEO is distinct. Change the H1 of Page B to ensure Google sees it as informational, not transactional.
    Faceted NavNoIndex / Robots.txtIf the URLs are generated by filters (e.g., /shoes?color=red vs /shoes?color=blue), apply noindex to the parameter pages.

    Phase 4: Automation Strategy (Advanced)

    If you want to build this into a fully automated pipeline:

    1. Data Source: Connect to the Google Search Console API directly instead of uploading CSVs.
    2. Clustering: Use a sentence-transformer model (like BERT) to cluster semantically similar queries.
      • Current Script Logic: Matches "running shoes" to "running shoes".
      • Advanced Logic: Matches "running shoes" to "footwear for running".
    3. Scheduler: Run the script weekly.
    4. Alerting: If a new URL enters the top 20 for a keyword that an existing URL already owns, send a Slack alert to the SEO team.