Build a competitor SERP benchmark and delta tracking.
- User
Build a competitor SERP benchmark and delta tracking.
Assistant
The user wants a "competitor SERP benchmark and delta tracking" system. This is a request for a system design or a Python script to monitor Search Engine Results Pages (SERPs) for specific keywords, compare them against competitors, and track changes (deltas) over time.
1. Understand the Goal:
- SERP: Search Engine Results Page.
- Benchmark: Establish a baseline of where my site vs. competitors ranks for a set of keywords.
- Delta Tracking: Measure the change (delta) in rankings, visibility, or features (like snippets) over time.
- Competitor Focus: specifically comparing "My Site" vs. "Competitor A", "Competitor B", etc.
2. Identify Key Components:
- Input: List of keywords, Target Domain (Me), Competitor Domains, Location/Device settings.
- Data Fetching: Need a way to get SERP data. Scraping Google directly is hard/blocked. Using an API (like SerpApi, ValueSERP, or a mock for demonstration) is better.
- Data Processing: Parse the raw SERP JSON/HTML. Extract ranks for the target domains.
- Storage: Save the results with timestamps to calculate history/deltas (SQLite or CSV for a simple version).
- Analysis/Output:
- Benchmark: Average rank, Share of Voice (SOV), Presence in Top 10/20.
- Delta: Rank change (+/-), New keywords gained, Keywords lost.
- Visualization/Reporting: A table or simple chart.
3. Choose the Implementation approach:
- Language: Python is the industry standard for this.
- Libraries:
pandasfor data manipulation,requestsfor APIs,matplotlib/seabornfor visualization (optional but nice). - Structure:
- Configuration (API Key, Keywords, Domains).
- Fetcher Class (Abstracted, uses a mock or real API).
- Tracker Class (Orchestrates fetching and parsing).
- Analyzer Class (Calculates deltas and benchmarks).
- Main execution block.
4. Drafting the Solution (Mental Sandbox):
- Scraping vs. API: I should write the code to be API-agnostic or use a generic library structure, but for it to be runnable, I might need to simulate data or ask the user to plug in an API key. I will use a mock function for the SERP fetching so the code runs immediately, but add comments on how to swap it for
requests.get("https://serpapi.com..."). This is safer and more educational. - Data Structure:
Date,Keyword,Domain,Position,URL.
- Metrics:
- Average Position: Mean rank.
- Visibility Score: A weighted metric (e.g., position 1 gets 100 points, 2 gets 90, etc.). This is better than just rank.
- Delta: Today's Rank - Yesterday's Rank.
5. Refining the Output: The user asked to build it. This implies code. I will provide a Python script that does the following:
- Simulates fetching SERP data (so it's runnable).
- Calculates "Share of Voice" (SOV) as the benchmark metric.
- Calculates rank deltas (changes).
- Outputs a Pandas DataFrame summary.
6. Step-by-Step Code Construction:
- Imports:
pandas,datetime,random(for mock). - Mock Data Generator: Needs to generate slightly different results for "Today" vs "Yesterday" to show the delta.
- Core Logic:
- Function
get_serps(keyword)-> returns list of organic results. - Function
process_data(results)-> extracts ranks for specific domains. - Function
calculate_metrics(df)-> groups by date/domain, calculates visibility.
- Function
- Delta Logic: Merge the dataframe with itself on Keyword, shifted by date.
7. Adding Professional Polish:
- Include a "Share of Voice" calculation (crucial for benchmarking).
- Handle "Not Ranking" (None vs 0 vs 100).
- Format the output clearly.
8. Review against constraints:
- Competitor Benchmark: Yes, compares MySite vs CompetitorA.
- Delta Tracking: Yes, shows rank change.
9. Final Output Structure:
- Concept: Explain the architecture (ETL).
- The Code: A complete Python script.
- Explanation: How to interpret the output.
- Scaling: Advice on moving from script to production (APIs, Databases).
(Self-Correction during drafting): A common mistake is just averaging ranks. If I rank #1 for a low volume keyword and #100 for a high volume one, my "average" might look okay, but I'm losing. Correction: I'll stick to position tracking for simplicity in the code, but mention that weighted visibility is better for production. I will implement a simple Visibility Score (1/Position) in the script to make it robust.
10. Writing the actual response. (This aligns with the provided good response).
- Define the inputs: Keywords = ['seo tool', 'rank tracker'], Domains = ['me.com', 'them.com'].
- Create a class
SERPTracker. - Generate mock data for T-1 (Yesterday) and T (Today).
- Compute Delta.
- Print Report.
Let's refine the "Delta" part. Delta = Current Position - Previous Position. Negative delta = improved rank (went from 10 to 5, delta is -5). Positive delta = dropped rank. This often confuses people. I'll add a "Change" column that says "Up", "Down", or "New".
Let's write the Python code.
- Mock Data: Needs to be realistic.
- Benchmarking: Define benchmark as "Average Position" and "Top 10 Share".
Final Plan:
- Architecture Overview: Data flow diagram.
- Python Implementation: The actual script.
- Key Metrics: Explanation of what to track (Visibility vs Rank).
- Scaling: Moving to APIs.
This looks solid. I will generate the response based on this plan. To build a robust Competitor SERP (Search Engine Results Page) Benchmark and Delta Tracking system, you need three core components:
- Ingestion: Fetching SERP data (via API or Scraping).
- Normalization: Cleaning the data to create a "Benchmark" (standardizing URLs and ranks).
- Diff Engine: Comparing the current state against the previous state to find "Deltas" (changes).
Below is a complete architectural guide and a functional Python prototype you can run immediately.
Part 1: The Architecture
For a production-grade system, do not rely on manual scraping. Use a SERP API (like SerpApi, ValueSERP, or DataForSEO).
- Input: List of
KeywordsandCompetitor Domains. - Scheduler: Runs daily/weekly.
- Fetcher: Calls the SERP API.
- Database (Time-Series): Stores snapshots of results (PostgreSQL or BigQuery).
- Analyzer:
- Benchmark: Calculates Average Rank, Share of Voice (SOV), and SERP Feature ownership (Snippets, People Also Ask).
- Delta: Subtracts data from data to find rank changes, new entrants, and drop-offs.
Part 2: The Python Prototype
This script uses
pandasfor data manipulation. It includes a Mock SERP Fetcher so you can run it immediately to see the logic, but I have commented where you would plug in a real API.Prerequisites:
pip install pandasimport pandas as pd import random from datetime import datetime, timedelta # ========================================== # 1. CONFIGURATION # ========================================== MY_DOMAIN = "myshop.com" COMPETITORS = ["competitor_a.com", "competitor_b.net", "bigcompetitor.com"] KEYWORDS = ["best running shoes", "marathon training gear", "buy orthopedic insoles"] # ========================================== # 2. MOCK DATA GENERATOR (Replace with Real API) # ========================================== def get_serp_data_mock(keyword): """ Simulates a SERP API response. In production, replace this with a request to SerpApi/ValueSERP. """ domains = [MY_DOMAIN] + COMPETITORS + ["irrelevant_site.com", "wikipedia.org"] results = [] # Generate 10 organic results for i in range(1, 11): # Randomly assign a domain to a rank (with some bias to make it interesting) domain = random.choice(domains) # Occasionally simulate a SERP Feature (e.g., Featured Snippet) has_snippet = True if i == 1 and random.random() > 0.7 else False results.append({ "rank": i, "domain": domain, "url": f"https://{domain}/page-{i}", "title": f"Best info for {keyword}", "is_featured_snippet": has_snippet }) return results # ========================================== # 3. BENCHMARKING ENGINE # ========================================== def process_serp_results(keyword, results): """ Normalizes raw SERP data into a structured DataFrame. """ df = pd.DataFrame(results) # Add metadata df['keyword'] = keyword df['timestamp'] = datetime.now() # Filter only for us and our competitors targets = [MY_DOMAIN] + COMPETITORS df_targets = df[df['domain'].isin(targets)].copy() return df_targets def calculate_benchmarks(df_history): """ Calculates aggregate metrics (Average Rank, SOV) from historical data. """ # SOV Logic: Simple visibility score (1/Rank) df_history['visibility'] = df_history['rank'].apply(lambda x: 1/x if x > 0 else 0) benchmark = df_history.groupby('domain').agg({ 'rank': 'mean', 'visibility': 'sum' }).rename(columns={'rank': 'avg_rank', 'visibility': 'total_visibility'}) # Normalize visibility to % (Share of Voice) benchmark['sov_percent'] = (benchmark['total_visibility'] / benchmark['total_visibility'].sum()) * 100 return benchmark.round(2) # ========================================== # 4. DELTA TRACKING ENGINE # ========================================== def calculate_deltas(current_df, previous_df): """ Compares current rank vs previous rank. """ # Merge current and previous data on Keyword and Domain merged = pd.merge( current_df[['keyword', 'domain', 'rank']], previous_df[['keyword', 'domain', 'rank']], on=['keyword', 'domain'], how='outer', suffixes=('_current', '_previous') ) # Fill NaNs (New rankings or Dropped out) merged['rank_previous'] = merged['rank_previous'].fillna(100) # Treat new as coming from rank 100 merged['rank_current'] = merged['rank_current'].fillna(100) # Treat dropped as going to rank 100 # Calculate Delta merged['delta'] = merged['rank_previous'] - merged['rank_current'] # Positive = Rank Up, Negative = Rank Down # Status Tag def get_status(row): if row['rank_current'] == 100: return "Dropped Out" if row['rank_previous'] == 100: return "New Entry" if row['delta'] > 0: return "Improved" if row['delta'] < 0: return "Declined" return "Unchanged" merged['status'] = merged.apply(get_status, axis=1) return merged[merged['domain'].isin([MY_DOMAIN] + COMPETITORS)] # ========================================== # 5. EXECUTION FLOW # ========================================== def main(): print(f"--- Running SERP Tracker for: {datetime.now().strftime('%Y-%m-%d')} ---\n") # A. Fetch Data current_data = [] for kw in KEYWORDS: serp = get_serp_data_mock(kw) # Call API here processed = process_serp_results(kw, serp) current_data.append(processed) df_current = pd.concat(current_data) # B. Generate Fake Historical Data (for demonstration only) # In production, you would query your DB: "SELECT * FROM serps WHERE date = YESTERDAY" df_previous = df_current.copy() # Simulate changes in previous data to make deltas visible df_previous['rank'] = df_previous['rank'].apply(lambda x: x + random.randint(-2, 2)) df_previous = df_previous[df_previous['rank'] <= 10] # Drop some rows to simulate "New Entries" # C. Calculate Benchmarks (Aggregate view) print("=== BENCHMARK REPORT (Aggregate Visibility) ===") benchmark_report = calculate_benchmarks(df_current) print(benchmark_report[['avg_rank', 'sov_percent']]) print("\n") # D. Calculate Deltas (Change view) print("=== DELTA TRACKING (Rank Changes) ===") # Filtering for specific keyword for readability delta_report = calculate_deltas(df_current, df_previous) print(delta_report.sort_values(by=['keyword', 'delta'], ascending=[True, False]).to_string(index=False)) if __name__ == "__main__": main()
Part 3: Explanation of Key Metrics
When analyzing the output of the script above, focus on these three specific metrics to derive value:
1. Share of Voice (SOV)
The script calculates SOV based on
1/Rank.- Why: Rank #1 is not just "1 better" than Rank #2; it usually captures ~30-40% of clicks, while #2 gets ~15%.
- Benchmark: If your SOV is 20% and Competitor A is 40%, you are half as visible as them, regardless of specific keyword positions.
2. Rank Delta (The Delta)
- Positive (+): You moved up (e.g., from 5 to 2).
- Negative (-): You moved down.
- "New Entry": You appeared in the top 10 (or top 100) where you weren't before.
- "Dropped Out": You fell off the first page. This is a critical alert signal.
3. SERP Feature Tracking
In the
process_serp_resultsfunction, I included a placeholder foris_featured_snippet.- Competitor Benchmark: If Competitor A holds the Featured Snippet for "best running shoes," they effectively own Rank 0.
- Delta: If you had the snippet yesterday and Competitor A took it today, that is a massive visibility loss, even if your organic rank stayed at #2.
Part 4: Moving to Production
To turn this script into a dashboard:
- Storage: Use a simple SQL schema.
CREATE TABLE serp_history ( id SERIAL PRIMARY KEY, keyword VARCHAR(255), domain VARCHAR(255), rank INT, date TIMESTAMP ); - Automation:
- Run the script daily via a Cron job or Airflow DAG.
- Save the results to the DB immediately after fetching.
- Alerting:
- Add logic:
if delta < -3: send_slack_alert() - Add logic:
if competitor captures featured_snippet: send_email()
- Add logic: