Skip to content
TheChatArchiveThe Chat Archive
GLMGLM-5 / Pony Alpha· unknownHugging Face Frontier Reasoning Traces

Create an XML/video/image sitemaps sanitation job with alerts.

  1. User

    Create an XML/video/image sitemaps sanitation job with alerts.

  2. GLM Assistant

    The user wants a "sitemaps sanitation job with alerts." This is a fairly broad request. "Sanitation" in this context usually means checking for broken links, incorrect formats, accessibility issues, non-indexable content, etc., and then alerting someone if things go wrong.

    I need to determine the best way to fulfill this. I cannot run a job directly on the user's server. I can provide:

    1. A conceptual architecture/plan.
    2. A script (Python is best for this) that performs the checks.
    3. Instructions on how to set up the alerts (e.g., via email, Slack, or a logging system).
    4. Instructions on how to schedule it (e.g., Cron, Airflow, GitHub Actions).

    Scope of the Job: The user mentioned "XML/video/image sitemaps."

    • XML: Standard URL sitemaps.
    • Video: Requires checking video-specific tags (video:content_loc, video:thumbnail_loc) and ensuring media isn't 404ing.
    • Image: Requires checking image-specific tags (image:loc).

    Sanitation Checks needed:

    1. XML Validity: Is it parsable XML?
    2. URL Validity (Loc): Do the main URLs return 200 status codes?
    3. Media Validity: Do the images/videos return 200?
    4. Sitemap Index: Does it point to valid child sitemaps?
    5. Size Limits: Is the file too big (>50MB uncompressed, >50k URLs)?
    6. Formatting: Are required namespaces present?

    Alerting Mechanism:

    • Simple: Email (SMTP).
    • Modern: Webhooks (Slack/Teams/Discord).
    • Developer-focused: stdout/exit codes (for CI/CD).

    Deliverable Strategy: I will write a Python script. It's the standard tool for this.

    • Libraries: requests, xml.etree.ElementTree (or lxml), defusedxml (security).
    • Structure:
      • SitemapChecker class.
      • Methods to fetch and parse.
      • Methods to validate URLs.
      • Alerting function.
      • Main execution block.

    Refining the Output: The user might just want the code, but a "job" implies scheduling. I should explain how to run it as a job (e.g., Cron).

    Let's structure the response:

    1. Overview: What the script does.
    2. Prerequisites: Python packages.
    3. The Python Script: Comprehensive, handling XML, Image, and Video namespaces.
    4. Alerting: A simple Slack/Email webhook implementation.
    5. Scheduling: How to make it a "job" (Cron/GitHub Actions).

    Drafting the Script Logic:

    • Input: URL of the sitemap (or sitemap index).
    • Step 1: Fetch content.
    • Step 2: Parse XML. Catch ParseErrors.
    • Step 3: Extract URLs.
      • Standard <loc>
      • Image <image:loc>
      • Video <video:content_loc>, <video:player_loc>
      • Video Thumbnails <video:thumbnail_loc>
    • Step 4: Head request (faster than GET) to check status. Store non-200s.
    • Step 5: Aggregate errors.
    • Step 6: Send Alert if errors exist.

    Self-Correction/Safety:

    • Rate limiting: Checking thousands of URLs might DDOS the target or get us banned. I should add a small time.sleep or concurrency limit (though simple synchronous is safer for a basic snippet, asyncio or threading makes it faster but more complex). I'll stick to synchronous with a small delay to keep the script readable and safe, but mention threading for production.
    • Namespaces: XML sitemaps rely heavily on namespaces. urlset vs sitemapindex. Standard xmlns vs image/video xmlns. I need to handle namespaces robustly in the parsing logic.

    The Alerting Part: I'll provide a modular send_alert function. I'll simulate a webhook post as it's the most common modern integration.

    Final Plan:

    1. Python Script: A robust sitemap_sanitizer.py.
    2. Configuration: Variables at the top for Sitemap URL and Webhook.
    3. Deployment: Instructions for Cron and Docker.

    Let's write the code mentally first.

    import requests
    import xml.etree.ElementTree as ET
    namespaces = {
        'sm': 'http://www.sitemaps.org/schemas/sitemap/0.9',
        'image': 'http://www.google.com/schemas/sitemap-image/1.1',
        'video': 'http://www.google.com/schemas/sitemap-video/1.1'
    }
    # Loop through 'sm:url'
    # extract 'sm:loc', 'image:loc', 'video:content_loc'
    # check status code
    # if bad, add to list.
    

    Looks solid. I will add a limit on checks so the demo doesn't run forever if the user has a 50k sitemap.

    Let's assemble the response. This solution provides a comprehensive Python-based sitemap sanitation tool. It checks for XML validity, HTTP status codes (404s, 500s), and specific media tags (Video/Image).

    It is designed to be run as a scheduled job (e.g., via Cron or GitHub Actions) and sends alerts to a Slack channel or Email.

    1. Prerequisites

    You will need Python 3 installed. Install the required libraries:

    pip install requests
    

    2. The Sanitation Script (sitemap_job.py)

    This script handles sitemap parsing, URL checking, and alerting. It supports standard Namespaces for Images and Video.

    import requests
    import xml.etree.ElementTree as ET
    import time
    import sys
    from urllib.parse import urlparse
    from datetime import datetime
    
    # --- CONFIGURATION ---
    CONFIG = {
        # The sitemap to check (can be a sitemap index)
        "SITEMAP_URL": "https://www.example.com/sitemap.xml",
        
        # How many URLs to check (set to None to check all, but be careful with rate limits)
        "MAX_URLS_TO_CHECK": 50, 
        
        # Pause between requests to avoid rate limiting (seconds)
        "REQUEST_DELAY": 0.5, 
        
        # Alerting Threshold (Only alert if this many errors are found)
        "ERROR_THRESHOLD": 1, 
    
        # Webhook URLs (Set to None if not used)
        "SLACK_WEBHOOK_URL": "https://hooks.slack.com/services/YOUR/WEBHOOK/HERE",
        "EMAIL_API_KEY": None, # Placeholder for email service (Sendgrid/Mailgun) logic
    }
    
    # XML Namespaces commonly used in sitemaps
    NAMESPACES = {
        'sm': 'http://www.sitemaps.org/schemas/sitemap/0.9',
        'image': 'http://www.google.com/schemas/sitemap-image/1.1',
        'video': 'http://www.google.com/schemas/sitemap-video/1.1'
    }
    
    class SitemapAuditor:
        def __init__(self):
            self.errors = []
            self.checked_count = 0
            self.headers = {'User-Agent': 'SitemapSanitationBot/1.0'}
    
        def run(self):
            print(f"Starting Sanitation Job: {datetime.now()}")
            print(f"Target: {CONFIG['SITEMAP_URL']}")
            
            try:
                self.process_sitemap(CONFIG['SITEMAP_URL'])
                self.report()
            except Exception as e:
                self.send_critical_alert(f"Job failed to complete: {str(e)}")
    
        def process_sitemap(self, url):
            """Recursively processes sitemaps and sitemap indexes."""
            try:
                response = requests.get(url, headers=self.headers, timeout=10)
                if response.status_code != 200:
                    self.add_error(url, f"Sitemap fetch failed with status {response.status_code}")
                    return
    
                root = ET.fromstring(response.content)
                
                # Check if this is a Sitemap Index (contains other sitemaps)
                # We strip namespaces loosely to handle varying declarations
                is_index = any('sitemapindex' in tag for tag in root.tag if isinstance(tag, str))
                
                if is_index:
                    print(f"Processing Sitemap Index: {url}")
                    for sitemap in root.findall('.//sm:loc', NAMESPACES):
                        self.process_sitemap(sitemap.text)
                else:
                    print(f"Processing URL Set: {url}")
                    self.check_urls(root)
    
            except ET.ParseError:
                self.add_error(url, "Invalid XML format (Parse Error)")
            except Exception as e:
                self.add_error(url, f"Processing Exception: {str(e)}")
    
        def check_urls(self, root):
            """Checks individual URLs and their media content."""
            # Find all URL entries
            urls = root.findall('.//sm:url', NAMESPACES)
            
            limit = CONFIG['MAX_URLS_TO_CHECK']
            
            for i, url_node in enumerate(urls):
                if limit and i >= limit:
                    print("URL Check limit reached.")
                    break
    
                # 1. Check Main Page URL
                loc = url_node.find('sm:loc', NAMESPACES)
                if loc is not None:
                    self.audit_url(loc.text, is_resource=False)
                
                # 2. Check Image URLs
                for img in url_node.findall('image:image', NAMESPACES):
                    img_loc = img.find('image:loc', NAMESPACES)
                    if img_loc is not None:
                        self.audit_url(img_loc.text, is_resource=True)
    
                # 3. Check Video Content/Thumbnail URLs
                for vid in url_node.findall('video:video', NAMESPACES):
                    # Check Thumbnail
                    thumb = vid.find('video:thumbnail_loc', NAMESPACES)
                    if thumb is not None:
                        self.audit_url(thumb.text, is_resource=True)
                    
                    # Check Content Location
                    content = vid.find('video:content_loc', NAMESPACES)
                    if content is not None:
                        self.audit_url(content.text, is_resource=True)
                    
                    # Check Player Location (can be tricky, usually returns 200 but requires embed logic)
                    player = vid.find('video:player_loc', NAMESPACES)
                    if player is not None:
                        # We usually just check if the player URL is reachable (200/301)
                        self.audit_url(player.text, is_resource=True)
    
                time.sleep(CONFIG['REQUEST_DELAY'])
    
        def audit_url(self, url, is_resource=False):
            """Performs a HEAD request to check status."""
            try:
                # Using HEAD request to save bandwidth
                r = requests.head(url, headers=self.headers, timeout=10, allow_redirects=True)
                
                # Logic: 200 is good. 3xx is followed. 
                # 4xx (Client Error) and 5xx (Server Error) are bad.
                # For images/video, we strictly want 200.
                
                status = r.status_code
                
                if status >= 400:
                    item_type = "Resource" if is_resource else "Page"
                    self.add_error(url, f"{item_type} broken (Status: {status})")
                
                self.checked_count += 1
                
            except requests.exceptions.RequestException as e:
                self.add_error(url, f"Connection Error: {str(e)}")
    
        def add_error(self, url, message):
            self.errors.append({"url": url, "msg": message})
            print(f"[ERROR] {url}: {message}")
    
        def report(self):
            print(f"Job Complete. Checked {self.checked_count} objects.")
            
            if len(self.errors) >= CONFIG['ERROR_THRESHOLD']:
                self.send_alerts()
            else:
                print("No significant errors detected.")
    
        def send_alerts(self):
            message = f"🚨 *Sitemap Sanitation Failed* ({len(self.errors)} issues found)\n"
            message += f"Target: `{CONFIG['SITEMAP_URL']}`\n\n"
            
            for err in self.errors[:10]: # Limit alert to first 10 errors to avoid huge messages
                message += f"• <{err['url']}|{err['url']}>\n   _{err['msg']}_\n"
                
            if len(self.errors) > 10:
                message += f"_...and {len(self.errors) - 10} more errors._"
    
            # --- Send to Slack ---
            if CONFIG['SLACK_WEBHOOK_URL']:
                try:
                    payload = {"text": message}
                    requests.post(CONFIG['SLACK_WEBHOOK_URL'], json=payload)
                    print("Slack Alert Sent.")
                except Exception as e:
                    print(f"Failed to send Slack alert: {e}")
    
            # --- Send Email (Placeholder) ---
            if CONFIG['EMAIL_API_KEY']:
                print("Email logic would trigger here.")
    
        def send_critical_alert(self, msg):
            print(f"CRITICAL: {msg}")
            if CONFIG['SLACK_WEBHOOK_URL']:
                requests.post(CONFIG['SLACK_WEBHOOK_URL'], json={"text": f"🚨 Critical Job Failure: {msg}"})
    
    if __name__ == "__main__":
        job = SitemapAuditor()
        job.run()
    

    3. How to Schedule This Job

    To make this a "Job" rather than a script you run manually, you need to schedule it.

    Option A: Cron (Linux/Mac)

    1. Open your terminal.
    2. Type crontab -e.
    3. Add the following line to run the job every day at 3:00 AM:
    0 3 * * * /usr/bin/python3 /path/to/sitemap_job.py >> /var/log/sitemap_job.log 2>&1
    

    Option B: GitHub Actions (Cloud/Free)

    This is excellent for maintenance as it runs in the cloud for free.

    1. Create a file in your repo at .github/workflows/sitemap_check.yml.
    2. Paste the following configuration:
    name: Sitemap Sanitation
    
    on:
      schedule:
        # Runs daily at 3:00 AM UTC
        - cron: '0 3 * * *'
      workflow_dispatch: # Allows manual triggering
    
    jobs:
      check-sitemaps:
        runs-on: ubuntu-latest
        
        # Add your Webhook URL as a Secret in Repository Settings -> Secrets -> Actions
        env:
          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK_URL }}
    
        steps:
          - uses: actions/checkout@v3
          
          - name: Set up Python
            uses: actions/setup-python@v4
            with:
              python-version: '3.9'
              
          - name: Install dependencies
            run: pip install requests
            
          - name: Run Sanitation Job
            run: python sitemap_job.py
    

    4. Logic Explained

    1. Namespaces (Video/Image): The script defines the image and video XML schemas. This allows it to dig into <video:content_loc> tags to ensure the actual MP4 file isn't returning a 404, which is a common issue in video SEO.
    2. HEAD Requests: The script uses the HTTP HEAD method. This downloads the headers but not the body (content). This saves significant bandwidth when checking thousands of images or videos.
    3. Sitemap Index Support: It detects if the input URL is a "Sitemap Index" (a list of other sitemaps). If it is, it recursively fetches and checks the child sitemaps.
    4. Rate Limiting: The REQUEST_DELAY variable is crucial. If you check 5,000 URLs simultaneously, your server might block the script for looking like a DDoS attack. The delay prevents this.
    5. Thresholds: If you have a massive site, one random 404 might not be an emergency. The ERROR_THRESHOLD ensures you only get woken up if a significant issue occurs (like a whole directory failing).