Create an XML/video/image sitemaps sanitation job with alerts.
- User
Create an XML/video/image sitemaps sanitation job with alerts.
Assistant
The user wants a "sitemaps sanitation job with alerts." This is a fairly broad request. "Sanitation" in this context usually means checking for broken links, incorrect formats, accessibility issues, non-indexable content, etc., and then alerting someone if things go wrong.
I need to determine the best way to fulfill this. I cannot run a job directly on the user's server. I can provide:
- A conceptual architecture/plan.
- A script (Python is best for this) that performs the checks.
- Instructions on how to set up the alerts (e.g., via email, Slack, or a logging system).
- Instructions on how to schedule it (e.g., Cron, Airflow, GitHub Actions).
Scope of the Job: The user mentioned "XML/video/image sitemaps."
- XML: Standard URL sitemaps.
- Video: Requires checking video-specific tags (video:content_loc, video:thumbnail_loc) and ensuring media isn't 404ing.
- Image: Requires checking image-specific tags (image:loc).
Sanitation Checks needed:
- XML Validity: Is it parsable XML?
- URL Validity (Loc): Do the main URLs return 200 status codes?
- Media Validity: Do the images/videos return 200?
- Sitemap Index: Does it point to valid child sitemaps?
- Size Limits: Is the file too big (>50MB uncompressed, >50k URLs)?
- Formatting: Are required namespaces present?
Alerting Mechanism:
- Simple: Email (SMTP).
- Modern: Webhooks (Slack/Teams/Discord).
- Developer-focused: stdout/exit codes (for CI/CD).
Deliverable Strategy: I will write a Python script. It's the standard tool for this.
- Libraries:
requests,xml.etree.ElementTree(orlxml),defusedxml(security). - Structure:
SitemapCheckerclass.- Methods to fetch and parse.
- Methods to validate URLs.
- Alerting function.
- Main execution block.
Refining the Output: The user might just want the code, but a "job" implies scheduling. I should explain how to run it as a job (e.g., Cron).
Let's structure the response:
- Overview: What the script does.
- Prerequisites: Python packages.
- The Python Script: Comprehensive, handling XML, Image, and Video namespaces.
- Alerting: A simple Slack/Email webhook implementation.
- Scheduling: How to make it a "job" (Cron/GitHub Actions).
Drafting the Script Logic:
- Input: URL of the sitemap (or sitemap index).
- Step 1: Fetch content.
- Step 2: Parse XML. Catch ParseErrors.
- Step 3: Extract URLs.
- Standard
<loc> - Image
<image:loc> - Video
<video:content_loc>,<video:player_loc> - Video Thumbnails
<video:thumbnail_loc>
- Standard
- Step 4: Head request (faster than GET) to check status. Store non-200s.
- Step 5: Aggregate errors.
- Step 6: Send Alert if errors exist.
Self-Correction/Safety:
- Rate limiting: Checking thousands of URLs might DDOS the target or get us banned. I should add a small
time.sleepor concurrency limit (though simple synchronous is safer for a basic snippet,asyncioor threading makes it faster but more complex). I'll stick to synchronous with a small delay to keep the script readable and safe, but mention threading for production. - Namespaces: XML sitemaps rely heavily on namespaces.
urlsetvssitemapindex. Standard xmlns vs image/video xmlns. I need to handle namespaces robustly in the parsing logic.
The Alerting Part: I'll provide a modular
send_alertfunction. I'll simulate a webhook post as it's the most common modern integration.Final Plan:
- Python Script: A robust
sitemap_sanitizer.py. - Configuration: Variables at the top for Sitemap URL and Webhook.
- Deployment: Instructions for Cron and Docker.
Let's write the code mentally first.
import requests import xml.etree.ElementTree as ET namespaces = { 'sm': 'http://www.sitemaps.org/schemas/sitemap/0.9', 'image': 'http://www.google.com/schemas/sitemap-image/1.1', 'video': 'http://www.google.com/schemas/sitemap-video/1.1' } # Loop through 'sm:url' # extract 'sm:loc', 'image:loc', 'video:content_loc' # check status code # if bad, add to list.Looks solid. I will add a limit on checks so the demo doesn't run forever if the user has a 50k sitemap.
Let's assemble the response. This solution provides a comprehensive Python-based sitemap sanitation tool. It checks for XML validity, HTTP status codes (404s, 500s), and specific media tags (Video/Image).
It is designed to be run as a scheduled job (e.g., via Cron or GitHub Actions) and sends alerts to a Slack channel or Email.
1. Prerequisites
You will need Python 3 installed. Install the required libraries:
pip install requests2. The Sanitation Script (
sitemap_job.py)This script handles sitemap parsing, URL checking, and alerting. It supports standard Namespaces for Images and Video.
import requests import xml.etree.ElementTree as ET import time import sys from urllib.parse import urlparse from datetime import datetime # --- CONFIGURATION --- CONFIG = { # The sitemap to check (can be a sitemap index) "SITEMAP_URL": "https://www.example.com/sitemap.xml", # How many URLs to check (set to None to check all, but be careful with rate limits) "MAX_URLS_TO_CHECK": 50, # Pause between requests to avoid rate limiting (seconds) "REQUEST_DELAY": 0.5, # Alerting Threshold (Only alert if this many errors are found) "ERROR_THRESHOLD": 1, # Webhook URLs (Set to None if not used) "SLACK_WEBHOOK_URL": "https://hooks.slack.com/services/YOUR/WEBHOOK/HERE", "EMAIL_API_KEY": None, # Placeholder for email service (Sendgrid/Mailgun) logic } # XML Namespaces commonly used in sitemaps NAMESPACES = { 'sm': 'http://www.sitemaps.org/schemas/sitemap/0.9', 'image': 'http://www.google.com/schemas/sitemap-image/1.1', 'video': 'http://www.google.com/schemas/sitemap-video/1.1' } class SitemapAuditor: def __init__(self): self.errors = [] self.checked_count = 0 self.headers = {'User-Agent': 'SitemapSanitationBot/1.0'} def run(self): print(f"Starting Sanitation Job: {datetime.now()}") print(f"Target: {CONFIG['SITEMAP_URL']}") try: self.process_sitemap(CONFIG['SITEMAP_URL']) self.report() except Exception as e: self.send_critical_alert(f"Job failed to complete: {str(e)}") def process_sitemap(self, url): """Recursively processes sitemaps and sitemap indexes.""" try: response = requests.get(url, headers=self.headers, timeout=10) if response.status_code != 200: self.add_error(url, f"Sitemap fetch failed with status {response.status_code}") return root = ET.fromstring(response.content) # Check if this is a Sitemap Index (contains other sitemaps) # We strip namespaces loosely to handle varying declarations is_index = any('sitemapindex' in tag for tag in root.tag if isinstance(tag, str)) if is_index: print(f"Processing Sitemap Index: {url}") for sitemap in root.findall('.//sm:loc', NAMESPACES): self.process_sitemap(sitemap.text) else: print(f"Processing URL Set: {url}") self.check_urls(root) except ET.ParseError: self.add_error(url, "Invalid XML format (Parse Error)") except Exception as e: self.add_error(url, f"Processing Exception: {str(e)}") def check_urls(self, root): """Checks individual URLs and their media content.""" # Find all URL entries urls = root.findall('.//sm:url', NAMESPACES) limit = CONFIG['MAX_URLS_TO_CHECK'] for i, url_node in enumerate(urls): if limit and i >= limit: print("URL Check limit reached.") break # 1. Check Main Page URL loc = url_node.find('sm:loc', NAMESPACES) if loc is not None: self.audit_url(loc.text, is_resource=False) # 2. Check Image URLs for img in url_node.findall('image:image', NAMESPACES): img_loc = img.find('image:loc', NAMESPACES) if img_loc is not None: self.audit_url(img_loc.text, is_resource=True) # 3. Check Video Content/Thumbnail URLs for vid in url_node.findall('video:video', NAMESPACES): # Check Thumbnail thumb = vid.find('video:thumbnail_loc', NAMESPACES) if thumb is not None: self.audit_url(thumb.text, is_resource=True) # Check Content Location content = vid.find('video:content_loc', NAMESPACES) if content is not None: self.audit_url(content.text, is_resource=True) # Check Player Location (can be tricky, usually returns 200 but requires embed logic) player = vid.find('video:player_loc', NAMESPACES) if player is not None: # We usually just check if the player URL is reachable (200/301) self.audit_url(player.text, is_resource=True) time.sleep(CONFIG['REQUEST_DELAY']) def audit_url(self, url, is_resource=False): """Performs a HEAD request to check status.""" try: # Using HEAD request to save bandwidth r = requests.head(url, headers=self.headers, timeout=10, allow_redirects=True) # Logic: 200 is good. 3xx is followed. # 4xx (Client Error) and 5xx (Server Error) are bad. # For images/video, we strictly want 200. status = r.status_code if status >= 400: item_type = "Resource" if is_resource else "Page" self.add_error(url, f"{item_type} broken (Status: {status})") self.checked_count += 1 except requests.exceptions.RequestException as e: self.add_error(url, f"Connection Error: {str(e)}") def add_error(self, url, message): self.errors.append({"url": url, "msg": message}) print(f"[ERROR] {url}: {message}") def report(self): print(f"Job Complete. Checked {self.checked_count} objects.") if len(self.errors) >= CONFIG['ERROR_THRESHOLD']: self.send_alerts() else: print("No significant errors detected.") def send_alerts(self): message = f"🚨 *Sitemap Sanitation Failed* ({len(self.errors)} issues found)\n" message += f"Target: `{CONFIG['SITEMAP_URL']}`\n\n" for err in self.errors[:10]: # Limit alert to first 10 errors to avoid huge messages message += f"• <{err['url']}|{err['url']}>\n _{err['msg']}_\n" if len(self.errors) > 10: message += f"_...and {len(self.errors) - 10} more errors._" # --- Send to Slack --- if CONFIG['SLACK_WEBHOOK_URL']: try: payload = {"text": message} requests.post(CONFIG['SLACK_WEBHOOK_URL'], json=payload) print("Slack Alert Sent.") except Exception as e: print(f"Failed to send Slack alert: {e}") # --- Send Email (Placeholder) --- if CONFIG['EMAIL_API_KEY']: print("Email logic would trigger here.") def send_critical_alert(self, msg): print(f"CRITICAL: {msg}") if CONFIG['SLACK_WEBHOOK_URL']: requests.post(CONFIG['SLACK_WEBHOOK_URL'], json={"text": f"🚨 Critical Job Failure: {msg}"}) if __name__ == "__main__": job = SitemapAuditor() job.run()3. How to Schedule This Job
To make this a "Job" rather than a script you run manually, you need to schedule it.
Option A: Cron (Linux/Mac)
- Open your terminal.
- Type
crontab -e. - Add the following line to run the job every day at 3:00 AM:
0 3 * * * /usr/bin/python3 /path/to/sitemap_job.py >> /var/log/sitemap_job.log 2>&1Option B: GitHub Actions (Cloud/Free)
This is excellent for maintenance as it runs in the cloud for free.
- Create a file in your repo at
.github/workflows/sitemap_check.yml. - Paste the following configuration:
name: Sitemap Sanitation on: schedule: # Runs daily at 3:00 AM UTC - cron: '0 3 * * *' workflow_dispatch: # Allows manual triggering jobs: check-sitemaps: runs-on: ubuntu-latest # Add your Webhook URL as a Secret in Repository Settings -> Secrets -> Actions env: SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK_URL }} steps: - uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4 with: python-version: '3.9' - name: Install dependencies run: pip install requests - name: Run Sanitation Job run: python sitemap_job.py4. Logic Explained
- Namespaces (Video/Image): The script defines the
imageandvideoXML schemas. This allows it to dig into<video:content_loc>tags to ensure the actual MP4 file isn't returning a 404, which is a common issue in video SEO. - HEAD Requests: The script uses the HTTP
HEADmethod. This downloads the headers but not the body (content). This saves significant bandwidth when checking thousands of images or videos. - Sitemap Index Support: It detects if the input URL is a "Sitemap Index" (a list of other sitemaps). If it is, it recursively fetches and checks the child sitemaps.
- Rate Limiting: The
REQUEST_DELAYvariable is crucial. If you check 5,000 URLs simultaneously, your server might block the script for looking like a DDoS attack. The delay prevents this. - Thresholds: If you have a massive site, one random 404 might not be an emergency. The
ERROR_THRESHOLDensures you only get woken up if a significant issue occurs (like a whole directory failing).