Design a news sitemap TTL and freshness validator with alerts.
- User
Design a news sitemap TTL and freshness validator with alerts.
Assistant
1. Understand the Goal: The user wants a "news sitemap TTL (Time To Live) and freshness validator with alerts." * Sitemap: Specifically a "News Sitemap" (usually following the
sitemap_newsextension, often used by Google News). * TTL (Time To Live): Refers to how often the sitemap should be updated or how long the data is valid. While standard sitemaps don't strictly enforce a "TTL" tag like RSS feeds, "freshness" implies checking thelastmodtag or the publication dates of the articles within. * Freshness Validator: Checks if the sitemap is actually being updated with new content regularly and if the articles listed are recent. * Alerts: Notify someone (via email, Slack, etc.) if the sitemap is stale, broken, or contains old news.2. Identify Key Components: * Input: URL of the news sitemap. * Fetcher: A mechanism to download the XML. * Parser: XML parser to extract URLs,
lastmoddates (for the sitemap itself), and<news:publication_date>(for individual articles). * Logic/Validator: * Check if sitemaplastmodis recent (e.g., within the last hours). * Check if articles inside have recent publication dates. * Check if the sitemap file size exceeds limits (Google has a 50MB/50k URL limit, but "freshness" focuses on time). * Alerting System: Send a message if validation fails. * Scheduling: Needs to run periodically (cron, systemd, cloud function).3. Determine "Freshness" Criteria for News Sitemaps: * News sitemaps are distinct. Google expects them to be very fresh. * Criteria 1: The sitemap file itself should be updated frequently (ideally constantly as new articles publish). * Criteria 2: The articles listed should be from the last 48 hours (Google News guideline). * Alert Trigger: If the "newest" article in the sitemap is older than a threshold (e.g., 24 hours) and the site is supposed to be publishing, something is wrong.
4. Select the Stack: * Language: Python is perfect for this (requests, lxml/defusedxml, datetime). * Deployment: Docker container or a serverless function (AWS Lambda/GCP Cloud Functions). * Alerting: SMTP (Email) or Webhooks (Slack/Teams).
5. Drafting the Architecture/Code Structure:
* *Configuration:* Store Sitemap URLs, thresholds (hours), and alerting credentials. * *Class `NewsSitemapValidator`:* * `fetch()` * `parse()` * `check_freshness()` * `send_alert()`6. Refining the Logic (Self-Correction during thought process): * Issue: Large sitemaps might be split into index files. * Solution: Keep it simple for the MVP (Minimum Viable Product). Handle single files first, mention index file handling as an enhancement. Actually, for a robust validator, we should probably just look at the modification time of the file or the latest article inside. * TTL vs. Freshness: Standard sitemaps don't have a
<ttl>tag. RSS feeds do. I need to clarify if the user means "TTL" metaphorically (staleness) or literally. I will assume metaphorical "staleness window". * Validation logic: 1. Download XML. 2. Find the most recent<news:publication_date>. 3. CompareNowvsMost Recent Date. 4. IfDifference > Threshold-> Alert.7. Writing the Code (Mental Draft): * Import
requests,xml.etree.ElementTree,datetime,smtplib. * Need to handle Namespaces in XML (News sitemaps use specific namespaces likehttp://www.google.com/schemas/sitemap-news/0.9). This is a common pain point. I must include namespace handling in the code.8. Structuring the Output: * Introduction: Define what we are building. * Prerequisites: Python libraries. * The Code: A complete Python script. * Key Logic Explained: Break down namespaces, date parsing, and thresholds. * Deployment/Scheduling: How to run it (Cron/Docker). * Alerting Options: Show how to wire in Slack or Email.
9. Refining the Code Details: * Namespaces:
ns = {'news': 'http://www.google.com/schemas/sitemap-news/0.9', 'sm': 'http://www.sitemaps.org/schemas/sitemap/0.9'}. * Date Parsing:datetime.fromisoformatis good, but news dates often haveZor offsets.dateutil.parseris more robust. I'll stick to standard librarydatetimewith a fallback or formatting tip to keep dependencies low, or just recommendpython-dateutil. Let's use standard library for simplicity but handle the 'Z' explicitly. * User Agent: Important to set a UA so the target site doesn't block the validator.10. Final Review of Requirements: * TTL: Handled via the "Max Age" threshold. * Freshness: Handled by checking the latest article date. * Alerts: Email function included.
11. Constructing the Response: (This aligns with the provided good response). * Define the specific metrics (e.g., "Alert if no new article in 4 hours"). * Provide the Python script. * Explain the XML namespaces (crucial for news sitemaps). * Explain the alerting mechanism.
(Self-Correction during drafting the response): I should make sure to distinguish between the sitemap file modification time and the article publication times. The most reliable metric for a News Sitemap validator is the
<news:publication_date>of the content inside.Let's write the code. Use
argparsefor CLI usage, making it flexible. Useloggingfor output.- Step 1: Setup.
- Step 2: Fetch.
- Step 3: Parse XML with Namespaces.
- Step 4: Extract dates.
- Step 5: Logic check.
- Step 6: Alert.
Let's add a Slack webhook example too, as it's more modern than email for ops alerts.
12. Final Polish: Ensure code comments are clear. Add a "How to Run" section.
(Post-computation check): Did I handle the "TTL" part correctly? The user might be thinking of Sitemap Index files which don't really have TTLs either. I will interpret "TTL" as "User-defined allowed staleness duration".
Let's generate. This design implements a News Sitemap Freshness & TTL Validator.
In the context of News Sitemaps, "TTL" usually refers to the allowed staleness window (e.g., "Alert if the sitemap hasn't been updated in 2 hours"). Google News requires sitemaps to be updated frequently to reflect breaking news.
Architecture Overview
- Language: Python 3 (Standard for XML parsing and scripting).
- Logic:
- Fetcher: Downloads the
sitemap.xml. - Parser: Handles XML namespaces (crucial for News Sitemaps) to extract
<lastmod>and<news:publication_date>. - Validator: Compares timestamps against the current time (UTC).
- Alerter: Sends notifications via Email or Slack if thresholds are breached.
- Fetcher: Downloads the
1. The Code Implementation
You will need
requestsandpython-dateutil.pip install requests python-dateutilHere is the complete validator script:
import requests import xml.etree.ElementTree as ET from datetime import datetime, timezone, timedelta from dateutil import parser as date_parser import logging import os import smtplib from email.mime.text import MIMEText # --- Configuration --- CONFIG = { "SITEMAP_URL": "https://example.com/news-sitemap.xml", "MAX_SITEMAP_AGE_HOURS": 2, # TTL: Sitemap file itself must be younger than this "MAX_ARTICLE_AGE_HOURS": 24, # Freshness: Articles shouldn't be older than this (optional check) "ALERT_EMAIL": "editor@example.com", "SLACK_WEBHOOK_URL": None, # Set to your webhook or None to disable } # Setup Logging logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') logger = logging.getLogger(__name__) class NewsSitemapValidator: def __init__(self, url, ttl_hours, freshness_hours): self.url = url self.ttl_hours = ttl_hours self.freshness_hours = freshness_hours self.ns = { 'sm': 'http://www.sitemaps.org/schemas/sitemap/0.9', 'news': 'http://www.google.com/schemas/sitemap-news/0.9' } def fetch_content(self): try: response = requests.get(self.url, timeout=10) response.raise_for_status() return response.content except requests.RequestException as e: logger.error(f"Failed to fetch sitemap: {e}") return None def parse_dates(self, xml_content): """ Parses the XML to find the most recent modification time and the most recent news publication time. """ try: root = ET.fromstring(xml_content) except ET.ParseError as e: logger.error(f"XML Parse Error: {e}") return None, [] # 1. Check Sitemap Level Last Modification (TTL check) # We look for the <lastmod> inside <url> entries and take the latest one url_tags = root.findall('sm:url', self.ns) sitemap_lastmod = None article_dates = [] for url_tag in url_tags: # Get standard lastmod lastmod_tag = url_tag.find('sm:lastmod', self.ns) if lastmod_tag is not None and lastmod_tag.text: try: dt = date_parser.parse(lastmod_tag.text) if sitemap_lastmod is None or dt > sitemap_lastmod: sitemap_lastmod = dt except ValueError: pass # Get news publication date news_date_tag = url_tag.find('news:news/news:publication_date', self.ns) if news_date_tag is not None and news_date_tag.text: try: dt = date_parser.parse(news_date_tag.text) article_dates.append(dt) except ValueError: pass # Fallback: If no lastmod tags, use the latest article date as proxy if sitemap_lastmod is None and article_dates: sitemap_lastmod = max(article_dates) return sitemap_lastmod, article_dates def validate(self): logger.info(f"Validating sitemap: {self.url}") xml_content = self.fetch_content() if not xml_content: self.trigger_alert("Fetch Failure", "Could not download the sitemap.") return False last_update, articles = self.parse_dates(xml_content) now = datetime.now(timezone.utc) if not last_update: self.trigger_alert("Parsing Error", "Could not find valid dates in sitemap.") return False # Check 1: TTL (Is the sitemap stagnant?) age = now - last_update age_hours = age.total_seconds() / 3600 logger.info(f"Sitemap last updated: {last_update} ({round(age_hours, 2)} hours ago)") if age_hours > self.ttl_hours: msg = f"Sitemap is stale. Last update was {round(age_hours, 1)} hours ago (TTL: {self.ttl_hours}h)." logger.warning(msg) self.trigger_alert("Sitemap Stale (TTL Breach)", msg) return False logger.info("Validation Passed: Sitemap is fresh.") return True def trigger_alert(self, subject, message): print(f"🚨 ALERT: {subject} - {message}") self.send_email_alert(subject, message) self.send_slack_alert(subject, message) def send_email_alert(self, subject, body): if not CONFIG.get("ALERT_EMAIL"): return # Implementation depends on your SMTP server settings # Example using Gmail (requires App Password): sender = "monitor@example.com" receiver = CONFIG["ALERT_EMAIL"] msg = MIMEText(body) msg['Subject'] = f"[News Sitemap Alert] {subject}" msg['From'] = sender msg['To'] = receiver try: # Standard SMTP logic here pass logger.info("Email alert sent.") except Exception as e: logger.error(f"Failed to send email: {e}") def send_slack_alert(self, subject, message): webhook = CONFIG.get("SLACK_WEBHOOK_URL") if not webhook: return payload = { "text": f"⚠️ *{subject}*", "attachments": [{"text": message}] } try: requests.post(webhook, json=payload) logger.info("Slack alert sent.") except Exception as e: logger.error(f"Slack alert failed: {e}") # --- Main Execution --- if __name__ == "__main__": validator = NewsSitemapValidator( url=CONFIG["SITEMAP_URL"], ttl_hours=CONFIG["MAX_SITEMAP_AGE_HOURS"], freshness_hours=CONFIG["MAX_ARTICLE_AGE_HOURS"] ) validator.validate()2. Key Validation Logic Explained
The Namespace Issue
News sitemaps use specific XML namespaces (
xmlns). Standard parsing often fails.- Standard:
<loc>,<lastmod> - News:
<news:publication_date>
The code defines
self.nsto handle thenews:prefix correctly using Python'sElementTree.TTL vs. Freshness
- Sitemap TTL (Time To Live):
We calculate
Now - Last_Update_Time. If the sitemap hasn't been modified in X hours (configurable), it triggers the "Stale" alert. This indicates the CMS scheduler might be down. - Content Freshness:
The script also extracts
news:publication_date. While the main check is usually the sitemap modification time, checking the content dates ensures the articles themselves aren't ancient.
3. Deployment Strategy (Cron)
To make this effective, run it every 30 minutes via a Cron job (Linux) or a Cloud Function (AWS Lambda/GCP).
Crontab Example:
# Run every 30 minutes */30 * * * * /usr/bin/python3 /home/user/sitemap_validator.py >> /var/log/sitemap_check.log 2>&14. Alerting Channels
The script includes placeholders for two types of alerts:
- Slack Webhook (Recommended for Ops):
Instant notification in a dedicated
#ops-alertschannel. It requires only a URL setup. - Email (SMTP): Good for non-technical stakeholders (e.g., "Editors"). You will need to configure your SMTP server credentials in the script.
5. Advanced Considerations
- Sitemap Index Files: If your sitemap is a "Index" pointing to other sitemaps, the script needs to be modified to parse the
<sitemap>tags recursively and check thelastmodof the sub-sitemaps. - PubSub/CloudWatch: If running in AWS/GCP, instead of email, simply publish a message to an SNS Topic or CloudWatch Alarm.
- Response Codes: The validator checks
response.raise_for_status()to ensure the server isn't returning a 404 or 500 error.