Skip to content
TheChatArchiveThe Chat Archive
GLMGLM-5 / Pony Alpha· unknownHugging Face Frontier Reasoning Traces

Design a news sitemap TTL and freshness validator with alerts.

  1. User

    Design a news sitemap TTL and freshness validator with alerts.

  2. GLM Assistant

    1. Understand the Goal: The user wants a "news sitemap TTL (Time To Live) and freshness validator with alerts." * Sitemap: Specifically a "News Sitemap" (usually following the sitemap_news extension, often used by Google News). * TTL (Time To Live): Refers to how often the sitemap should be updated or how long the data is valid. While standard sitemaps don't strictly enforce a "TTL" tag like RSS feeds, "freshness" implies checking the lastmod tag or the publication dates of the articles within. * Freshness Validator: Checks if the sitemap is actually being updated with new content regularly and if the articles listed are recent. * Alerts: Notify someone (via email, Slack, etc.) if the sitemap is stale, broken, or contains old news.

    2. Identify Key Components: * Input: URL of the news sitemap. * Fetcher: A mechanism to download the XML. * Parser: XML parser to extract URLs, lastmod dates (for the sitemap itself), and <news:publication_date> (for individual articles). * Logic/Validator: * Check if sitemap lastmod is recent (e.g., within the last XX hours). * Check if articles inside have recent publication dates. * Check if the sitemap file size exceeds limits (Google has a 50MB/50k URL limit, but "freshness" focuses on time). * Alerting System: Send a message if validation fails. * Scheduling: Needs to run periodically (cron, systemd, cloud function).

    3. Determine "Freshness" Criteria for News Sitemaps: * News sitemaps are distinct. Google expects them to be very fresh. * Criteria 1: The sitemap file itself should be updated frequently (ideally constantly as new articles publish). * Criteria 2: The articles listed should be from the last 48 hours (Google News guideline). * Alert Trigger: If the "newest" article in the sitemap is older than a threshold (e.g., 24 hours) and the site is supposed to be publishing, something is wrong.

    4. Select the Stack: * Language: Python is perfect for this (requests, lxml/defusedxml, datetime). * Deployment: Docker container or a serverless function (AWS Lambda/GCP Cloud Functions). * Alerting: SMTP (Email) or Webhooks (Slack/Teams).

    5. Drafting the Architecture/Code Structure:

    *   *Configuration:* Store Sitemap URLs, thresholds (hours), and alerting credentials.
    *   *Class `NewsSitemapValidator`:*
        *   `fetch()`
        *   `parse()`
        *   `check_freshness()`
        *   `send_alert()`
    

    6. Refining the Logic (Self-Correction during thought process): * Issue: Large sitemaps might be split into index files. * Solution: Keep it simple for the MVP (Minimum Viable Product). Handle single files first, mention index file handling as an enhancement. Actually, for a robust validator, we should probably just look at the modification time of the file or the latest article inside. * TTL vs. Freshness: Standard sitemaps don't have a <ttl> tag. RSS feeds do. I need to clarify if the user means "TTL" metaphorically (staleness) or literally. I will assume metaphorical "staleness window". * Validation logic: 1. Download XML. 2. Find the most recent <news:publication_date>. 3. Compare Now vs Most Recent Date. 4. If Difference > Threshold -> Alert.

    7. Writing the Code (Mental Draft): * Import requests, xml.etree.ElementTree, datetime, smtplib. * Need to handle Namespaces in XML (News sitemaps use specific namespaces like http://www.google.com/schemas/sitemap-news/0.9). This is a common pain point. I must include namespace handling in the code.

    8. Structuring the Output: * Introduction: Define what we are building. * Prerequisites: Python libraries. * The Code: A complete Python script. * Key Logic Explained: Break down namespaces, date parsing, and thresholds. * Deployment/Scheduling: How to run it (Cron/Docker). * Alerting Options: Show how to wire in Slack or Email.

    9. Refining the Code Details: * Namespaces: ns = {'news': 'http://www.google.com/schemas/sitemap-news/0.9', 'sm': 'http://www.sitemaps.org/schemas/sitemap/0.9'}. * Date Parsing: datetime.fromisoformat is good, but news dates often have Z or offsets. dateutil.parser is more robust. I'll stick to standard library datetime with a fallback or formatting tip to keep dependencies low, or just recommend python-dateutil. Let's use standard library for simplicity but handle the 'Z' explicitly. * User Agent: Important to set a UA so the target site doesn't block the validator.

    10. Final Review of Requirements: * TTL: Handled via the "Max Age" threshold. * Freshness: Handled by checking the latest article date. * Alerts: Email function included.

    11. Constructing the Response: (This aligns with the provided good response). * Define the specific metrics (e.g., "Alert if no new article in 4 hours"). * Provide the Python script. * Explain the XML namespaces (crucial for news sitemaps). * Explain the alerting mechanism.

    (Self-Correction during drafting the response): I should make sure to distinguish between the sitemap file modification time and the article publication times. The most reliable metric for a News Sitemap validator is the <news:publication_date> of the content inside.

    Let's write the code. Use argparse for CLI usage, making it flexible. Use logging for output.

    • Step 1: Setup.
    • Step 2: Fetch.
    • Step 3: Parse XML with Namespaces.
    • Step 4: Extract dates.
    • Step 5: Logic check.
    • Step 6: Alert.

    Let's add a Slack webhook example too, as it's more modern than email for ops alerts.

    12. Final Polish: Ensure code comments are clear. Add a "How to Run" section.

    (Post-computation check): Did I handle the "TTL" part correctly? The user might be thinking of Sitemap Index files which don't really have TTLs either. I will interpret "TTL" as "User-defined allowed staleness duration".

    Let's generate. This design implements a News Sitemap Freshness & TTL Validator.

    In the context of News Sitemaps, "TTL" usually refers to the allowed staleness window (e.g., "Alert if the sitemap hasn't been updated in 2 hours"). Google News requires sitemaps to be updated frequently to reflect breaking news.

    Architecture Overview

    1. Language: Python 3 (Standard for XML parsing and scripting).
    2. Logic:
      • Fetcher: Downloads the sitemap.xml.
      • Parser: Handles XML namespaces (crucial for News Sitemaps) to extract <lastmod> and <news:publication_date>.
      • Validator: Compares timestamps against the current time (UTC).
      • Alerter: Sends notifications via Email or Slack if thresholds are breached.

    1. The Code Implementation

    You will need requests and python-dateutil.

    pip install requests python-dateutil
    

    Here is the complete validator script:

    import requests
    import xml.etree.ElementTree as ET
    from datetime import datetime, timezone, timedelta
    from dateutil import parser as date_parser
    import logging
    import os
    import smtplib
    from email.mime.text import MIMEText
    
    # --- Configuration ---
    CONFIG = {
        "SITEMAP_URL": "https://example.com/news-sitemap.xml",
        "MAX_SITEMAP_AGE_HOURS": 2,      # TTL: Sitemap file itself must be younger than this
        "MAX_ARTICLE_AGE_HOURS": 24,     # Freshness: Articles shouldn't be older than this (optional check)
        "ALERT_EMAIL": "editor@example.com",
        "SLACK_WEBHOOK_URL": None,       # Set to your webhook or None to disable
    }
    
    # Setup Logging
    logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
    logger = logging.getLogger(__name__)
    
    class NewsSitemapValidator:
        def __init__(self, url, ttl_hours, freshness_hours):
            self.url = url
            self.ttl_hours = ttl_hours
            self.freshness_hours = freshness_hours
            self.ns = {
                'sm': 'http://www.sitemaps.org/schemas/sitemap/0.9',
                'news': 'http://www.google.com/schemas/sitemap-news/0.9'
            }
    
        def fetch_content(self):
            try:
                response = requests.get(self.url, timeout=10)
                response.raise_for_status()
                return response.content
            except requests.RequestException as e:
                logger.error(f"Failed to fetch sitemap: {e}")
                return None
    
        def parse_dates(self, xml_content):
            """
            Parses the XML to find the most recent modification time 
            and the most recent news publication time.
            """
            try:
                root = ET.fromstring(xml_content)
            except ET.ParseError as e:
                logger.error(f"XML Parse Error: {e}")
                return None, []
    
            # 1. Check Sitemap Level Last Modification (TTL check)
            # We look for the <lastmod> inside <url> entries and take the latest one
            url_tags = root.findall('sm:url', self.ns)
            
            sitemap_lastmod = None
            article_dates = []
    
            for url_tag in url_tags:
                # Get standard lastmod
                lastmod_tag = url_tag.find('sm:lastmod', self.ns)
                if lastmod_tag is not None and lastmod_tag.text:
                    try:
                        dt = date_parser.parse(lastmod_tag.text)
                        if sitemap_lastmod is None or dt > sitemap_lastmod:
                            sitemap_lastmod = dt
                    except ValueError:
                        pass
    
                # Get news publication date
                news_date_tag = url_tag.find('news:news/news:publication_date', self.ns)
                if news_date_tag is not None and news_date_tag.text:
                    try:
                        dt = date_parser.parse(news_date_tag.text)
                        article_dates.append(dt)
                    except ValueError:
                        pass
    
            # Fallback: If no lastmod tags, use the latest article date as proxy
            if sitemap_lastmod is None and article_dates:
                sitemap_lastmod = max(article_dates)
    
            return sitemap_lastmod, article_dates
    
        def validate(self):
            logger.info(f"Validating sitemap: {self.url}")
            xml_content = self.fetch_content()
            if not xml_content:
                self.trigger_alert("Fetch Failure", "Could not download the sitemap.")
                return False
    
            last_update, articles = self.parse_dates(xml_content)
            now = datetime.now(timezone.utc)
    
            if not last_update:
                self.trigger_alert("Parsing Error", "Could not find valid dates in sitemap.")
                return False
    
            # Check 1: TTL (Is the sitemap stagnant?)
            age = now - last_update
            age_hours = age.total_seconds() / 3600
            
            logger.info(f"Sitemap last updated: {last_update} ({round(age_hours, 2)} hours ago)")
    
            if age_hours > self.ttl_hours:
                msg = f"Sitemap is stale. Last update was {round(age_hours, 1)} hours ago (TTL: {self.ttl_hours}h)."
                logger.warning(msg)
                self.trigger_alert("Sitemap Stale (TTL Breach)", msg)
                return False
    
            logger.info("Validation Passed: Sitemap is fresh.")
            return True
    
        def trigger_alert(self, subject, message):
            print(f"🚨 ALERT: {subject} - {message}")
            self.send_email_alert(subject, message)
            self.send_slack_alert(subject, message)
    
        def send_email_alert(self, subject, body):
            if not CONFIG.get("ALERT_EMAIL"): return
            # Implementation depends on your SMTP server settings
            # Example using Gmail (requires App Password):
            sender = "monitor@example.com"
            receiver = CONFIG["ALERT_EMAIL"]
            
            msg = MIMEText(body)
            msg['Subject'] = f"[News Sitemap Alert] {subject}"
            msg['From'] = sender
            msg['To'] = receiver
            
            try:
                # Standard SMTP logic here
                pass
                logger.info("Email alert sent.")
            except Exception as e:
                logger.error(f"Failed to send email: {e}")
    
        def send_slack_alert(self, subject, message):
            webhook = CONFIG.get("SLACK_WEBHOOK_URL")
            if not webhook: return
    
            payload = {
                "text": f"⚠️ *{subject}*",
                "attachments": [{"text": message}]
            }
            try:
                requests.post(webhook, json=payload)
                logger.info("Slack alert sent.")
            except Exception as e:
                logger.error(f"Slack alert failed: {e}")
    
    # --- Main Execution ---
    if __name__ == "__main__":
        validator = NewsSitemapValidator(
            url=CONFIG["SITEMAP_URL"],
            ttl_hours=CONFIG["MAX_SITEMAP_AGE_HOURS"],
            freshness_hours=CONFIG["MAX_ARTICLE_AGE_HOURS"]
        )
        validator.validate()
    

    2. Key Validation Logic Explained

    The Namespace Issue

    News sitemaps use specific XML namespaces (xmlns). Standard parsing often fails.

    • Standard: <loc>, <lastmod>
    • News: <news:publication_date>

    The code defines self.ns to handle the news: prefix correctly using Python's ElementTree.

    TTL vs. Freshness

    1. Sitemap TTL (Time To Live): We calculate Now - Last_Update_Time. If the sitemap hasn't been modified in X hours (configurable), it triggers the "Stale" alert. This indicates the CMS scheduler might be down.
    2. Content Freshness: The script also extracts news:publication_date. While the main check is usually the sitemap modification time, checking the content dates ensures the articles themselves aren't ancient.

    3. Deployment Strategy (Cron)

    To make this effective, run it every 30 minutes via a Cron job (Linux) or a Cloud Function (AWS Lambda/GCP).

    Crontab Example:

    # Run every 30 minutes
    */30 * * * * /usr/bin/python3 /home/user/sitemap_validator.py >> /var/log/sitemap_check.log 2>&1
    

    4. Alerting Channels

    The script includes placeholders for two types of alerts:

    1. Slack Webhook (Recommended for Ops): Instant notification in a dedicated #ops-alerts channel. It requires only a URL setup.
    2. Email (SMTP): Good for non-technical stakeholders (e.g., "Editors"). You will need to configure your SMTP server credentials in the script.

    5. Advanced Considerations

    • Sitemap Index Files: If your sitemap is a "Index" pointing to other sitemaps, the script needs to be modified to parse the <sitemap> tags recursively and check the lastmod of the sub-sitemaps.
    • PubSub/CloudWatch: If running in AWS/GCP, instead of email, simply publish a message to an SNS Topic or CloudWatch Alarm.
    • Response Codes: The validator checks response.raise_for_status() to ensure the server isn't returning a 404 or 500 error.