Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-opus-4-20250514-thinking-16k· AnthropicHugging Face LMArena Human Preference 140K

i want collect all news and use ChatGPT to give each news a significance score

  1. User

    i want collect all news and use ChatGPT to give each news a significance score

  2. Claude Assistant

    Here's a practical approach to building a news collection and significance scoring system:

    News Collection

    RSS Feeds: Most major news outlets provide RSS feeds. You can use libraries like:

    • Python: feedparser
    • JavaScript: rss-parser

    News APIs: Several services aggregate news from multiple sources:

    • NewsAPI.org (free tier available)
    • MediaStack
    • NewsData.io
    • Google News RSS feeds

    Web Scraping: For sources without APIs, you can use:

    • BeautifulSoup + Requests (Python)
    • Puppeteer (JavaScript)
    • Scrapy (Python framework)

    Significance Scoring with AI

    To score news significance, you could structure your prompts to include:

    Rate this news article's significance on a scale of 1-10 based on:
    - Global impact
    - Number of people affected
    - Long-term consequences
    - Economic implications
    - Political importance
    
    Article: [Your article text here]
    
    Return a JSON object with the score and brief reasoning.
    

    Implementation Architecture

    1. Data Pipeline:

      • Collect articles → Remove duplicates → Extract text content → Send to AI for scoring → Store results
    2. Database Structure:

      • Article ID, title, source, URL, publication date, full text
      • Significance score, scoring timestamp, AI reasoning
    3. Automation:

      • Run collection scripts on a schedule (hourly/daily)
      • Batch process articles to manage API costs
      • Set up alerts for high-significance stories

    Would you like me to elaborate on any specific part of this system, such as the technical implementation or the scoring criteria?

  3. User

    what do you recommend

  4. Claude Assistant

    For a practical news significance scoring system, I'd recommend this approach:

    Start Simple with NewsAPI + OpenAI

    Phase 1 - MVP

    • Use NewsAPI.org (free tier: 100 requests/day)
    • Focus on top headlines from major sources
    • Use OpenAI's API with a structured prompt
    • Store results in a simple SQLite database

    Why this approach:

    • NewsAPI handles the messy work of aggregating sources
    • You avoid legal issues with scraping
    • Quick to prototype (can build in a weekend)
    • Low initial cost

    Recommended Scoring Prompt

    prompt = f"""
    Analyze this news article and return a JSON with:
    - significance_score (1-10)
    - impact_scope (local/national/global)
    - categories (list: politics/economy/technology/health/environment)
    - time_sensitivity (breaking/developing/ongoing)
    - brief_reasoning (50 words max)
    
    Article Title: {title}
    Source: {source}
    Content: {content[:1000]}  # First 1000 chars to save tokens
    """
    

    Simple Python Implementation

    import requests
    import openai
    from datetime import datetime
    import sqlite3
    import json
    
    def collect_and_score_news():
        # Get news
        news_response = requests.get(
            'https://newsapi.org/v2/top-headlines',
            params={'country': 'us', 'apiKey': YOUR_KEY}
        )
        
        articles = news_response.json()['articles']
        
        # Score each article
        for article in articles:
            # Call OpenAI
            ai_response = openai.chat.completions.create(
                model="gpt-3.5-turbo",
                messages=[{"role": "user", "content": create_prompt(article)}],
                response_format={"type": "json_object"}
            )
            
            score_data = json.loads(ai_response.choices[0].message.content)
            
            # Save to database
            save_article_with_score(article, score_data)
    

    Cost Optimization Tips

    1. Filter before scoring: Only score articles from reputable sources
    2. Cache results: Don't re-score the same article
    3. Use GPT-3.5: Much cheaper than GPT-4, adequate for this task
    4. Summarize first: Use first 1000 characters instead of full text

    Future Enhancements

    Once working, you could:

    • Add more news sources via RSS feeds
    • Create a web dashboard to view results
    • Set up email alerts for 8+ significance scores
    • Track scoring accuracy by comparing to actual impact over time

    This approach gets you started quickly while keeping costs under $10/month initially. Would you like help with any specific part of the implementation?