Skip to content
The Chat Archive
gpt-4o-search-preview-high-loc· OpenAI LMArena Search Arena 24K

help me fix my script: import requests from youtubetranscriptapi import…

  1. User

    help me fix my script:

    import requests from youtube_transcript_api import YouTubeTranscriptApi import json import time from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry

    Configure retries for requests

    retry_strategy = Retry( total=3, backoff_factor=1, status_forcelist=[408, 429, 500, 502, 503, 504], ) adapter = HTTPAdapter(max_retries=retry_strategy) http = requests.Session() http.mount("http://", adapter) http.mount("https://", adapter)

    def get_video_id(url): """Extract video ID from YouTube URL""" if "v=" in url: return url.split("v=")[1][:11] return None

    def get_transcript(video_id): """Get transcript from YouTube video""" try: transcript_list = YouTubeTranscriptApi.get_transcript(video_id) return " ".join([item['text'] for item in transcript_list]) except Exception as e: return f"Error getting transcript: {str(e)}"

    def chunk_text(text, chunk_size=1000): # Reduced from 4000 to 2000 """Split text into chunks of approximately chunk_size characters""" words = text.split() chunks = [] current_chunk = [] current_length = 0

    for word in words:
        if current_length + len(word) + 1 <= chunk_size:
            current_chunk.append(word)
            current_length += len(word) + 1
        else:
            chunks.append(' '.join(current_chunk))
            current_chunk = [word]
            current_length = len(word)
            
    if current_chunk:
        chunks.append(' '.join(current_chunk))
    return chunks
    

    def get_summary(text): """Get summary using local API""" try: chunks = chunk_text(text) summaries = [] total_chunks = len(chunks)

        print(f"\nProcessing {total_chunks} chunks...")
        
        # Initial analysis to get main topic
        topic_data = {
            "model": "goekdenizguelmez/JOSIEFIED-Qwen2.5:3b",
            "messages": [
                {
                    "role": "system",
                    "content": "Identify the main topic of this text in 5-10 words."
                },
                {
                    "role": "user", 
                    "content": text[:2000]  # Use first chunk for topic identification
                }
            ],
            "stream": False,
        }
        
        try:
            topic_response = http.post(
                "http://localhost:11434/api/chat",
                headers={"Content-Type": "application/json"},
                json=topic_data,
                timeout=60
            )
            main_topic = topic_response.json()['message']['content']
        except:
            main_topic = "the discussed topic"
        
        # Process each chunk with structured format
        summary_template = f"""
    

    Here's a breakdown of the key points and arguments presented in the video about {main_topic}:

    1. Key Points Analysis: [Main Arguments]: Identify and summarize the main arguments [Supporting Evidence]: List any evidence or examples provided [Context]: Provide relevant context or background information

    2. Detailed Breakdown: [Technical Details]: Explain any technical concepts [Practical Applications]: Describe real-world applications [Challenges/Limitations]: Discuss any challenges or limitations mentioned

    3. Additional Insights: [Expert Opinions]: Summarize any expert views or citations [Future Implications]: Discuss potential future impacts [Related Considerations]: Include any related topics or considerations

    Overall Takeaway: [Provide a concise summary of the main message and implications] """

        for i, chunk in enumerate(chunks, 1):
            print(f"\rProcessing chunk {i}/{total_chunks}...", end="", flush=True)
            
            headers = {"Content-Type": "application/json"}
            data = {
                "model": "goekdenizguelmez/JOSIEFIED-Qwen2.5:3b",
                "messages": [
                    {
                        "role": "system",
                        "content": f"Analyze this text segment and provide a detailed summary following this structure:\n{summary_template}"
                    },
                    {
                        "role": "user", 
                        "content": chunk
                    }
                ],
                "stream": False,
            }
            
            try:
                response = http.post(
                    "http://localhost:11434/api/chat",
                    headers=headers,
                    json=data,
                    timeout=45  # Increased timeout for detailed analysis
                )
                
                if response.status_code != 400:
                    print(f"\nWarning: Chunk {i} failed with status {response.status_code}")
                    continue
                    
                response_json = response.json()
                
                if 'message' in response_json and 'content' in response_json['message']:
                    summaries.append(response_json['message']['content'])
                else:
                    print(f"\nWarning: Unexpected response format for chunk {i}")
                
                time.sleep(2)  # Increased delay between requests
                
            except requests.exceptions.Timeout:
                print(f"\nWarning: Timeout on chunk {i}, skipping...")
                continue
                
        print("\nProcessing complete!")
        
        if not summaries:
            return "Failed to generate any summaries"
            
        # Combine and structure final summary
        if len(summaries) > 1:
            combined_text = "\n\n".join(summaries)
            final_data = {
                "model": "goekdenizguelmez/JOSIEFIED-Qwen2.5:3b",
                "messages": [
                    {
                        "role": "system",
                        "content": f"Create a final structured summary following this template, ensuring it's at least 1500 words:\n{summary_template}"
                    },
                    {
                        "role": "user", 
                        "content": combined_text
                    }
                ],
                "stream": False,
            }
            
            try:
                final_response = http.post(
                    "http://localhost:11434/api/chat",
                    headers=headers,
                    json=final_data,
                    timeout=60
                )
                
                if final_response.status_code == 200:
                    return final_response.json()['message']['content']
            except:
                return combined_text
                
        return summaries[0]
            
    except Exception as e:
        return f"Error generating summary: {str(e)}"
    

    def main(): url = input("Enter YouTube URL: ") video_id = get_video_id(url)

    if not video_id:
        print("Invalid YouTube URL")
        return
        
    print("\nFetching transcript...")
    transcript = get_transcript(video_id)
    
    if transcript.startswith("Error"):
        print(transcript)
        return
        
    print("\nGenerating summary...")
    summary = get_summary(transcript)
    
    print("\nSUMMARY:\n")
    print(summary)
    

    if name == "main": main()

  2. Assistant

    Your script is well-structured, but there are a few areas that could be improved to enhance its robustness and efficiency. Below are some suggestions:

    1. Extracting Video ID from YouTube URL:

      • The current implementation may not handle all YouTube URL formats. Consider using a regular expression to extract the video ID more reliably.

        import re
        
        def get_video_id(url):
            """Extract video ID from YouTube URL"""
            pattern = r'(?:v=|\/)([0-9A-Za-z_-]{11}).*'
            match = re.search(pattern, url)
            return match.group(1) if match else None
        
    2. Handling Timeouts in HTTP Requests:

      • It's crucial to set timeouts for HTTP requests to prevent the program from hanging indefinitely if a server doesn't respond. You can specify timeouts in the requests library by using the timeout parameter. For example:

        response = requests.get('https://example.com', timeout=10)  # 10 seconds
        

        This sets a timeout of 10 seconds for the request. If the server doesn't respond within this time, a requests.exceptions.Timeout exception will be raised. It's good practice to handle this exception to maintain the program's stability[1].

    3. Handling Exceptions in HTTP Requests:

      • When making HTTP requests, it's important to handle potential exceptions to prevent the program from crashing. For instance, when fetching the transcript, you can handle exceptions as follows:

        def get_transcript(video_id):
            """Get transcript from YouTube video"""
            try:
                transcript_list = YouTubeTranscriptApi.get_transcript(video_id)
                return " ".join([item['text'] for item in transcript_list])
            except YouTubeTranscriptApi.CouldNotRetrieveTranscript as e:
                return f"Error getting transcript: {str(e)}"
            except Exception as e:
                return f"An unexpected error occurred: {str(e)}"
        
    4. Improving Chunking of Text:

      • The chunk_text function splits the transcript into chunks of approximately chunk_size characters. However, the current implementation may split words across chunks. To avoid this, consider adjusting the logic to ensure that words are not split:

        def chunk_text(text, chunk_size=1000):
            """Split text into chunks of approximately chunk_size characters"""
            words = text.split()
            chunks = []
            current_chunk = []
            current_length = 0
        
            for word in words:
                if current_length + len(word) + 1 &lt;= chunk_size:
                    current_chunk.append(word)
                    current_length += len(word) + 1
                else:
                    chunks.append(' '.join(current_chunk))
                    current_chunk = [word]
                    current_length = len(word)
        
            if current_chunk:
                chunks.append(' '.join(current_chunk))
            return chunks
        
    5. Handling API Responses:

      • When making POST requests to the local API, it's important to check the response status code and handle errors appropriately. For example:

        response = http.post(
            "http://localhost:11434/api/chat",
            headers=headers,
            json=data,
            timeout=45
        )
        
        if response.status_code == 200:
            response_json = response.json()
            if 'message' in response_json and 'content' in response_json['message']:
                summaries.append(response_json['message']['content'])
            else:
                print(f"\nWarning: Unexpected response format for chunk {i}")
        else:
            print(f"\nWarning: Chunk {i} failed with status {response.status_code}")
        
    6. Reducing Delay Between Requests:

      • The time.sleep(2) call introduces a 2-second delay between requests. While this can help prevent rate limiting, it may slow down the script. Consider adjusting the delay based on the API's rate limits and your application's performance requirements.
    7. Ensuring Proper Imports:

      • Ensure that all necessary modules are imported at the beginning of your script. For example:

        import requests
        from youtube_transcript_api import YouTubeTranscriptApi
        import json
        import time
        import re
        from requests.adapters import HTTPAdapter
        from urllib3.util.retry import Retry
        

    By implementing these improvements, your script will be more robust, efficient, and better equipped to handle various edge cases.

  3. User

    (venv) kali@kali:~/Documents/Python Scripts/webpage_downloader$ python "webpage text downloader_v9.py" Enter YouTube URL: https://www.youtube.com/watch?v=he0_W5iCv-I

    Fetching transcript...

    Generating summary...

    SUMMARY:

    Error: Local API is not reachable. Please ensure it is running.

  4. Assistant

    The error message indicates that your script is unable to connect to the local API server at http://localhost:11434/api/chat. To resolve this issue, please follow these steps:

    1. Verify the API Server is Running:

      • Ensure that the API server intended to handle requests at http://localhost:11434/api/chat is actively running on your machine.
      • If you're using a specific application or framework to run this server, confirm that it has been started successfully.
    2. Check the Server's Listening Port:

      • Confirm that the server is configured to listen on port 11434.
      • If the server is set to listen on a different port, update your script to match the correct port number.
    3. Test the API Endpoint:

      • Use a tool like Postman to send a test request to http://localhost:11434/api/chat.
      • If the request fails, it indicates that the server is not accessible at that endpoint.
    4. Review Firewall and Security Settings:

      • Ensure that your firewall or security software is not blocking connections to port 11434.
      • Adjust the settings to allow traffic through this port if necessary.
    5. Inspect the Hosts File:

      • Check your system's hosts file to ensure that the localhost entry is correctly configured.
      • The hosts file should include the line:
        127.0.0.1       localhost
        
      • If this line is missing or commented out, add or uncomment it to ensure proper name resolution.
    6. Restart the API Server:

      • If the server was running, try restarting it to resolve any potential issues.
    7. Update the Script's API Endpoint:

      • If the server is running on a different port or hostname, update the api_url in your script to match the correct address.

    By systematically following these steps, you should be able to identify and resolve the connectivity issue between your script and the local API server.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.