help me fix my script: import requests from youtubetranscriptapi import…
- User
help me fix my script:
import requests from youtube_transcript_api import YouTubeTranscriptApi import json import time from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry
Configure retries for requests
retry_strategy = Retry( total=3, backoff_factor=1, status_forcelist=[408, 429, 500, 502, 503, 504], ) adapter = HTTPAdapter(max_retries=retry_strategy) http = requests.Session() http.mount("http://", adapter) http.mount("https://", adapter)
def get_video_id(url): """Extract video ID from YouTube URL""" if "v=" in url: return url.split("v=")[1][:11] return None
def get_transcript(video_id): """Get transcript from YouTube video""" try: transcript_list = YouTubeTranscriptApi.get_transcript(video_id) return " ".join([item['text'] for item in transcript_list]) except Exception as e: return f"Error getting transcript: {str(e)}"
def chunk_text(text, chunk_size=1000): # Reduced from 4000 to 2000 """Split text into chunks of approximately chunk_size characters""" words = text.split() chunks = [] current_chunk = [] current_length = 0
for word in words: if current_length + len(word) + 1 <= chunk_size: current_chunk.append(word) current_length += len(word) + 1 else: chunks.append(' '.join(current_chunk)) current_chunk = [word] current_length = len(word) if current_chunk: chunks.append(' '.join(current_chunk)) return chunksdef get_summary(text): """Get summary using local API""" try: chunks = chunk_text(text) summaries = [] total_chunks = len(chunks)
print(f"\nProcessing {total_chunks} chunks...") # Initial analysis to get main topic topic_data = { "model": "goekdenizguelmez/JOSIEFIED-Qwen2.5:3b", "messages": [ { "role": "system", "content": "Identify the main topic of this text in 5-10 words." }, { "role": "user", "content": text[:2000] # Use first chunk for topic identification } ], "stream": False, } try: topic_response = http.post( "http://localhost:11434/api/chat", headers={"Content-Type": "application/json"}, json=topic_data, timeout=60 ) main_topic = topic_response.json()['message']['content'] except: main_topic = "the discussed topic" # Process each chunk with structured format summary_template = f"""Here's a breakdown of the key points and arguments presented in the video about {main_topic}:
-
Key Points Analysis: [Main Arguments]: Identify and summarize the main arguments [Supporting Evidence]: List any evidence or examples provided [Context]: Provide relevant context or background information
-
Detailed Breakdown: [Technical Details]: Explain any technical concepts [Practical Applications]: Describe real-world applications [Challenges/Limitations]: Discuss any challenges or limitations mentioned
-
Additional Insights: [Expert Opinions]: Summarize any expert views or citations [Future Implications]: Discuss potential future impacts [Related Considerations]: Include any related topics or considerations
Overall Takeaway: [Provide a concise summary of the main message and implications] """
for i, chunk in enumerate(chunks, 1): print(f"\rProcessing chunk {i}/{total_chunks}...", end="", flush=True) headers = {"Content-Type": "application/json"} data = { "model": "goekdenizguelmez/JOSIEFIED-Qwen2.5:3b", "messages": [ { "role": "system", "content": f"Analyze this text segment and provide a detailed summary following this structure:\n{summary_template}" }, { "role": "user", "content": chunk } ], "stream": False, } try: response = http.post( "http://localhost:11434/api/chat", headers=headers, json=data, timeout=45 # Increased timeout for detailed analysis ) if response.status_code != 400: print(f"\nWarning: Chunk {i} failed with status {response.status_code}") continue response_json = response.json() if 'message' in response_json and 'content' in response_json['message']: summaries.append(response_json['message']['content']) else: print(f"\nWarning: Unexpected response format for chunk {i}") time.sleep(2) # Increased delay between requests except requests.exceptions.Timeout: print(f"\nWarning: Timeout on chunk {i}, skipping...") continue print("\nProcessing complete!") if not summaries: return "Failed to generate any summaries" # Combine and structure final summary if len(summaries) > 1: combined_text = "\n\n".join(summaries) final_data = { "model": "goekdenizguelmez/JOSIEFIED-Qwen2.5:3b", "messages": [ { "role": "system", "content": f"Create a final structured summary following this template, ensuring it's at least 1500 words:\n{summary_template}" }, { "role": "user", "content": combined_text } ], "stream": False, } try: final_response = http.post( "http://localhost:11434/api/chat", headers=headers, json=final_data, timeout=60 ) if final_response.status_code == 200: return final_response.json()['message']['content'] except: return combined_text return summaries[0] except Exception as e: return f"Error generating summary: {str(e)}"def main(): url = input("Enter YouTube URL: ") video_id = get_video_id(url)
if not video_id: print("Invalid YouTube URL") return print("\nFetching transcript...") transcript = get_transcript(video_id) if transcript.startswith("Error"): print(transcript) return print("\nGenerating summary...") summary = get_summary(transcript) print("\nSUMMARY:\n") print(summary)if name == "main": main()
-
- Assistant
Your script is well-structured, but there are a few areas that could be improved to enhance its robustness and efficiency. Below are some suggestions:
-
Extracting Video ID from YouTube URL:
-
The current implementation may not handle all YouTube URL formats. Consider using a regular expression to extract the video ID more reliably.
import re def get_video_id(url): """Extract video ID from YouTube URL""" pattern = r'(?:v=|\/)([0-9A-Za-z_-]{11}).*' match = re.search(pattern, url) return match.group(1) if match else None
-
-
Handling Timeouts in HTTP Requests:
-
It's crucial to set timeouts for HTTP requests to prevent the program from hanging indefinitely if a server doesn't respond. You can specify timeouts in the
requestslibrary by using thetimeoutparameter. For example:response = requests.get('https://example.com', timeout=10) # 10 secondsThis sets a timeout of 10 seconds for the request. If the server doesn't respond within this time, a
requests.exceptions.Timeoutexception will be raised. It's good practice to handle this exception to maintain the program's stability[1].
-
-
Handling Exceptions in HTTP Requests:
-
When making HTTP requests, it's important to handle potential exceptions to prevent the program from crashing. For instance, when fetching the transcript, you can handle exceptions as follows:
def get_transcript(video_id): """Get transcript from YouTube video""" try: transcript_list = YouTubeTranscriptApi.get_transcript(video_id) return " ".join([item['text'] for item in transcript_list]) except YouTubeTranscriptApi.CouldNotRetrieveTranscript as e: return f"Error getting transcript: {str(e)}" except Exception as e: return f"An unexpected error occurred: {str(e)}"
-
-
Improving Chunking of Text:
-
The
chunk_textfunction splits the transcript into chunks of approximatelychunk_sizecharacters. However, the current implementation may split words across chunks. To avoid this, consider adjusting the logic to ensure that words are not split:def chunk_text(text, chunk_size=1000): """Split text into chunks of approximately chunk_size characters""" words = text.split() chunks = [] current_chunk = [] current_length = 0 for word in words: if current_length + len(word) + 1 <= chunk_size: current_chunk.append(word) current_length += len(word) + 1 else: chunks.append(' '.join(current_chunk)) current_chunk = [word] current_length = len(word) if current_chunk: chunks.append(' '.join(current_chunk)) return chunks
-
-
Handling API Responses:
-
When making POST requests to the local API, it's important to check the response status code and handle errors appropriately. For example:
response = http.post( "http://localhost:11434/api/chat", headers=headers, json=data, timeout=45 ) if response.status_code == 200: response_json = response.json() if 'message' in response_json and 'content' in response_json['message']: summaries.append(response_json['message']['content']) else: print(f"\nWarning: Unexpected response format for chunk {i}") else: print(f"\nWarning: Chunk {i} failed with status {response.status_code}")
-
-
Reducing Delay Between Requests:
- The
time.sleep(2)call introduces a 2-second delay between requests. While this can help prevent rate limiting, it may slow down the script. Consider adjusting the delay based on the API's rate limits and your application's performance requirements.
- The
-
Ensuring Proper Imports:
-
Ensure that all necessary modules are imported at the beginning of your script. For example:
import requests from youtube_transcript_api import YouTubeTranscriptApi import json import time import re from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry
-
By implementing these improvements, your script will be more robust, efficient, and better equipped to handle various edge cases.
-
- User
(venv) kali@kali:~/Documents/Python Scripts/webpage_downloader$ python "webpage text downloader_v9.py" Enter YouTube URL: https://www.youtube.com/watch?v=he0_W5iCv-I
Fetching transcript...
Generating summary...
SUMMARY:
Error: Local API is not reachable. Please ensure it is running.
- Assistant
The error message indicates that your script is unable to connect to the local API server at
http://localhost:11434/api/chat. To resolve this issue, please follow these steps:-
Verify the API Server is Running:
- Ensure that the API server intended to handle requests at
http://localhost:11434/api/chatis actively running on your machine. - If you're using a specific application or framework to run this server, confirm that it has been started successfully.
- Ensure that the API server intended to handle requests at
-
Check the Server's Listening Port:
- Confirm that the server is configured to listen on port
11434. - If the server is set to listen on a different port, update your script to match the correct port number.
- Confirm that the server is configured to listen on port
-
Test the API Endpoint:
- Use a tool like Postman to send a test request to
http://localhost:11434/api/chat. - If the request fails, it indicates that the server is not accessible at that endpoint.
- Use a tool like Postman to send a test request to
-
Review Firewall and Security Settings:
- Ensure that your firewall or security software is not blocking connections to port
11434. - Adjust the settings to allow traffic through this port if necessary.
- Ensure that your firewall or security software is not blocking connections to port
-
Inspect the Hosts File:
- Check your system's hosts file to ensure that the
localhostentry is correctly configured. - The hosts file should include the line:
127.0.0.1 localhost - If this line is missing or commented out, add or uncomment it to ensure proper name resolution.
- Check your system's hosts file to ensure that the
-
Restart the API Server:
- If the server was running, try restarting it to resolve any potential issues.
-
Update the Script's API Endpoint:
- If the server is running on a different port or hostname, update the
api_urlin your script to match the correct address.
- If the server is running on a different port or hostname, update the
By systematically following these steps, you should be able to identify and resolve the connectivity issue between your script and the local API server.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.