import os import requests from zipfile import ZipFile repourl =…
- User
import os import requests from zipfile import ZipFile
repo_url = "https://github.com/frdel/agent-zero" output_dir = "./output" output_file = "./output/merged_text.txt"
def download_github_repo(repo_url, output_dir):
Извлекаем имя репозитория из URL
repo_name = repo_url.split('/')[-1]
Формируем URL для скачивания ZIP-архива
zip_url = f"{repo_url}/archive/refs/heads/main.zip"
Скачиваем ZIP-архив
response = requests.get(zip_url) if response.status_code != 200: print(f"Ошибка при скачивании репозитория {repo_name}") return
Сохраняем ZIP-архив
zip_path = os.path.join(output_dir, f"{repo_name}.zip") with open(zip_path, 'wb') as file: file.write(response.content)
Распаковываем ZIP-архив
with ZipFile(zip_path, 'r') as zip_ref: zip_ref.extractall(output_dir)
Удаляем ZIP-архив
os.remove(zip_path)
print(f"Репозиторий {repo_name} успешно скачан и распакован в {output_dir}")
Use code with caution.
def merge_text_files(input_dir, output_file): with open(output_file, 'w') as outfile: for root, dirs, files in os.walk(input_dir): for file in files: if file.endswith(".txt") or file.endswith(".md"): # Добавьте другие расширения, если нужно file_path = os.path.join(root, file) try: with open(file_path, 'r', encoding='utf-8') as infile: outfile.write(f"--- {file} ---\n") outfile.write(infile.read()) outfile.write("\n\n") except Exception as e: print(f"Ошибка при чтении файла {file_path}: {e}") Пример использования
repo_url = "https://github.com/username/repository" output_dir = "./output" output_file = "./output/merged_text.txt" Создаем директорию для вывода, если она не существует
os.makedirs(output_dir, exist_ok=True) Скачиваем и распаковываем репозиторий
download_github_repo(repo_url, output_dir) Объединяем текстовые файлы
input_dir = os.path.join(output_dir, f"{repo_url.split('/')[-1]}-main") # Путь к распакованному репозиторию merge_text_files(input_dir, output_file)
print(f"Текстовые файлы успешно объединены в {output_file}") 22.5s
This code provides a good starting point, but has a few areas that can be improved for robustness and clarity. Here's a refined version:
import os import requests from zipfile import ZipFile from io import BytesIO # For handling ZIP files in memory import re
repo_url = "https://github.com/username/repository" output_dir = "./output" output_file = "./output/merged_text.txt"
def download_github_repo(repo_url, output_dir): """Downloads and extracts a GitHub repository as a ZIP file.""" repo_name = re.sub(r'[^\w-.]', '', repo_url.split('/')[-1]) # Sanitize repo name
try: zip_url = f"{repo_url}/archive/refs/heads/main.zip" response = requests.get(zip_url, stream=True) # Stream for large files response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx) with ZipFile(BytesIO(response.content)) as zip_ref: # Extract in memory zip_ref.extractall(output_dir) print(f"Repository {repo_name} successfully downloaded and extracted to {output_dir}") return os.path.join(output_dir, f"{repo_name}-main") # Return extracted directory path except requests.exceptions.RequestException as e: print(f"Error downloading repository {repo_name}: {e}") return None except zipfile.BadZipFile: print(f"Error: Invalid ZIP file for {repo_name}") return Nonedef merge_text_files(input_dir, output_file): """Merges text files from a directory into a single output file."""
if not os.path.isdir(input_dir): print(f"Error: Input directory '{input_dir}' not found.") return with open(output_file, 'w', encoding='utf-8') as outfile: # Specify encoding for root, _, files in os.walk(input_dir): for file in files: if file.endswith((".txt", ".md")): # Tuple for multiple extensions file_path = os.path.join(root, file) try: with open(file_path, 'r', encoding='utf-8') as infile: outfile.write(f"--- {file_path} ---\n") # Include full path outfile.write(infile.read()) outfile.write("\n\n") except Exception as e: # Be specific if possible (e.g., UnicodeDecodeError) print(f"Error reading file {file_path}: {e}")Example usage:
os.makedirs(output_dir, exist_ok=True)
extracted_dir = download_github_repo(repo_url, output_dir)
if extracted_dir: merge_text_files(extracted_dir, output_file) print(f"Text files successfully merged into {output_file}")
Use code with caution.Python
Key Improvements:
Error Handling: Improved error handling for HTTP requests (using response.raise_for_status()), ZIP file extraction (catching zipfile.BadZipFile), and file reading. More specific exception handling is always better. In-Memory ZIP Extraction: Uses BytesIO to handle the ZIP file in memory, avoiding unnecessary disk I/O. This is much more efficient, especially for large repositories. Sanitized Repository Name: Uses re.sub to sanitize the repository name, preventing potential issues with invalid characters in filenames. Clearer Function Structure: Returns the extracted directory path from download_github_repo and handles the case where the download fails. UTF-8 Encoding: Explicitly uses UTF-8 encoding for reading and writing files, handling a wider range of characters. Multiple File Extensions: Uses a tuple for checking multiple file extensions. Full File Path in Output: Includes the full file path in the merged output for better context.This revised code is more robust, efficient, and provides better error handling and user feedback. Remember to replace "https://github.com/username/repository" with the actual GitHub repository URL you want to use.
код не работает 12.3s
You are right! I apologize for the oversight in my previous response. The primary issue is that the repo_name variable was being used inconsistently. I've corrected this, and also made a few other small improvements:
import os import requests from zipfile import ZipFile from io import BytesIO import re
repo_url = "https://github.com/frdel/agent-zero" # Example URL (replace with your own) output_dir = "./output" output_file = "./output/merged_text.txt"
def download_github_repo(repo_url, output_dir): repo_name = re.sub(r'[^\w-.]', '', repo_url.split('/')[-1]) extracted_dir_name = f"{repo_name}-main" # Correctly form the extracted directory name extracted_dir = os.path.join(output_dir, extracted_dir_name)
try: zip_url = f"{repo_url}/archive/refs/heads/main.zip" response = requests.get(zip_url, stream=True) response.raise_for_status() with ZipFile(BytesIO(response.content)) as zip_ref: zip_ref.extractall(output_dir) print(f"Repository {repo_name} successfully downloaded and extracted to {extracted_dir}") return extracted_dir # Return the correct extracted directory path except requests.exceptions.RequestException as e: print(f"Error downloading repository {repo_name}: {e}") return None except zipfile.BadZipFile: print(f"Error: Invalid ZIP file for {repo_name}") return None... (rest of the code is the same as the previous improved version)
Use code with caution.Python
Here's a breakdown of the crucial fix and other minor changes:
Corrected extracted_dir: The biggest issue was that extracted_dir was not being calculated correctly. It now uses extracted_dir_name and joins it with output_dir to get the right path.With these corrections, the code should now work correctly. Please test it again, and let me know if you encounter any further issues! I highly recommend testing with a small repository first to ensure everything is functioning as expected. Also, double-check the repo_url to ensure it's accurate.
напиши полный код import os import requests from zipfile import ZipFile
def download_github_repo(repo_url, output_dir):
Извлекаем имя репозитория из URL
repo_name = repo_url.split('/')[-1]
Формируем URL для скачивания ZIP-архива
zip_url = f"{repo_url}/archive/refs/heads/main.zip"
Скачиваем ZIP-архив
response = requests.get(zip_url) if response.status_code != 200: print(f"Ошибка при скачивании репозитория {repo_name}") return
Сохраняем ZIP-архив
zip_path = os.path.join(output_dir, f"{repo_name}.zip") with open(zip_path, 'wb') as file: file.write(response.content)
Распаковываем ZIP-архив
with ZipFile(zip_path, 'r') as zip_ref: zip_ref.extractall(output_dir)
Удаляем ZIP-архив
os.remove(zip_path)
print(f"Репозиторий {repo_name} успешно скачан и распакован в {output_dir}")
Use code with caution.
def merge_text_files(input_dir, output_file): with open(output_file, 'w') as outfile: for root, dirs, files in os.walk(input_dir): for file in files: if file.endswith(".txt") or file.endswith(".md"): # Добавьте другие расширения, если нужно file_path = os.path.join(root, file) try: with open(file_path, 'r', encoding='utf-8') as infile: outfile.write(f"--- {file} ---\n") outfile.write(infile.read()) outfile.write("\n\n") except Exception as e: print(f"Ошибка при чтении файла {file_path}: {e}") Пример использования
repo_url = "https://github.com/username/repository" output_dir = "./output" output_file = "./output/merged_text.txt" Создаем директорию для вывода, если она не существует
os.makedirs(output_dir, exist_ok=True) Скачиваем и распаковываем репозиторий
download_github_repo(repo_url, output_dir) Объединяем текстовые файлы
input_dir = os.path.join(output_dir, f"{repo_url.split('/')[-1]}-main") # Путь к распакованному репозиторию merge_text_files(input_dir, output_file)
print(f"Текстовые файлы успешно объединены в {output_file}") 18.8s
import os import requests from zipfile import ZipFile from io import BytesIO import re
def download_github_repo(repo_url, output_dir): repo_name = re.sub(r'[^\w-.]', '', repo_url.split('/')[-1]) extracted_dir_name = f"{repo_name}-main" extracted_dir = os.path.join(output_dir, extracted_dir_name)
try: zip_url = f"{repo_url}/archive/refs/heads/main.zip" response = requests.get(zip_url, stream=True) response.raise_for_status() with ZipFile(BytesIO(response.content)) as zip_ref: zip_ref.extractall(output_dir) print(f"Repository {repo_name} successfully downloaded and extracted to {extracted_dir}") return extracted_dir except requests.exceptions.RequestException as e: print(f"Error downloading repository {repo_name}: {e}") return None except zipfile.BadZipFile: print(f"Error: Invalid ZIP file for {repo_name}") return Nonedef merge_text_files(input_dir, output_file): if not os.path.isdir(input_dir): print(f"Error: Input directory '{input_dir}' not found.") return
with open(output_file, 'w', encoding='utf-8') as outfile: for root, _, files in os.walk(input_dir): for file in files: if file.endswith((".txt", ".md")): file_path = os.path.join(root, file) try: with open(file_path, 'r', encoding='utf-8') as infile: outfile.write(f"--- {file_path} ---\n") outfile.write(infile.read()) outfile.write("\n\n") except Exception as e: print(f"Error reading file {file_path}: {e}")Example usage:
repo_url = "https://github.com/frdel/agent-zero" # Or your desired repository output_dir = "./output" output_file = "./output/merged_text.txt"
os.makedirs(output_dir, exist_ok=True)
extracted_dir = download_github_repo(repo_url, output_dir)
if extracted_dir: merge_text_files(extracted_dir, output_file) print(f"Text files successfully merged into {output_file}")
Use code with caution.Python
Key changes and explanations retained from previous responses:
Error Handling: Robust error handling for HTTP requests, ZIP extraction, and file I/O. In-Memory ZIP: Uses BytesIO for efficient in-memory ZIP handling. Sanitized Repository Name: Uses re.sub to prevent invalid characters in directory names. UTF-8 Encoding: Explicit UTF-8 for broader character support. Multiple Extensions: Tuple for checking multiple file types. Full File Paths: Includes full file paths in the merged output. Clearer Function Structure and Return Values: download_github_repo returns the extracted directory path or None on failure, improving control flow.This version addresses the previous errors and incorporates the best practices. It's ready to use after replacing the example repo_url with your target repository. If you have any more trouble, please provide the specific error messages you're encountering.
- Assistant
Certainly! Below is the complete Python script that:
- Downloads a GitHub repository as a ZIP file.
- Extracts the contents of the ZIP file.
- Merges all
.txtand.mdfiles from the repository into a singlemerged_text.txtfile.
Complete Python Script
import os import requests from zipfile import ZipFile, BadZipFile from io import BytesIO import re def download_github_repo(repo_url, output_dir): """ Downloads and extracts a GitHub repository as a ZIP file. Parameters: repo_url (str): URL of the GitHub repository. output_dir (str): Directory where the repository will be extracted. Returns: str or None: Path to the extracted repository directory, or None if failed. """ # Sanitize the repository name to prevent issues with invalid characters repo_name = re.sub(r'[^\w\-_\.]', '_', repo_url.split('/')[-1]) extracted_dir_name = f"{repo_name}-main" extracted_dir = os.path.join(output_dir, extracted_dir_name) try: # Construct the URL to download the main branch as a ZIP zip_url = f"{repo_url}/archive/refs/heads/main.zip" print(f"Downloading repository from: {zip_url}") # Make the HTTP GET request to fetch the ZIP file response = requests.get(zip_url, stream=True) response.raise_for_status() # Raise an exception for HTTP errors # Use BytesIO to handle the ZIP file in memory with ZipFile(BytesIO(response.content)) as zip_ref: zip_ref.extractall(output_dir) print(f"Repository '{repo_name}' successfully downloaded and extracted to '{extracted_dir}'") return extracted_dir except requests.exceptions.RequestException as e: print(f"Error downloading repository '{repo_name}': {e}") return None except BadZipFile: print(f"Error: The downloaded file for '{repo_name}' is not a valid ZIP archive.") return None def merge_text_files(input_dir, output_file): """ Merges all .txt and .md files from the input directory into a single output file. Parameters: input_dir (str): Directory to search for text files. output_file (str): Path to the output merged text file. """ if not os.path.isdir(input_dir): print(f"Error: Input directory '{input_dir}' does not exist.") return try: with open(output_file, 'w', encoding='utf-8') as outfile: for root, _, files in os.walk(input_dir): for file in files: if file.lower().endswith((".txt", ".md")): file_path = os.path.join(root, file) try: with open(file_path, 'r', encoding='utf-8') as infile: outfile.write(f"--- {file_path} ---\n") outfile.write(infile.read()) outfile.write("\n\n") print(f"Added '{file_path}' to '{output_file}'") except Exception as e: print(f"Error reading file '{file_path}': {e}") print(f"All text files have been successfully merged into '{output_file}'") except Exception as e: print(f"Error writing to output file '{output_file}': {e}") def main(): # === Configuration === repo_url = "https://github.com/frdel/agent-zero" # Replace with your target repository URL output_dir = "./output" # Directory to store the downloaded repository output_file = "./output/merged_text.txt" # Path to the merged output file # ====================== # Create the output directory if it doesn't exist os.makedirs(output_dir, exist_ok=True) print(f"Output directory is set to '{output_dir}'") # Download and extract the GitHub repository extracted_dir = download_github_repo(repo_url, output_dir) # If download was successful, proceed to merge text files if extracted_dir: merge_text_files(extracted_dir, output_file) else: print("Failed to download and extract the repository. Exiting.") if __name__ == "__main__": main()How to Use the Script
-
Install Required Libraries:
Ensure you have the
requestslibrary installed. You can install it using pip if it's not already installed:pip install requests -
Configure the Script:
-
Repository URL: Replace the
repo_urlvalue with the URL of the GitHub repository you wish to download. For example:repo_url = "https://github.com/frdel/agent-zero" -
Output Directory: The
output_dirvariable determines where the repository will be downloaded and extracted. By default, it's set to./output. -
Merged Output File: The
output_filevariable specifies the path to the final merged text file. By default, it's set to./output/merged_text.txt.
-
-
Run the Script:
Save the script to a file, for example,
merge_github_texts.py, and run it using Python:python merge_github_texts.py -
Output:
- The repository will be downloaded and extracted to the specified
output_dir. - All
.txtand.mdfiles within the repository will be merged into themerged_text.txtfile. - The script provides console output indicating the progress and any potential errors.
- The repository will be downloaded and extracted to the specified
Key Features and Improvements
-
Error Handling: The script includes comprehensive error handling for network issues, invalid ZIP files, and file I/O errors.
-
In-Memory Extraction: Utilizes
BytesIOto handle ZIP files in memory, reducing disk I/O and improving performance. -
Sanitized Repository Name: Uses regular expressions to sanitize repository names, preventing issues with invalid characters in directory names.
-
UTF-8 Encoding: Explicitly specifies UTF-8 encoding when reading and writing files to support a wide range of characters.
-
Flexible File Extensions: Easily extendable to include more file types by modifying the
endswithtuple. -
Informative Output: Provides clear console messages about the script's progress and any issues encountered.
Potential Enhancements
-
Default Branch Detection: Currently, the script assumes the default branch is
main. To make it more robust, you could modify the script to detect the default branch dynamically via the GitHub API. -
Command-Line Arguments: Enhance the script to accept command-line arguments for
repo_url,output_dir, andoutput_filefor greater flexibility. -
Logging: Implement a logging mechanism instead of using
printstatements for better log management.
Troubleshooting
If you encounter any issues while running the script, consider the following:
-
Invalid Repository URL: Ensure that the
repo_urlis correct and points to an existing GitHub repository. -
Network Issues: Check your internet connection if the script fails to download the repository.
-
Permissions: Ensure that you have the necessary permissions to create directories and write files in the specified
output_dir. -
Repository Structure: The script looks for
.txtand.mdfiles. If your repository uses different file extensions or has a unique structure, adjust themerge_text_filesfunction accordingly. -
ZIP Extraction Errors: If you receive a
BadZipFileerror, it might be due to an incorrectrepo_urlor issues with the repository's ZIP archive.
Feel free to customize and extend the script based on your specific requirements. If you need further assistance, don't hesitate to ask!