Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

import os import requests from zipfile import ZipFile repourl =…

  1. User

    import os import requests from zipfile import ZipFile

    repo_url = "https://github.com/frdel/agent-zero" output_dir = "./output" output_file = "./output/merged_text.txt"

    def download_github_repo(repo_url, output_dir):

    Извлекаем имя репозитория из URL

    repo_name = repo_url.split('/')[-1]

    Формируем URL для скачивания ZIP-архива

    zip_url = f"{repo_url}/archive/refs/heads/main.zip"

    Скачиваем ZIP-архив

    response = requests.get(zip_url) if response.status_code != 200: print(f"Ошибка при скачивании репозитория {repo_name}") return

    Сохраняем ZIP-архив

    zip_path = os.path.join(output_dir, f"{repo_name}.zip") with open(zip_path, 'wb') as file: file.write(response.content)

    Распаковываем ZIP-архив

    with ZipFile(zip_path, 'r') as zip_ref: zip_ref.extractall(output_dir)

    Удаляем ZIP-архив

    os.remove(zip_path)

    print(f"Репозиторий {repo_name} успешно скачан и распакован в {output_dir}")

    Use code with caution.

    def merge_text_files(input_dir, output_file): with open(output_file, 'w') as outfile: for root, dirs, files in os.walk(input_dir): for file in files: if file.endswith(".txt") or file.endswith(".md"): # Добавьте другие расширения, если нужно file_path = os.path.join(root, file) try: with open(file_path, 'r', encoding='utf-8') as infile: outfile.write(f"--- {file} ---\n") outfile.write(infile.read()) outfile.write("\n\n") except Exception as e: print(f"Ошибка при чтении файла {file_path}: {e}") Пример использования

    repo_url = "https://github.com/username/repository" output_dir = "./output" output_file = "./output/merged_text.txt" Создаем директорию для вывода, если она не существует

    os.makedirs(output_dir, exist_ok=True) Скачиваем и распаковываем репозиторий

    download_github_repo(repo_url, output_dir) Объединяем текстовые файлы

    input_dir = os.path.join(output_dir, f"{repo_url.split('/')[-1]}-main") # Путь к распакованному репозиторию merge_text_files(input_dir, output_file)

    print(f"Текстовые файлы успешно объединены в {output_file}") 22.5s

    This code provides a good starting point, but has a few areas that can be improved for robustness and clarity. Here's a refined version:

    import os import requests from zipfile import ZipFile from io import BytesIO # For handling ZIP files in memory import re

    repo_url = "https://github.com/username/repository" output_dir = "./output" output_file = "./output/merged_text.txt"

    def download_github_repo(repo_url, output_dir): """Downloads and extracts a GitHub repository as a ZIP file.""" repo_name = re.sub(r'[^\w-.]', '', repo_url.split('/')[-1]) # Sanitize repo name

    try:
        zip_url = f"{repo_url}/archive/refs/heads/main.zip"
        response = requests.get(zip_url, stream=True)  # Stream for large files
        response.raise_for_status()  # Raise HTTPError for bad responses (4xx or 5xx)
    
        with ZipFile(BytesIO(response.content)) as zip_ref:  # Extract in memory
            zip_ref.extractall(output_dir)
    
        print(f"Repository {repo_name} successfully downloaded and extracted to {output_dir}")
        return os.path.join(output_dir, f"{repo_name}-main")  # Return extracted directory path
    
    except requests.exceptions.RequestException as e:
        print(f"Error downloading repository {repo_name}: {e}")
        return None
    except zipfile.BadZipFile:
        print(f"Error: Invalid ZIP file for {repo_name}")
        return None
    

    def merge_text_files(input_dir, output_file): """Merges text files from a directory into a single output file."""

    if not os.path.isdir(input_dir):
        print(f"Error: Input directory '{input_dir}' not found.")
        return
    
    with open(output_file, 'w', encoding='utf-8') as outfile:  # Specify encoding
        for root, _, files in os.walk(input_dir):
            for file in files:
                if file.endswith((".txt", ".md")):  # Tuple for multiple extensions
                    file_path = os.path.join(root, file)
                    try:
                        with open(file_path, 'r', encoding='utf-8') as infile:
                            outfile.write(f"--- {file_path} ---\n")  # Include full path
                            outfile.write(infile.read())
                            outfile.write("\n\n")
                    except Exception as e:  # Be specific if possible (e.g., UnicodeDecodeError)
                        print(f"Error reading file {file_path}: {e}")
    

    Example usage:

    os.makedirs(output_dir, exist_ok=True)

    extracted_dir = download_github_repo(repo_url, output_dir)

    if extracted_dir: merge_text_files(extracted_dir, output_file) print(f"Text files successfully merged into {output_file}")

    Use code with caution.Python

    Key Improvements:

    Error Handling: Improved error handling for HTTP requests (using response.raise_for_status()), ZIP file extraction (catching zipfile.BadZipFile), and file reading. More specific exception handling is always better.
    
    In-Memory ZIP Extraction: Uses BytesIO to handle the ZIP file in memory, avoiding unnecessary disk I/O. This is much more efficient, especially for large repositories.
    
    Sanitized Repository Name: Uses re.sub to sanitize the repository name, preventing potential issues with invalid characters in filenames.
    
    Clearer Function Structure: Returns the extracted directory path from download_github_repo and handles the case where the download fails.
    
    UTF-8 Encoding: Explicitly uses UTF-8 encoding for reading and writing files, handling a wider range of characters.
    
    Multiple File Extensions: Uses a tuple for checking multiple file extensions.
    
    Full File Path in Output: Includes the full file path in the merged output for better context.
    

    This revised code is more robust, efficient, and provides better error handling and user feedback. Remember to replace "https://github.com/username/repository" with the actual GitHub repository URL you want to use.

    код не работает 12.3s

    You are right! I apologize for the oversight in my previous response. The primary issue is that the repo_name variable was being used inconsistently. I've corrected this, and also made a few other small improvements:

    import os import requests from zipfile import ZipFile from io import BytesIO import re

    repo_url = "https://github.com/frdel/agent-zero" # Example URL (replace with your own) output_dir = "./output" output_file = "./output/merged_text.txt"

    def download_github_repo(repo_url, output_dir): repo_name = re.sub(r'[^\w-.]', '', repo_url.split('/')[-1]) extracted_dir_name = f"{repo_name}-main" # Correctly form the extracted directory name extracted_dir = os.path.join(output_dir, extracted_dir_name)

    try:
        zip_url = f"{repo_url}/archive/refs/heads/main.zip"
        response = requests.get(zip_url, stream=True)
        response.raise_for_status()
    
        with ZipFile(BytesIO(response.content)) as zip_ref:
            zip_ref.extractall(output_dir)
    
        print(f"Repository {repo_name} successfully downloaded and extracted to {extracted_dir}")
        return extracted_dir  # Return the correct extracted directory path
    
    except requests.exceptions.RequestException as e:
        print(f"Error downloading repository {repo_name}: {e}")
        return None
    except zipfile.BadZipFile:
        print(f"Error: Invalid ZIP file for {repo_name}")
        return None
    

    ... (rest of the code is the same as the previous improved version)

    Use code with caution.Python

    Here's a breakdown of the crucial fix and other minor changes:

    Corrected extracted_dir: The biggest issue was that extracted_dir was not being calculated correctly. It now uses extracted_dir_name and joins it with output_dir to get the right path.
    

    With these corrections, the code should now work correctly. Please test it again, and let me know if you encounter any further issues! I highly recommend testing with a small repository first to ensure everything is functioning as expected. Also, double-check the repo_url to ensure it's accurate.

    напиши полный код import os import requests from zipfile import ZipFile

    def download_github_repo(repo_url, output_dir):

    Извлекаем имя репозитория из URL

    repo_name = repo_url.split('/')[-1]

    Формируем URL для скачивания ZIP-архива

    zip_url = f"{repo_url}/archive/refs/heads/main.zip"

    Скачиваем ZIP-архив

    response = requests.get(zip_url) if response.status_code != 200: print(f"Ошибка при скачивании репозитория {repo_name}") return

    Сохраняем ZIP-архив

    zip_path = os.path.join(output_dir, f"{repo_name}.zip") with open(zip_path, 'wb') as file: file.write(response.content)

    Распаковываем ZIP-архив

    with ZipFile(zip_path, 'r') as zip_ref: zip_ref.extractall(output_dir)

    Удаляем ZIP-архив

    os.remove(zip_path)

    print(f"Репозиторий {repo_name} успешно скачан и распакован в {output_dir}")

    Use code with caution.

    def merge_text_files(input_dir, output_file): with open(output_file, 'w') as outfile: for root, dirs, files in os.walk(input_dir): for file in files: if file.endswith(".txt") or file.endswith(".md"): # Добавьте другие расширения, если нужно file_path = os.path.join(root, file) try: with open(file_path, 'r', encoding='utf-8') as infile: outfile.write(f"--- {file} ---\n") outfile.write(infile.read()) outfile.write("\n\n") except Exception as e: print(f"Ошибка при чтении файла {file_path}: {e}") Пример использования

    repo_url = "https://github.com/username/repository" output_dir = "./output" output_file = "./output/merged_text.txt" Создаем директорию для вывода, если она не существует

    os.makedirs(output_dir, exist_ok=True) Скачиваем и распаковываем репозиторий

    download_github_repo(repo_url, output_dir) Объединяем текстовые файлы

    input_dir = os.path.join(output_dir, f"{repo_url.split('/')[-1]}-main") # Путь к распакованному репозиторию merge_text_files(input_dir, output_file)

    print(f"Текстовые файлы успешно объединены в {output_file}") 18.8s

    import os import requests from zipfile import ZipFile from io import BytesIO import re

    def download_github_repo(repo_url, output_dir): repo_name = re.sub(r'[^\w-.]', '', repo_url.split('/')[-1]) extracted_dir_name = f"{repo_name}-main" extracted_dir = os.path.join(output_dir, extracted_dir_name)

    try:
        zip_url = f"{repo_url}/archive/refs/heads/main.zip"
        response = requests.get(zip_url, stream=True)
        response.raise_for_status()
    
        with ZipFile(BytesIO(response.content)) as zip_ref:
            zip_ref.extractall(output_dir)
    
        print(f"Repository {repo_name} successfully downloaded and extracted to {extracted_dir}")
        return extracted_dir
    
    except requests.exceptions.RequestException as e:
        print(f"Error downloading repository {repo_name}: {e}")
        return None
    except zipfile.BadZipFile:
        print(f"Error: Invalid ZIP file for {repo_name}")
        return None
    

    def merge_text_files(input_dir, output_file): if not os.path.isdir(input_dir): print(f"Error: Input directory '{input_dir}' not found.") return

    with open(output_file, 'w', encoding='utf-8') as outfile:
        for root, _, files in os.walk(input_dir):
            for file in files:
                if file.endswith((".txt", ".md")):
                    file_path = os.path.join(root, file)
                    try:
                        with open(file_path, 'r', encoding='utf-8') as infile:
                            outfile.write(f"--- {file_path} ---\n")
                            outfile.write(infile.read())
                            outfile.write("\n\n")
                    except Exception as e:
                        print(f"Error reading file {file_path}: {e}")
    

    Example usage:

    repo_url = "https://github.com/frdel/agent-zero" # Or your desired repository output_dir = "./output" output_file = "./output/merged_text.txt"

    os.makedirs(output_dir, exist_ok=True)

    extracted_dir = download_github_repo(repo_url, output_dir)

    if extracted_dir: merge_text_files(extracted_dir, output_file) print(f"Text files successfully merged into {output_file}")

    Use code with caution.Python

    Key changes and explanations retained from previous responses:

    Error Handling: Robust error handling for HTTP requests, ZIP extraction, and file I/O.
    
    In-Memory ZIP: Uses BytesIO for efficient in-memory ZIP handling.
    
    Sanitized Repository Name: Uses re.sub to prevent invalid characters in directory names.
    
    UTF-8 Encoding: Explicit UTF-8 for broader character support.
    
    Multiple Extensions: Tuple for checking multiple file types.
    
    Full File Paths: Includes full file paths in the merged output.
    
    Clearer Function Structure and Return Values: download_github_repo returns the extracted directory path or None on failure, improving control flow.
    

    This version addresses the previous errors and incorporates the best practices. It's ready to use after replacing the example repo_url with your target repository. If you have any more trouble, please provide the specific error messages you're encountering.

  2. Assistant

    Certainly! Below is the complete Python script that:

    1. Downloads a GitHub repository as a ZIP file.
    2. Extracts the contents of the ZIP file.
    3. Merges all .txt and .md files from the repository into a single merged_text.txt file.

    Complete Python Script

    import os
    import requests
    from zipfile import ZipFile, BadZipFile
    from io import BytesIO
    import re
    
    def download_github_repo(repo_url, output_dir):
        """
        Downloads and extracts a GitHub repository as a ZIP file.
    
        Parameters:
            repo_url (str): URL of the GitHub repository.
            output_dir (str): Directory where the repository will be extracted.
    
        Returns:
            str or None: Path to the extracted repository directory, or None if failed.
        """
        # Sanitize the repository name to prevent issues with invalid characters
        repo_name = re.sub(r'[^\w\-_\.]', '_', repo_url.split('/')[-1])
        extracted_dir_name = f"{repo_name}-main"
        extracted_dir = os.path.join(output_dir, extracted_dir_name)
    
        try:
            # Construct the URL to download the main branch as a ZIP
            zip_url = f"{repo_url}/archive/refs/heads/main.zip"
            print(f"Downloading repository from: {zip_url}")
    
            # Make the HTTP GET request to fetch the ZIP file
            response = requests.get(zip_url, stream=True)
            response.raise_for_status()  # Raise an exception for HTTP errors
    
            # Use BytesIO to handle the ZIP file in memory
            with ZipFile(BytesIO(response.content)) as zip_ref:
                zip_ref.extractall(output_dir)
    
            print(f"Repository '{repo_name}' successfully downloaded and extracted to '{extracted_dir}'")
            return extracted_dir
    
        except requests.exceptions.RequestException as e:
            print(f"Error downloading repository '{repo_name}': {e}")
            return None
        except BadZipFile:
            print(f"Error: The downloaded file for '{repo_name}' is not a valid ZIP archive.")
            return None
    
    def merge_text_files(input_dir, output_file):
        """
        Merges all .txt and .md files from the input directory into a single output file.
    
        Parameters:
            input_dir (str): Directory to search for text files.
            output_file (str): Path to the output merged text file.
        """
        if not os.path.isdir(input_dir):
            print(f"Error: Input directory '{input_dir}' does not exist.")
            return
    
        try:
            with open(output_file, 'w', encoding='utf-8') as outfile:
                for root, _, files in os.walk(input_dir):
                    for file in files:
                        if file.lower().endswith((".txt", ".md")):
                            file_path = os.path.join(root, file)
                            try:
                                with open(file_path, 'r', encoding='utf-8') as infile:
                                    outfile.write(f"--- {file_path} ---\n")
                                    outfile.write(infile.read())
                                    outfile.write("\n\n")
                                print(f"Added '{file_path}' to '{output_file}'")
                            except Exception as e:
                                print(f"Error reading file '{file_path}': {e}")
            print(f"All text files have been successfully merged into '{output_file}'")
        except Exception as e:
            print(f"Error writing to output file '{output_file}': {e}")
    
    def main():
        # === Configuration ===
        repo_url = "https://github.com/frdel/agent-zero"  # Replace with your target repository URL
        output_dir = "./output"                            # Directory to store the downloaded repository
        output_file = "./output/merged_text.txt"           # Path to the merged output file
        # ======================
    
        # Create the output directory if it doesn't exist
        os.makedirs(output_dir, exist_ok=True)
        print(f"Output directory is set to '{output_dir}'")
    
        # Download and extract the GitHub repository
        extracted_dir = download_github_repo(repo_url, output_dir)
    
        # If download was successful, proceed to merge text files
        if extracted_dir:
            merge_text_files(extracted_dir, output_file)
        else:
            print("Failed to download and extract the repository. Exiting.")
    
    if __name__ == "__main__":
        main()
    

    How to Use the Script

    1. Install Required Libraries:

      Ensure you have the requests library installed. You can install it using pip if it's not already installed:

      pip install requests
      
    2. Configure the Script:

      • Repository URL: Replace the repo_url value with the URL of the GitHub repository you wish to download. For example:

        repo_url = "https://github.com/frdel/agent-zero"
        
      • Output Directory: The output_dir variable determines where the repository will be downloaded and extracted. By default, it's set to ./output.

      • Merged Output File: The output_file variable specifies the path to the final merged text file. By default, it's set to ./output/merged_text.txt.

    3. Run the Script:

      Save the script to a file, for example, merge_github_texts.py, and run it using Python:

      python merge_github_texts.py
      
    4. Output:

      • The repository will be downloaded and extracted to the specified output_dir.
      • All .txt and .md files within the repository will be merged into the merged_text.txt file.
      • The script provides console output indicating the progress and any potential errors.

    Key Features and Improvements

    • Error Handling: The script includes comprehensive error handling for network issues, invalid ZIP files, and file I/O errors.

    • In-Memory Extraction: Utilizes BytesIO to handle ZIP files in memory, reducing disk I/O and improving performance.

    • Sanitized Repository Name: Uses regular expressions to sanitize repository names, preventing issues with invalid characters in directory names.

    • UTF-8 Encoding: Explicitly specifies UTF-8 encoding when reading and writing files to support a wide range of characters.

    • Flexible File Extensions: Easily extendable to include more file types by modifying the endswith tuple.

    • Informative Output: Provides clear console messages about the script's progress and any issues encountered.

    Potential Enhancements

    • Default Branch Detection: Currently, the script assumes the default branch is main. To make it more robust, you could modify the script to detect the default branch dynamically via the GitHub API.

    • Command-Line Arguments: Enhance the script to accept command-line arguments for repo_url, output_dir, and output_file for greater flexibility.

    • Logging: Implement a logging mechanism instead of using print statements for better log management.

    Troubleshooting

    If you encounter any issues while running the script, consider the following:

    1. Invalid Repository URL: Ensure that the repo_url is correct and points to an existing GitHub repository.

    2. Network Issues: Check your internet connection if the script fails to download the repository.

    3. Permissions: Ensure that you have the necessary permissions to create directories and write files in the specified output_dir.

    4. Repository Structure: The script looks for .txt and .md files. If your repository uses different file extensions or has a unique structure, adjust the merge_text_files function accordingly.

    5. ZIP Extraction Errors: If you receive a BadZipFile error, it might be due to an incorrect repo_url or issues with the repository's ZIP archive.

    Feel free to customize and extend the script based on your specific requirements. If you need further assistance, don't hesitate to ask!