Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-reasoning-pro-high· PerplexityHugging Face LMArena Search Arena 24K

write bash or python script (depending which is more appropriate) to download all files…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4o-search-preview-high-loc’s answer.
  1. User

    write bash or python script (depending which is more appropriate) to download all files with extension xyz from url with links. files are in form of links with href to actual file

  2. Perplexity Assistant

    Python Script to Download Files with Specific Extension from URL

    For downloading files with a specific extension from a URL, Python provides a more robust solution with its powerful HTML parsing capabilities. Here's a comprehensive script that will find all the links with a specific extension and download them:

    #!/usr/bin/env python3
    import os
    import argparse
    import requests
    from urllib.parse import urljoin
    from bs4 import BeautifulSoup
    
    def download_files_with_extension(url, extension, output_dir='.'):
        """
        Download all files with specified extension from a URL
        
        Args:
            url (str): The URL to scrape for files
            extension (str): The file extension to filter (without the dot)
            output_dir (str): Directory to save downloaded files
        """
        # Create output directory if it doesn't exist
        if not os.path.exists(output_dir):
            os.makedirs(output_dir)
            print(f"Created directory: {output_dir}")
        
        # Make sure extension doesn't start with a dot
        if extension.startswith('.'):
            extension = extension[1:]
        
        # Get the page content
        try:
            response = requests.get(url)
            response.raise_for_status()  # Raise an exception for HTTP errors
        except requests.exceptions.RequestException as e:
            print(f"Error fetching URL: {e}")
            return
        
        # Parse HTML
        soup = BeautifulSoup(response.text, 'html.parser')
        
        # Find all links
        links = soup.find_all('a', href=True)
        
        # Filter links with the specified extension
        file_links = []
        for link in links:
            href = link['href']
            if href.lower().endswith(f'.{extension.lower()}'):
                # Convert relative URLs to absolute URLs
                absolute_url = urljoin(url, href)
                file_links.append(absolute_url)
        
        if not file_links:
            print(f"No files with extension '.{extension}' found at {url}")
            return
        
        print(f"Found {len(file_links)} files with extension '.{extension}'")
        
        # Download each file
        for i, file_url in enumerate(file_links, 1):
            filename = os.path.basename(file_url)
            file_path = os.path.join(output_dir, filename)
            
            print(f"Downloading [{i}/{len(file_links)}]: {filename}")
            try:
                file_response = requests.get(file_url, stream=True)
                file_response.raise_for_status()
                
                with open(file_path, 'wb') as f:
                    for chunk in file_response.iter_content(chunk_size=8192):
                        if chunk:
                            f.write(chunk)
                
                print(f"Successfully downloaded: {filename}")
            except requests.exceptions.RequestException as e:
                print(f"Error downloading {filename}: {e}")
            except IOError as e:
                print(f"Error saving {filename}: {e}")
    
    if __name__ == "__main__":
        parser = argparse.ArgumentParser(description='Download files with specific extension from URL')
        parser.add_argument('url', help='URL to scrape for files')
        parser.add_argument('extension', help='File extension to download (e.g., pdf, jpg, xyz)')
        parser.add_argument('--output-dir', '-o', default='.', help='Directory to save downloaded files')
        
        args = parser.parse_args()
        
        download_files_with_extension(args.url, args.extension, args.output_dir)
    

    How to Use the Python Script

    1. Save the script as download_files.py
    2. Make it executable: chmod +x download_files.py
    3. Run it with the URL and extension:
      ./download_files.py https://example.com xyz --output-dir downloads
      

    This will download all files with the .xyz extension from example.com and save them to the "downloads" directory.

    Bash Script Alternative

    If you prefer using Bash, here's a script that accomplishes the same task:

    #!/bin/bash
    
    # Function to display usage information
    usage() {
        echo "Usage: $0 <url> <extension> [output_directory]"
        echo "  url               - URL to scrape for files"
        echo "  extension         - File extension to download (without the dot)"
        echo "  output_directory  - Directory to save downloaded files (optional, defaults to current directory)"
        exit 1
    }
    
    # Check if required arguments are provided
    if [ $# -lt 2 ]; then
        usage
    fi
    
    # Assign arguments to variables
    URL="$1"
    EXTENSION="$2"
    OUTPUT_DIR="${3:-.}"  # Default to current directory if not specified
    
    # Create output directory if it doesn't exist
    if [ ! -d "$OUTPUT_DIR" ]; then
        mkdir -p "$OUTPUT_DIR"
        echo "Created directory: $OUTPUT_DIR"
    fi
    
    # Make sure extension doesn't start with a dot
    EXTENSION=$(echo "$EXTENSION" | sed 's/^\.//')
    
    # Check if wget or curl is available
    if command -v wget &gt; /dev/null; then
        DOWNLOADER="wget"
    elif command -v curl &gt; /dev/null; then
        DOWNLOADER="curl"
    else
        echo "Error: Neither wget nor curl is installed. Please install one of them."
        exit 1
    fi
    
    echo "Fetching links from $URL..."
    
    # Get the page and extract links with the specified extension
    # This handles both relative and absolute URLs
    LINKS=$(curl -s "$URL" | grep -o ']*href="[^"]*\.'$EXTENSION'"[^&gt;]*&gt;' | sed -n 's/.*href="\([^"]*\.'$EXTENSION'\)".*/\1/p')
    
    # If no links were found, exit
    if [ -z "$LINKS" ]; then
        echo "No files with extension '.$EXTENSION' found at $URL"
        exit 0
    fi
    
    # Count the number of links
    LINK_COUNT=$(echo "$LINKS" | wc -l)
    echo "Found $LINK_COUNT files with extension '.$EXTENSION'"
    
    # Download each file
    COUNT=0
    for LINK in $LINKS; do
        COUNT=$((COUNT + 1))
        
        # Handle relative URLs
        if [[ "$LINK" != http* ]]; then
            # Remove trailing slash from URL if it exists
            BASE_URL=$(echo "$URL" | sed 's/\/$//')
            # If the link starts with /, append it to the domain
            if [[ "$LINK" == /* ]]; then
                DOMAIN=$(echo "$URL" | sed -n 's/\(https\?:\/\/[^\/]*\).*/\1/p')
                LINK="$DOMAIN$LINK"
            else
                # Otherwise, append it to the base URL
                LINK="$BASE_URL/$LINK"
            fi
        fi
        
        # Extract filename from the link
        FILENAME=$(basename "$LINK")
        
        echo "Downloading [$COUNT/$LINK_COUNT]: $FILENAME"
        
        # Download the file using the available downloader
        if [ "$DOWNLOADER" = "wget" ]; then
            wget -q "$LINK" -O "$OUTPUT_DIR/$FILENAME"
        else
            curl -s "$LINK" -o "$OUTPUT_DIR/$FILENAME"
        fi
        
        # Check if download was successful
        if [ $? -eq 0 ]; then
            echo "Successfully downloaded: $FILENAME"
        else
            echo "Error downloading: $FILENAME"
        fi
    done
    
    echo "Download completed. Files saved to $OUTPUT_DIR"
    

    How to Use the Bash Script

    1. Save the script as download_files.sh
    2. Make it executable: chmod +x download_files.sh
    3. Run it with the URL and extension:
      ./download_files.sh https://example.com xyz downloads
      

    This will download all files with the .xyz extension from example.com and save them to the "downloads" directory.

    Key Differences Between the Scripts

    • The Python script uses BeautifulSoup for robust HTML parsing, while the Bash script uses grep and sed patterns
    • The Python script handles errors more gracefully
    • The Python script downloads files in chunks, which is better for large files
    • The Bash script uses either wget or curl (whichever is available)
    • The Python script is more cross-platform (works on Windows too)

    Conclusion

    For this specific task, the Python script is more robust and versatile, especially when dealing with complex HTML and various URL formats. However, the Bash script is a good alternative if you want to avoid installing additional dependencies and if the HTML structure of the target website is relatively simple.Citations[1]https://www.ninjaone.com/script-hub/how-to-download-files-from-urls-with-a-bash-script/[2]https://superuser.com/questions/1033195/how-to-download-a-file-from-url-in-linux[3]https://linuxconfig.org/download-file-from-url-on-linux-using-command-line[4]https://stackoverflow.com/questions/23463991/download-files-using-bash-script-using-wget[5]https://www.shell-tips.com/bash/download-files-from-shell/[6]https://www.geeksforgeeks.org/how-to-download-files-from-urls-with-python/[7]https://stackoverflow.com/questions/56186655/how-to-set-up-curl-with-bash-script-to-download-a-file[8]https://github.com/elminson/bash-download-all-file-from-url[9]https://realpython.com/python-download-file-from-url/[10]https://askubuntu.com/questions/694031/why-does-python-in-linux-require-the-line-usr-bin-python[11]https://superuser.com/questions/1699337/how-to-download-all-files-in-a-webpages-directory-sub-folder-all-at-once[12]https://gist.github.com/verginer/808b7c271049bd2c5d360c7bf5276665[13]https://superuser.com/questions/413446/parsing-out-specific-urls-from-local-html-file[14]https://www.youtube.com/watch?v=CsRR8gmrwic[15]https://www.geeksforgeeks.org/downloading-files-web-using-python/[16]https://www.xandermar.com/articles/bash-scrape-file-links-hrefs-url-and-save-files-directory[17]https://www.youtube.com/watch?v=LyymFN9t4kw

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.