write bash or python script (depending which is more appropriate) to download all files…
- User
write bash or python script (depending which is more appropriate) to download all files with extension xyz from url with links. files are in form of links with href to actual file
Assistant
Python Script to Download Files with Specific Extension from URL
For downloading files with a specific extension from a URL, Python provides a more robust solution with its powerful HTML parsing capabilities. Here's a comprehensive script that will find all the links with a specific extension and download them:
#!/usr/bin/env python3 import os import argparse import requests from urllib.parse import urljoin from bs4 import BeautifulSoup def download_files_with_extension(url, extension, output_dir='.'): """ Download all files with specified extension from a URL Args: url (str): The URL to scrape for files extension (str): The file extension to filter (without the dot) output_dir (str): Directory to save downloaded files """ # Create output directory if it doesn't exist if not os.path.exists(output_dir): os.makedirs(output_dir) print(f"Created directory: {output_dir}") # Make sure extension doesn't start with a dot if extension.startswith('.'): extension = extension[1:] # Get the page content try: response = requests.get(url) response.raise_for_status() # Raise an exception for HTTP errors except requests.exceptions.RequestException as e: print(f"Error fetching URL: {e}") return # Parse HTML soup = BeautifulSoup(response.text, 'html.parser') # Find all links links = soup.find_all('a', href=True) # Filter links with the specified extension file_links = [] for link in links: href = link['href'] if href.lower().endswith(f'.{extension.lower()}'): # Convert relative URLs to absolute URLs absolute_url = urljoin(url, href) file_links.append(absolute_url) if not file_links: print(f"No files with extension '.{extension}' found at {url}") return print(f"Found {len(file_links)} files with extension '.{extension}'") # Download each file for i, file_url in enumerate(file_links, 1): filename = os.path.basename(file_url) file_path = os.path.join(output_dir, filename) print(f"Downloading [{i}/{len(file_links)}]: {filename}") try: file_response = requests.get(file_url, stream=True) file_response.raise_for_status() with open(file_path, 'wb') as f: for chunk in file_response.iter_content(chunk_size=8192): if chunk: f.write(chunk) print(f"Successfully downloaded: {filename}") except requests.exceptions.RequestException as e: print(f"Error downloading {filename}: {e}") except IOError as e: print(f"Error saving {filename}: {e}") if __name__ == "__main__": parser = argparse.ArgumentParser(description='Download files with specific extension from URL') parser.add_argument('url', help='URL to scrape for files') parser.add_argument('extension', help='File extension to download (e.g., pdf, jpg, xyz)') parser.add_argument('--output-dir', '-o', default='.', help='Directory to save downloaded files') args = parser.parse_args() download_files_with_extension(args.url, args.extension, args.output_dir)How to Use the Python Script
- Save the script as
download_files.py - Make it executable:
chmod +x download_files.py - Run it with the URL and extension:
./download_files.py https://example.com xyz --output-dir downloads
This will download all files with the
.xyzextension from example.com and save them to the "downloads" directory.Bash Script Alternative
If you prefer using Bash, here's a script that accomplishes the same task:
#!/bin/bash # Function to display usage information usage() { echo "Usage: $0 <url> <extension> [output_directory]" echo " url - URL to scrape for files" echo " extension - File extension to download (without the dot)" echo " output_directory - Directory to save downloaded files (optional, defaults to current directory)" exit 1 } # Check if required arguments are provided if [ $# -lt 2 ]; then usage fi # Assign arguments to variables URL="$1" EXTENSION="$2" OUTPUT_DIR="${3:-.}" # Default to current directory if not specified # Create output directory if it doesn't exist if [ ! -d "$OUTPUT_DIR" ]; then mkdir -p "$OUTPUT_DIR" echo "Created directory: $OUTPUT_DIR" fi # Make sure extension doesn't start with a dot EXTENSION=$(echo "$EXTENSION" | sed 's/^\.//') # Check if wget or curl is available if command -v wget > /dev/null; then DOWNLOADER="wget" elif command -v curl > /dev/null; then DOWNLOADER="curl" else echo "Error: Neither wget nor curl is installed. Please install one of them." exit 1 fi echo "Fetching links from $URL..." # Get the page and extract links with the specified extension # This handles both relative and absolute URLs LINKS=$(curl -s "$URL" | grep -o ']*href="[^"]*\.'$EXTENSION'"[^>]*>' | sed -n 's/.*href="\([^"]*\.'$EXTENSION'\)".*/\1/p') # If no links were found, exit if [ -z "$LINKS" ]; then echo "No files with extension '.$EXTENSION' found at $URL" exit 0 fi # Count the number of links LINK_COUNT=$(echo "$LINKS" | wc -l) echo "Found $LINK_COUNT files with extension '.$EXTENSION'" # Download each file COUNT=0 for LINK in $LINKS; do COUNT=$((COUNT + 1)) # Handle relative URLs if [[ "$LINK" != http* ]]; then # Remove trailing slash from URL if it exists BASE_URL=$(echo "$URL" | sed 's/\/$//') # If the link starts with /, append it to the domain if [[ "$LINK" == /* ]]; then DOMAIN=$(echo "$URL" | sed -n 's/\(https\?:\/\/[^\/]*\).*/\1/p') LINK="$DOMAIN$LINK" else # Otherwise, append it to the base URL LINK="$BASE_URL/$LINK" fi fi # Extract filename from the link FILENAME=$(basename "$LINK") echo "Downloading [$COUNT/$LINK_COUNT]: $FILENAME" # Download the file using the available downloader if [ "$DOWNLOADER" = "wget" ]; then wget -q "$LINK" -O "$OUTPUT_DIR/$FILENAME" else curl -s "$LINK" -o "$OUTPUT_DIR/$FILENAME" fi # Check if download was successful if [ $? -eq 0 ]; then echo "Successfully downloaded: $FILENAME" else echo "Error downloading: $FILENAME" fi done echo "Download completed. Files saved to $OUTPUT_DIR"How to Use the Bash Script
- Save the script as
download_files.sh - Make it executable:
chmod +x download_files.sh - Run it with the URL and extension:
./download_files.sh https://example.com xyz downloads
This will download all files with the
.xyzextension from example.com and save them to the "downloads" directory.Key Differences Between the Scripts
- The Python script uses BeautifulSoup for robust HTML parsing, while the Bash script uses grep and sed patterns
- The Python script handles errors more gracefully
- The Python script downloads files in chunks, which is better for large files
- The Bash script uses either wget or curl (whichever is available)
- The Python script is more cross-platform (works on Windows too)
Conclusion
For this specific task, the Python script is more robust and versatile, especially when dealing with complex HTML and various URL formats. However, the Bash script is a good alternative if you want to avoid installing additional dependencies and if the HTML structure of the target website is relatively simple.Citations[1]https://www.ninjaone.com/script-hub/how-to-download-files-from-urls-with-a-bash-script/[2]https://superuser.com/questions/1033195/how-to-download-a-file-from-url-in-linux[3]https://linuxconfig.org/download-file-from-url-on-linux-using-command-line[4]https://stackoverflow.com/questions/23463991/download-files-using-bash-script-using-wget[5]https://www.shell-tips.com/bash/download-files-from-shell/[6]https://www.geeksforgeeks.org/how-to-download-files-from-urls-with-python/[7]https://stackoverflow.com/questions/56186655/how-to-set-up-curl-with-bash-script-to-download-a-file[8]https://github.com/elminson/bash-download-all-file-from-url[9]https://realpython.com/python-download-file-from-url/[10]https://askubuntu.com/questions/694031/why-does-python-in-linux-require-the-line-usr-bin-python[11]https://superuser.com/questions/1699337/how-to-download-all-files-in-a-webpages-directory-sub-folder-all-at-once[12]https://gist.github.com/verginer/808b7c271049bd2c5d360c7bf5276665[13]https://superuser.com/questions/413446/parsing-out-specific-urls-from-local-html-file[14]https://www.youtube.com/watch?v=CsRR8gmrwic[15]https://www.geeksforgeeks.org/downloading-files-web-using-python/[16]https://www.xandermar.com/articles/bash-scrape-file-links-hrefs-url-and-save-files-directory[17]https://www.youtube.com/watch?v=LyymFN9t4kw
- Save the script as
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.