Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

please give me python code to scrape the web and save best fit of Images of following:…

  1. User

    please give me python code to scrape the web and save best fit of Images of following: علائم + following list: بانک آمبولانس بیمارستان آتش نشانی پلیس بیمارستان اداره آب و فاضلاب اداره گاز برق خطرناک هلال احمر

  2. Assistant

    Certainly! Below is a Python script that uses the icrawler library to search and download images for the specified Persian terms. This script automates image scraping from Google Images and saves them into organized directories.

    Prerequisites

    1. Python Installation: Ensure you have Python 3.x installed on your system. You can download it from here.

    2. Install Required Libraries:

      • icrawler: A powerful image crawler framework.
      • pip: Python package installer (usually comes with Python).

      Open your terminal or command prompt and run:

      pip install icrawler
      

    Python Script

    import os
    from icrawler.builtin import GoogleImageCrawler
    
    # Define the list of keywords in Persian
    keywords = [
        "بانک",
        "آمبولانس",
        "بیمارستان",
        "آتش نشانی",
        "پلیس",
        "اداره آب و فاضلاب",
        "اداره گاز",
        "برق خطرناک",
        "هلال احمر"
    ]
    
    # Directory where images will be saved
    output_dir = "downloaded_images"
    
    # Number of images to download per keyword
    num_images = 20  # You can adjust this number as needed
    
    def create_directory(path):
        """Create directory if it doesn't exist."""
        if not os.path.exists(path):
            os.makedirs(path)
    
    def download_images(keyword, save_dir, max_num):
        """Download images for a given keyword."""
        google_crawler = GoogleImageCrawler(storage={'root_dir': save_dir})
        print(f"Downloading images for keyword: {keyword}")
        google_crawler.crawl(
            keyword=keyword,
            max_num=max_num,
            min_size=(200, 200),  # Minimum size of images
            file_idx_offset='auto'
        )
        print(f"Finished downloading images for keyword: {keyword}\n")
    
    def main():
        # Create the main output directory
        create_directory(output_dir)
        
        for keyword in keywords:
            # Create a subdirectory for each keyword
            keyword_dir = os.path.join(output_dir, keyword)
            create_directory(keyword_dir)
            
            # Download images
            download_images(keyword, keyword_dir, num_images)
    
        print("All images have been downloaded successfully.")
    
    if __name__ == "__main__":
        main()
    

    How the Script Works

    1. Import Libraries:

      • os: For handling directory operations.
      • GoogleImageCrawler from icrawler.builtin: To perform image crawling from Google Images.
    2. Define Keywords:

      • The keywords list contains all the Persian terms for which you want to download images.
    3. Set Output Directory and Image Count:

      • output_dir: The main directory where all images will be stored.
      • num_images: Number of images to download per keyword. Adjust this based on your requirements.
    4. Directory Creation:

      • The create_directory function ensures that the required directories exist. If not, it creates them.
    5. Downloading Images:

      • The download_images function initializes the Google Image Crawler for each keyword and downloads the specified number of images into the corresponding directory.
      • min_size: Ensures that only images above a certain resolution are downloaded to maintain quality.
      • file_idx_offset='auto': Automatically manages file indexing to prevent overwriting.
    6. Execution Flow:

      • The main function orchestrates the creation of directories and initiates the download process for each keyword.
      • After processing all keywords, it prints a completion message.

    Running the Script

    1. Save the Script:

      • Save the above Python script in a file named, for example, download_images.py.
    2. Run the Script:

      • Navigate to the directory containing download_images.py using your terminal or command prompt.

      • Execute the script by running:

        python download_images.py
        
    3. Monitor the Download:

      • The script will print messages indicating the progress of image downloads for each keyword.
      • Images will be saved in the downloaded_images directory, organized into subdirectories named after each keyword.

    Notes and Best Practices

    • Respect Copyright:

      • Ensure that the images downloaded are used in compliance with their respective licenses. The icrawler library downloads images based on their availability on the web, but it doesn't check for usage rights.
    • Adjust Image Count:

      • Be mindful of the number of images you download to avoid excessive bandwidth usage or potential blocking by search engines. Adjust the num_images variable as needed.
    • Handle Duplicates:

      • The script does not handle duplicate images. If you run the script multiple times, it might download the same images again. Implement additional checks if this is a concern.
    • Performance Optimization:

      • For downloading a large number of images or handling many keywords, consider implementing multithreading or asynchronous processing to speed up the process.
    • Error Handling:

      • The script has basic print statements for progress tracking. For a more robust solution, consider adding try-except blocks to handle potential errors during the crawling process.

    Conclusion

    This script provides a straightforward way to automate the downloading of images based on Persian keywords using Python. By leveraging the icrawler library, you can efficiently gather a collection of images for various applications. Always ensure that you use the downloaded images responsibly and adhere to copyright laws.