Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0314· openaiHugging Face Suzhen/CodeChat

I am writing scrapers using residential proxies(quite expensive), which I noticed the…

  1. User

    I am writing scrapers using residential proxies(quite expensive), which I noticed the traffic was quite heavy and normally Selenium sends more than one request for a single URL. What can i do to reduce traffic using selenium in python?

  2. Assistant

    To reduce traffic while using Selenium in Python, consider the following techniques:

    1. Disable images and CSS: Images and CSS files can take up a significant amount of data. You can disable them using a browser profile or preferences.

    For Chrome:

    from selenium import webdriver
    
    chrome_options = webdriver.ChromeOptions()
    prefs = {"profile.managed_default_content_settings.images": 2, "profile.managed_default_content_settings.stylesheets": 2}
    chrome_options.add_experimental_option("prefs", prefs)
    driver = webdriver.Chrome(options=chrome_options)
    

    For Firefox:

    from selenium import webdriver
    
    firefox_profile = webdriver.FirefoxProfile()
    firefox_profile.set_preference("permissions.default.image", 2)  # disable images
    firefox_profile.set_preference("permissions.default.stylesheet", 2)  # disable CSS
    driver = webdriver.Firefox(firefox_profile)
    
    1. Block specific requests (like analytics, ads, fonts, etc.): You can block specific requests using Proxy Auto-Configuration (PAC).
    from selenium import webdriver
    
    chrome_options = webdriver.ChromeOptions()
    pac_script = """
    function FindProxyForURL(url, host) {
        if (shExpMatch(url, "*analytics*")) {
            return "PROXY 0.0.0.0:18080";
        }
        if (shExpMatch(url, "*ads*")) {
            return "PROXY 0.0.0.0:18080";
        }
        return "DIRECT";
    }
    """
    chrome_options.add_argument('--proxy-pac-url=data:text/pac,' + pac_script)
    driver = webdriver.Chrome(options=chrome_options)
    
    1. Set cache size and disk cache: You can enable disk cache and set cache size to reduce network requests.

    For Chrome:

    from selenium import webdriver
    
    chrome_options = webdriver.ChromeOptions()
    prefs = {"cache.enabled": True, "cache.cache_size": 1000000}
    chrome_options.add_experimental_option("prefs", prefs)
    driver = webdriver.Chrome(options=chrome_options)
    

    For Firefox:

    from selenium import webdriver
    
    firefox_profile = webdriver.FirefoxProfile()
    firefox_profile.set_preference("browser.cache.disk.enable", True)
    firefox_profile.set_preference("browser.cache.disk.capacity", 1000000)
    driver = webdriver.Firefox(firefox_profile)
    
    1. Use headless mode: Running the browser in headless mode will reduce the overall resource usage. Make sure to test your scraper in both headless and non-headless mode to ensure code compatibility.

    For Chrome:

    from selenium import webdriver
    
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument("--headless")
    driver = webdriver.Chrome(options=chrome_options)
    

    For Firefox:

    from selenium import webdriver
    
    firefox_options = webdriver.FirefoxOptions()
    firefox_options.add_argument("--headless")
    driver = webdriver.Firefox(options=firefox_options)
    
    1. Limit the browser's bandwidth: You can use browsermob-proxy to limit the bandwidth of the browser. You can find the library at https://github.com/AutomatedTester/browsermob-proxy-py.

    Remember to balance the trade-off between reducing traffic and maintaining the functionality of the websites being scraped when applying these techniques. Some websites might not work correctly with some of these optimizations. Make sure to test your scraper thoroughly after implementing these changes.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.