Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0314· openaiHugging Face Suzhen/CodeChat

python, crawl the pdf's that are behind the download link that DOES NOT have a .pdf in…

  1. User

    python, crawl the pdf's that are behind the download link that DOES NOT have a .pdf in the inital link in this website https://www.hilti.ch/downloads?showAll=true&continue=true

  2. Assistant

    I cannot crawl the website directly, but I can guide you on how to do it using a Python library like BeautifulSoup and requests.

    First, make sure to install the necessary libraries:

    pip install beautifulsoup4
    pip install requests
    

    Here is a script to extract the non-PDF links on the Hilti.ch download page:

    from bs4 import BeautifulSoup
    import requests
    
    url = "https://www.hilti.ch/downloads?showAll=true&continue=true"
    response = requests.get(url)
    
    soup = BeautifulSoup(response.text, 'html.parser')
    
    all_links = soup.find_all("a", href=True)
    non_pdf_links = [link["href"] for link in all_links if not link["href"].endswith(".pdf")]
    
    print("Non-PDF links:")
    for link in non_pdf_links:
        print(link)
    

    To download the PDF files behind these links, you can modify the script to request these links and save the content in PDF files:

    import os
    
    output_directory = "output_pdfs"
    os.makedirs(output_directory, exist_ok=True)
    
    for link in non_pdf_links:
        response = requests.get(link)
        if response.headers['content-type'] == 'application/pdf':
            filename = os.path.join(output_directory, link.split("/")[-1] + ".pdf")
            with open(filename, 'wb') as f:
                f.write(response.content)
            print(f"Downloaded {filename}")
        else:
            print(f"Link {link} is not a downloadable PDF")
    

    This script will download the PDFs from the non-PDF links and save them in the output_pdfs folder. Keep in mind that this script can break if the structure of the website changes, and you may need to adapt it.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.