Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

write a python script to scrape novel title from xpath //a[@class="novel-title"] and the…

  1. User

    write a python script to scrape novel title from xpath //a[@class="novel-title"] and the chapter title from the xpath //a[@class="chr-title"]/span and the chapter content , Grand Ancestral Bloodlines

                                $chapterComment = $('#fb-comment-chapter');
    
                                function toggleChapterComment() {
                                    if ($chapterComment.is(":hidden")) {
    
                                        (function () { // DON'T EDIT BELOW THIS LINE
                                            var d = document, s = d.createElement('script');
                                            s.src = 'https://webnovel-3.disqus.com/embed.js';
                                            s.setAttribute('data-timestamp', +new Date());
                                            (d.head || d.body).appendChild(s);
                                        })();
    
                                    }
                                    $chapterComment.toggle();
                                }
    
                                
                                setTimeout(function () {
    
                                    const paragraphss = $("p");
    
                                    paragraphss.each(function () {
                                        const original11Content = $(this).html();
                                        const updated11Content = original11Content.replace("", `<span id="span"></span>`);
                                        $(this).html(updated11Content);
                                    });
                                }, 600000);
                                
                            </script>
    

    Please enable JavaScript to view the comments powered by Disqus.

  2. Assistant

    Certainly! Below is a Python script that uses the requests library to fetch web pages and lxml with XPath support to parse and extract the novel title, chapter titles, and chapter content based on the provided XPath expressions. Additionally, it navigates through the chapters by following the "Next Chapter" link until no further chapters are found.

    Prerequisites

    Before running the script, ensure you have the necessary Python packages installed. You can install them using pip:

    pip install requests lxml
    

    Python Script

    import requests
    from lxml import html
    import time
    import os
    
    def fetch_page(url):
        """Fetches the content of a webpage."""
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
                          '(KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}
        try:
            response = requests.get(url, headers=headers)
            response.raise_for_status()
            return response.content
        except requests.RequestException as e:
            print(f"Error fetching {url}: {e}")
            return None
    
    def parse_page(content):
        """Parses the HTML content and extracts required information."""
        tree = html.fromstring(content)
        
        # Extract Novel Title
        novel_title_elements = tree.xpath('//a[@class="novel-title"]')
        novel_title = novel_title_elements[0].text_content().strip() if novel_title_elements else "Unknown Novel"
        
        # Extract Chapter Title
        chapter_title_elements = tree.xpath('//a[@class="chr-title"]/span')
        chapter_title = chapter_title_elements[0].text_content().strip() if chapter_title_elements else "Unknown Chapter"
        
        # Extract Chapter Content
        chapter_content_elements = tree.xpath('//div[@id="chr-content"]')
        if chapter_content_elements:
            # Remove scripts and other unwanted tags
            for script in chapter_content_elements[0].xpath('.//script'):
                script.getparent().remove(script)
            # Get text content preserving paragraphs
            paragraphs = chapter_content_elements[0].xpath('.//p/text()')
            chapter_content = "\n\n".join([para.strip() for para in paragraphs if para.strip()])
        else:
            chapter_content = "No Content Found."
        
        # Find Next Chapter URL
        next_chap_elements = tree.xpath('//a[@id="next_chap"]/@href')
        next_chap_url = next_chap_elements[0] if next_chap_elements else None
        
        return novel_title, chapter_title, chapter_content, next_chap_url
    
    def save_chapter(novel_title, chapter_title, chapter_content, output_dir):
        """Saves the chapter content to a text file."""
        # Ensure the output directory exists
        os.makedirs(output_dir, exist_ok=True)
        
        # Clean filenames
        safe_novel_title = "".join(c for c in novel_title if c.isalnum() or c in " _-").rstrip()
        safe_chapter_title = "".join(c for c in chapter_title if c.isalnum() or c in " _-").rstrip()
        
        filename = f"{safe_novel_title} - {safe_chapter_title}.txt"
        filepath = os.path.join(output_dir, filename)
        
        with open(filepath, 'w', encoding='utf-8') as f:
            f.write(f"{novel_title}\n\n")
            f.write(f"{chapter_title}\n\n")
            f.write(chapter_content)
        
        print(f"Saved: {filepath}")
    
    def scrape_novel(start_url, output_dir="novel_output"):
        """Main function to scrape the novel starting from the start_url."""
        current_url = start_url
        chapter_count = 0
        novel_title = ""
        
        while current_url:
            print(f"Fetching: {current_url}")
            content = fetch_page(current_url)
            if not content:
                print("Failed to retrieve content. Exiting.")
                break
            
            title, chapter_title, chapter_content, next_url = parse_page(content)
            
            if not novel_title:
                novel_title = title  # Assume the novel title remains the same
            
            save_chapter(novel_title, chapter_title, chapter_content, output_dir)
            
            chapter_count += 1
            print(f"Chapter {chapter_count} scraped: {chapter_title}\n")
            
            if next_url:
                current_url = next_url
                time.sleep(1)  # Be polite and avoid overwhelming the server
            else:
                print("No more chapters found. Scraping complete.")
                break
    
    if __name__ == "__main__":
        # Starting URL of the novel's first chapter
        start_chapter_url = "https://novelbin.com/b/grand-ancestral-bloodlines/chapter-1"
        
        # Directory where chapters will be saved
        output_directory = "Grand_Ancestral_Bloodlines"
        
        scrape_novel(start_chapter_url, output_directory)
    

    How the Script Works

    1. Fetching the Webpage:

      • The fetch_page function uses requests to get the HTML content of the given URL. It includes a User-Agent header to mimic a real browser, which helps in preventing the request from being blocked by the website.
    2. Parsing the HTML Content:

      • The parse_page function uses lxml to parse the HTML content.
      • It extracts the Novel Title using the XPath //a[@class="novel-title"].
      • It extracts the Chapter Title using the XPath //a[@class="chr-title"]/span.
      • It extracts the Chapter Content from the <div> with id="chr-content". It removes any <script> tags to avoid including JavaScript code and concatenates all paragraph texts.
      • It finds the Next Chapter URL by looking for the href attribute of the <a> tag with id="next_chap".
    3. Saving the Extracted Data:

      • The save_chapter function saves each chapter's content into a separate text file within the specified output directory. Filenames are sanitized to remove any illegal characters.
    4. Navigating Through Chapters:

      • The scrape_novel function orchestrates the scraping process. It starts from the initial chapter URL and iteratively follows the "Next Chapter" link until no further chapters are found.
      • It includes a short delay (time.sleep(1)) between requests to be polite to the website's server and avoid being rate-limited or blocked.
    5. Execution:

      • The script starts scraping from chapter-1 of "Grand Ancestral Bloodlines". You can change the start_chapter_url variable to point to a different starting chapter if needed.

    Notes and Best Practices

    • Respect Robots.txt and Terms of Service:

      • Before scraping any website, always check its robots.txt file and terms of service to ensure that you are allowed to scrape the content. Unauthorized scraping may violate the website's policies and legal regulations.
    • Rate Limiting:

      • Implementing delays between requests (as done with time.sleep(1)) helps prevent overloading the server and reduces the risk of being blocked.
    • Error Handling:

      • The script includes basic error handling for network-related issues. Depending on the website's structure and possible changes, you might need to enhance error handling and parsing logic.
    • Dynamic Content:

      • If the website uses JavaScript to load content dynamically, requests and lxml might not suffice since they don't execute JavaScript. In such cases, consider using tools like Selenium or Playwright which can interact with a real browser.
    • Extensibility:

      • This script is a basic example. Depending on your needs, you might want to extend it to handle more features like downloading images, handling multiple novels, or integrating with databases.

    Sample Output

    Upon running the script, you should see output similar to:

    Fetching: https://novelbin.com/b/grand-ancestral-bloodlines/chapter-1
    Saved: Grand_Ancestral_Bloodlines/Grand Ancestral Bloodlines - Chapter 1 Shrine Mountain.txt
    Chapter 1 scraped: Chapter 1: Shrine Mountain
    
    Fetching: https://novelbin.com/b/grand-ancestral-bloodlines/chapter-2
    Saved: Grand_Ancestral_Bloodlines/Grand Ancestral Bloodlines - Chapter 2 Nuri.txt
    Chapter 2 scraped: Chapter 2: Nuri
    
    ...
    No more chapters found. Scraping complete.
    

    Each chapter will be saved as a separate .txt file in the specified output directory (Grand_Ancestral_Bloodlines in this case).

    Conclusion

    This script provides a foundation for scraping novel titles, chapter titles, and chapter contents from websites using XPath selectors. Depending on the website's structure and your specific requirements, you may need to adjust the XPath expressions or extend the script's functionality.