Skip to content
The Chat Archive
llama-3.3-70b-instruct· Meta LMArena Human Preference 140K

I want to create an application to extract all the note clippings that I have made from…

  1. User

    I want to create an application to extract all the note clippings that I have made from the books I read on Kindle. While the kind clippings Book has text extracts, I often mark text that is formatted as bullets, so I lose that formatting. I also highlight images of interest which can't be captured in the clippings. How do I write the logic to scour the books in EPUB /mobi/azwe format from my calibre library and and extract the original book formatting as html or images into new book of clippings.

  2. Assistant

    Extracting note clippings from Kindle books while preserving the original formatting can be a challenging task. To achieve this, you'll need to develop a script that can parse the book files, identify the highlighted text and images, and then export them to a new book with the desired formatting. Here's a high-level overview of the steps you can follow:

    Step 1: Choose a Programming Language and Required Libraries

    You can use Python as the programming language for this task, as it has excellent libraries for working with EPUB, MOBI, and AZW files. You'll need the following libraries:

    • ebooklib for working with EPUB files
    • mobi for working with MOBI files
    • pykdx for working with AZW files
    • BeautifulSoup for parsing HTML
    • Pillow for image processing

    Step 2: Parse the Book Files

    You'll need to write a function that can parse the book files and extract the text and images. Here's a basic example of how you can do this:

    import ebooklib
    from ebooklib import epub
    from mobi import Mobi
    from pykdx import KDX
    from bs4 import BeautifulSoup
    from PIL import Image
    
    def parse_epub(file_path):
        book = epub.read_epub(file_path)
        for doc in book.get_items():
            if doc.get_type() == ebooklib.ITEM_DOCUMENT:
                soup = BeautifulSoup(doc.get_content(), 'html.parser')
                # Extract text and images from the soup
                yield soup
    
    def parse_mobi(file_path):
        book = Mobi(file_path)
        for record in book.records:
            if record.type == 4:  # Text record
                soup = BeautifulSoup(record.content, 'html.parser')
                # Extract text and images from the soup
                yield soup
    
    def parse_azw(file_path):
        book = KDX(file_path)
        for section in book.sections:
            soup = BeautifulSoup(section.content, 'html.parser')
            # Extract text and images from the soup
            yield soup
    

    Step 3: Identify Highlighted Text and Images

    To identify the highlighted text and images, you'll need to access the Kindle's annotation database. Unfortunately, this database is not publicly accessible, and Amazon doesn't provide an official API for accessing it.

    However, you can use the kindle_highlights library, which provides a way to extract highlights from Kindle books. This library uses a combination of heuristics and machine learning algorithms to identify the highlighted text.

    import kindle_highlights
    
    def extract_highlights(file_path):
        highlights = kindle_highlights.extract_highlights(file_path)
        return highlights
    

    Step 4: Export Highlights to a New Book

    Once you have the highlighted text and images, you can export them to a new book using a library like ebooklib. You can create a new EPUB book and add the highlighted text and images to it.

    import ebooklib
    
    def create_new_book(highlights):
        book = epub.EpubBook()
        for highlight in highlights:
            # Create a new chapter for each highlight
            chapter = epub.EpubHtml(title=highlight['title'], file_name=highlight['file_name'])
            chapter.set_content(highlight['content'])
            book.add_item(chapter)
        return book
    
    def save_book(book, file_path):
        epub.write_epub(file_path, book)
    

    Step 5: Integrate with Calibre Library

    To integrate your script with the Calibre library, you can use the calibre library, which provides a Python API for accessing the Calibre database.

    import calibre
    
    def get_books_from_calibre(library_path):
        db = calibre.db(library_path)
        books = db.get_all_books()
        return books
    
    def get_book_path(book):
        return book.path
    
    def main():
        library_path = '/path/to/calibre/library'
        books = get_books_from_calibre(library_path)
        for book in books:
            file_path = get_book_path(book)
            highlights = extract_highlights(file_path)
            new_book = create_new_book(highlights)
            save_book(new_book, 'highlights.epub')
    
    if __name__ == '__main__':
        main()
    

    Note that this is a high-level overview of the steps you can follow, and you'll need to modify the code to suit your specific requirements. Additionally, you may need to handle errors and exceptions that may occur during the execution of the script.

    Also, please note that this script is for personal use only, and you should respect the copyright laws and terms of service of the books you are processing.

    Example Use Case

    Let's say you have a Calibre library with several books, and you want to extract the highlights from all of them. You can run the script, and it will create a new EPUB book with the highlighted text and images from all the books in your library.

    python script.py
    

    This will create a new file called highlights.epub in the current directory, which contains the highlighted text and images from all the books in your Calibre library.

    Advice

    • Make sure to handle errors and exceptions properly to avoid crashes and data loss.
    • Use a consistent naming convention for your variables and functions to make the code easier to read and understand.
    • Consider using a more robust library for parsing EPUB, MOBI, and AZW files, such as ebooklib or mobi.
    • Use a more efficient algorithm for extracting highlights, such as using a machine learning model or a natural language processing library.
    • Consider adding more features to the script, such as the ability to filter highlights by book or author, or to export highlights to a different format, such as PDF or CSV.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.