Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-reasoning-pro-high· PerplexityHugging Face LMArena Search Arena 24K

Write an R script that produces a dataframe of the most popular downloaded books on…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-pro-high’s answer.
  1. User

    Write an R script that produces a dataframe of the most popular downloaded books on Project Gutenberg with the gutenbergr package

  2. Perplexity Assistant

    Creating a Dataframe of Popular Project Gutenberg Books with gutenbergr

    The gutenbergr package provides a convenient way to access and download public domain works from Project Gutenberg. While it doesn't directly provide popularity metrics, we can create a script that generates a useful dataframe of notable books that are likely to be popular.

    Here's an R script that produces a dataframe of books from Project Gutenberg that are likely to be popular or notable:

    # Install and load required packages
    if (!require("gutenbergr")) {
      install.packages("gutenbergr")
    }
    if (!require("dplyr")) {
      install.packages("dplyr")
    }
    
    library(gutenbergr)
    library(dplyr)
    
    # Get latest metadata (updated as of September 14, 2024)
    gutenberg_metadata_date <- attr(gutenberg_metadata, "date_updated")
    cat("Using Project Gutenberg metadata updated on:", gutenberg_metadata_date, "\n")
    
    # Create a function to find popular books based on various criteria
    get_popular_books <- function(n = 50) {
      # Start with English language works that have text available
      popular_books <- gutenberg_works() %>%
        # Filter for works with recognized authors (not NULL)
        filter(!is.na(author)) %>%
        # Join with subjects to get more context
        left_join(
          gutenberg_subjects %>%
            filter(subject_type == "lcsh") %>%
            group_by(gutenberg_id) %>%
            summarize(subjects = paste(subject, collapse = " | ")),
          by = "gutenberg_id"
        )
      
      # Extract classic literature by joining with authors dataset
      popular_classics <- popular_books %>%
        left_join(gutenberg_authors, by = c("gutenberg_author_id" = "gutenberg_author_id")) %>%
        # Find authors born before 1900 (likely classics)
        filter(!is.na(birthdate) & birthdate < 1900) %>%
        # Select fields for our final dataframe
        select(gutenberg_id, title, author, birthdate, deathdate, subjects) %>%
        # Get the top entries
        head(n)
      
      # Return our results
      return(popular_classics)
    }
    
    # Create a function to find the most downloaded books (based on available metadata)
    get_classic_fiction <- function(n = 25) {
      fiction_books <- gutenberg_works() %>%
        # Join with subjects
        inner_join(
          gutenberg_subjects %>%
            filter(
              subject_type == "lcsh",
              # Look for fiction categories
              grepl("Fiction", subject) | 
              grepl("Novel", subject) |
              grepl("story", subject, ignore.case = TRUE)
            ),
          by = "gutenberg_id"
        ) %>%
        # Select unique books
        distinct(gutenberg_id, .keep_all = TRUE) %>%
        select(gutenberg_id, title, author, subject) %>%
        head(n)
      
      return(fiction_books)
    }
    
    # Create a function to find books by specific popular authors
    get_books_by_popular_authors <- function(n = 25) {
      # List of some popular authors
      popular_authors <- c(
        "Shakespeare, William",
        "Austen, Jane", 
        "Dickens, Charles",
        "Doyle, Arthur Conan",
        "Twain, Mark",
        "Poe, Edgar Allan",
        "Wilde, Oscar"
      )
      
      books_by_popular_authors <- gutenberg_works() %>%
        filter(author %in% popular_authors) %>%
        select(gutenberg_id, title, author) %>%
        # Sort by author then title
        arrange(author, title) %>%
        head(n)
      
      return(books_by_popular_authors)
    }
    
    # Generate our dataframes
    popular_classics <- get_popular_books(50)
    classic_fiction <- get_classic_fiction(25)
    popular_author_books <- get_books_by_popular_authors(25)
    
    # Print a summary
    cat("Found", nrow(popular_classics), "popular classic books\n")
    cat("Found", nrow(classic_fiction), "classic fiction works\n")
    cat("Found", nrow(popular_author_books), "books by popular authors\n")
    
    # Example of how to download one of these books
    if (interactive()) {
      # Get the ID of a popular book (e.g., Pride and Prejudice)
      pride_and_prejudice <- popular_author_books %>%
        filter(title == "Pride and Prejudice")
      
      if (nrow(pride_and_prejudice) > 0) {
        book_id <- pride_and_prejudice$gutenberg_id[1]
        # Download the book text
        book_text <- gutenberg_download(book_id)
        cat("Downloaded", nrow(book_text), "lines of text from book ID:", book_id, "\n")
      }
    }
    
    # Return our main dataframe
    popular_classics
    

    How the Script Works

    1. Installing and Loading Packages: The script starts by ensuring the necessary packages are installed and loaded.

    2. Checking Metadata Date: The script displays when the Project Gutenberg metadata was last updated, which as of writing is September 14, 2024[1][5].

    3. Finding Popular Classic Books: The script creates a function that:

      • Starts with English language works from the gutenberg_works() function
      • Joins with subject data for more context
      • Filters for authors from before 1900 to focus on classics
      • Creates a comprehensive dataframe with book IDs, titles, authors, and subjects[2][8]
    4. Finding Classic Fiction: A second function focuses specifically on fiction works by looking for fiction-related subjects.

    5. Books by Popular Authors: The third function creates a dataframe of works by specific authors known to be popular, such as Shakespeare, Austen, and Dickens[10].

    6. Downloading Example: The script includes an example of how to download the text of one of the identified popular books using gutenberg_download()[5].

    Example Output

    The resulting dataframe (popular_classics) includes columns for:

    • gutenberg_id: The unique identifier for downloading the book
    • title: The book's title
    • author: The author's name
    • birthdate: When the author was born
    • deathdate: When the author died
    • subjects: Library of Congress subject classifications

    This script provides a useful starting point for identifying potentially popular or notable books in the Project Gutenberg collection. While it doesn't directly measure download counts (which would require scraping the Project Gutenberg website), it uses available metadata to identify books that are likely to be popular based on their classic status and authors.

  3. User

    How could I use the pageviews package to identify trends in popular books people are looking into?

  4. Perplexity Assistant

    Using the pageviews Package to Identify Book Popularity Trends

    The pageviews R package provides access to Wikimedia's traffic data, which you can leverage to identify trends in book popularity by analyzing Wikipedia page traffic patterns. This approach allows you to see which books are receiving attention over time, potentially indicating rising or declining interest in specific titles, authors, or genres.

    Setting Up Your Environment

    First, install and load the necessary packages:

    # Install packages if needed
    if (!require("pageviews")) {
      install.packages("pageviews")
    }
    if (!require("dplyr")) {
      install.packages("dplyr")
    }
    if (!require("ggplot2")) {
      install.packages("ggplot2")
    }
    
    # Load packages
    library(pageviews)
    library(dplyr)
    library(ggplot2)
    

    Tracking Individual Book Trends

    You can analyze how interest in specific books changes over time by examining Wikipedia pageview data for those titles:

    # Function to get pageviews for a list of books
    get_book_pageviews <- function(book_titles, start_date, end_date) {
      results <- data.frame()
      
      for (title in book_titles) {
        # Get pageviews for this book
        book_data <- article_pageviews(
          article = title,
          project = "en.wikipedia",
          start = pageview_timestamps(start_date),
          end = pageview_timestamps(end_date),
          granularity = "daily"
        )
        
        # Add to our results
        results <- rbind(results, book_data)
      }
      
      return(results)
    }
    
    # Example: Track popular books
    popular_books <- c(
      "To_Kill_a_Mockingbird",
      "1984_(novel)",
      "The_Great_Gatsby",
      "Pride_and_Prejudice",
      "Harry_Potter_and_the_Philosopher's_Stone"
    )
    
    # Get data for the past 3 months
    three_months_ago <- Sys.Date() - 90
    book_trends <- get_book_pageviews(popular_books, three_months_ago, Sys.Date())
    

    Identifying Trending Books

    You can discover which books are currently trending by examining the top articles in the literature category:

    # Get top articles, then filter for books
    literature_trending <- top_articles(
      project = "en.wikipedia",
      platform = "all",
      start = Sys.Date() - 7  # Last week
    )
    
    # Filter for book-related articles
    book_keywords <- c("novel", "book", "fiction", "author", "literature")
    trending_books <- literature_trending %>%
      filter(grepl(paste(book_keywords, collapse="|"), article, ignore.case=TRUE))
    
    # Display top 10 trending book-related articles
    head(trending_books, 10)
    

    Comparing Book Categories or Genres

    You can also compare different book categories to identify broader trends:

    # Define lists of books by genre
    science_fiction <- c("Dune_(novel)", "Neuromancer", "The_Hitchhiker's_Guide_to_the_Galaxy")
    fantasy <- c("The_Lord_of_the_Rings", "A_Game_of_Thrones", "The_Name_of_the_Wind")
    classic_literature <- c("Moby-Dick", "War_and_Peace", "Jane_Eyre")
    
    # Get pageviews for each genre
    start_date <- as.Date("2025-01-01")  # Beginning of the year
    sci_fi_views <- get_book_pageviews(science_fiction, start_date, Sys.Date())
    fantasy_views <- get_book_pageviews(fantasy, start_date, Sys.Date())
    classics_views <- get_book_pageviews(classic_literature, start_date, Sys.Date())
    
    # Add genre column to each dataset
    sci_fi_views$genre <- "Science Fiction"
    fantasy_views$genre <- "Fantasy"
    classics_views$genre <- "Classic Literature"
    
    # Combine datasets
    genre_comparison <- rbind(sci_fi_views, fantasy_views, classics_views)
    
    # Summarize by genre and date
    genre_summary <- genre_comparison %>%
      group_by(date, genre) %>%
      summarize(total_views = sum(views))
    

    Visualizing Book Trends

    Create visualizations to better understand the trends:

    # Plot trends for individual books
    ggplot(book_trends, aes(x = date, y = views, color = article)) +
      geom_line() +
      theme_minimal() +
      labs(title = "Wikipedia Pageviews for Popular Books",
           x = "Date", 
           y = "Number of Views",
           color = "Book") +
      theme(legend.position = "bottom")
    
    # Plot genre comparison
    ggplot(genre_summary, aes(x = date, y = total_views, color = genre)) +
      geom_line() +
      theme_minimal() +
      labs(title = "Comparison of Book Genre Popularity",
           x = "Date", 
           y = "Total Pageviews",
           color = "Genre") +
      theme(legend.position = "bottom")
    

    Analyzing Seasonal Trends

    Identify patterns in book popularity across different seasons or time periods:

    # Function to aggregate monthly data
    get_seasonal_book_trends <- function(book_titles, years = 1) {
      start_date <- Sys.Date() - (365 * years)
      
      results <- data.frame()
      for (title in book_titles) {
        # Get data with monthly granularity
        book_data <- article_pageviews(
          article = title,
          project = "en.wikipedia",
          start = pageview_timestamps(start_date),
          end = pageview_timestamps(Sys.Date()),
          granularity = "monthly"
        )
        results <- rbind(results, book_data)
      }
      
      # Extract month from date
      results$month <- format(results$date, "%m")
      return(results)
    }
    
    # Get seasonal data for popular books
    seasonal_trends <- get_seasonal_book_trends(popular_books)
    
    # Summarize by month
    monthly_summary <- seasonal_trends %>%
      group_by(month, article) %>%
      summarize(avg_views = mean(views))
    

    Advanced Analysis: Book Popularity Following Media Adaptations

    Investigate how book pageviews change following movie or TV adaptations:

    # Function to examine pageviews before and after an adaptation release
    analyze_adaptation_impact <- function(book_title, adaptation_date, window_days = 30) {
      # Convert to Date object
      adaptation_date <- as.Date(adaptation_date)
      
      # Get data for window before and after
      before_start <- adaptation_date - window_days
      after_end <- adaptation_date + window_days
      
      impact_data <- article_pageviews(
        article = book_title,
        project = "en.wikipedia",
        start = pageview_timestamps(before_start),
        end = pageview_timestamps(after_end),
        granularity = "daily"
      )
      
      # Add a flag for before/after
      impact_data$period <- ifelse(impact_data$date < adaptation_date, 
                                  "Before Adaptation", 
                                  "After Adaptation")
      
      return(impact_data)
    }
    
    # Example: Analyze "Dune" before and after the 2024 movie release
    dune_impact <- analyze_adaptation_impact("Dune_(novel)", "2024-03-01")
    
    # Compare average views
    dune_summary <- dune_impact %>%
      group_by(period) %>%
      summarize(avg_daily_views = mean(views))
    

    Cross-Referencing with Bestseller Lists

    While not directly available through pageviews, you could create a dataset that cross-references Wikipedia traffic with bestseller lists:

    # Hypothetical bestseller list (you would need to obtain this data separately)
    bestsellers_2025 <- data.frame(
      title = c("New_Bestseller_1", "New_Bestseller_2", "New_Bestseller_3"),
      list_rank = c(1, 2, 3),
      stringsAsFactors = FALSE
    )
    
    # Get pageviews for these bestsellers
    bestseller_views <- get_book_pageviews(
      bestsellers_2025$title, 
      Sys.Date() - 30, 
      Sys.Date()
    )
    
    # Merge with bestseller ranks
    bestseller_analysis <- bestseller_views %>%
      left_join(bestsellers_2025, by = c("article" = "title"))
    

    Considerations and Limitations

    1. Wikipedia as a proxy: Wikipedia pageviews are an indirect measure of interest rather than actual sales or readership.

    2. Article naming conventions: You'll need to know the exact Wikipedia article title (with underscores instead of spaces).

    3. External factors: Pageviews might spike due to media coverage, controversies, or adaptations rather than actual reading interest.

    4. Historical data: For broader historical analysis, consider using old_pageviews() for data from 2008-2016.

    5. API limitations: Be mindful of rate limits when making numerous requests.

    By systematically analyzing Wikipedia pageview data, you can gain valuable insights into book popularity trends that can complement traditional publishing metrics and inform marketing strategies, inventory decisions, or research questions about literary interests.

  5. User

    write me a function that downloads a project gutenberg text and then spilts the text by sentence

  6. Perplexity Assistant

    Function to Download and Split Project Gutenberg Texts by Sentence

    Here's a comprehensive R function that downloads texts from Project Gutenberg and splits them into sentences:

    download_and_split_by_sentence <- function(gutenberg_id, 
                                              method = "tokenizers", 
                                              strip_headers = TRUE,
                                              meta_fields = c("title", "author")) {
      
      # Install required packages if not already installed
      required_packages <- c("gutenbergr", "stringr", "dplyr")
      
      # Add packages based on the method chosen
      if (method == "tokenizers") {
        required_packages <- c(required_packages, "tokenizers")
      } else if (method == "corpus") {
        required_packages <- c(required_packages, "corpus")
      }
      
      # Check and install missing packages
      new_packages <- required_packages[!required_packages %in% installed.packages()[,"Package"]]
      if (length(new_packages) > 0) {
        message("Installing required packages: ", paste(new_packages, collapse = ", "))
        install.packages(new_packages)
      }
      
      # Load required packages
      library(gutenbergr)
      library(stringr)
      library(dplyr)
      
      # Download the text from Project Gutenberg
      message("Downloading text with ID: ", gutenberg_id)
      book_text <- gutenberg_download(gutenberg_id, 
                                     strip = strip_headers,
                                     meta_fields = meta_fields)
      
      # Extract metadata
      metadata <- book_text %>%
        select(any_of(meta_fields)) %>%
        distinct()
      
      # Extract and collapse text
      full_text <- paste(book_text$text, collapse = " ")
      
      # Clean the text by handling multiple spaces and line breaks
      full_text <- str_replace_all(full_text, "\\s+", " ")
      
      message("Splitting text by sentences using method: ", method)
      
      # Split text into sentences based on chosen method
      sentences <- switch(method,
        "tokenizers" = {
          # Use tokenizers package
          library(tokenizers)
          unlist(tokenize_sentences(full_text))
        },
        "corpus" = {
          # Use corpus package
          library(corpus)
          unlist(text_split(full_text, units = "sentences"))
        },
        "regex" = {
          # Use regex approach
          # This pattern looks for sentence boundaries: periods, question marks, or exclamation marks
          # followed by whitespace and an uppercase letter
          sentence_pattern <- "(?<=[.!?])\\s+(?=[A-Z])"
          str_split(full_text, sentence_pattern, simplify = FALSE)[[1]]
        },
        "basic" = {
          # Simple approach replacing ending punctuation with a code word then splitting
          coded_text <- str_replace_all(full_text, "[.!?;]", "SENTENCEBREAK")
          sentences <- unlist(str_split(coded_text, "SENTENCEBREAK"))
          # Remove empty sentences and trim whitespace
          sentences <- sentences[sentences != ""]
          trimws(sentences)
        },
        {
          # Default method if none of the above match
          warning("Unknown method '", method, "'. Using 'regex' method instead.")
          sentence_pattern <- "(?<=[.!?])\\s+(?=[A-Z])"
          str_split(full_text, sentence_pattern, simplify = FALSE)[[1]]
        }
      )
      
      # Clean sentences by trimming whitespace
      sentences <- trimws(sentences)
      
      # Remove empty sentences
      sentences <- sentences[sentences != ""]
      
      # Create a result list containing both metadata and sentences
      result <- list(
        metadata = metadata,
        sentences = sentences,
        sentence_count = length(sentences),
        method = method
      )
      
      message("Successfully split text into ", length(sentences), " sentences")
      return(result)
    }
    

    Usage Examples

    Here are several ways to use this function:

    # Example 1: Download and split "Pride and Prejudice" using the default tokenizers method
    pride_and_prejudice <- download_and_split_by_sentence(1342)
    
    # Example 2: Download "The Adventures of Sherlock Holmes" using the corpus package
    sherlock <- download_and_split_by_sentence(1661, method = "corpus")
    
    # Example 3: Download "Moby Dick" using regex approach
    moby_dick <- download_and_split_by_sentence(2701, method = "regex")
    
    # Example 4: Download "A Tale of Two Cities" using the basic approach
    tale_of_two_cities <- download_and_split_by_sentence(98, method = "basic")
    
    # Display information about the text
    cat("Title:", pride_and_prejudice$metadata$title, "\n")
    cat("Author:", pride_and_prejudice$metadata$author, "\n")
    cat("Number of sentences:", pride_and_prejudice$sentence_count, "\n")
    
    # Display the first 5 sentences
    cat("First 5 sentences:\n")
    for (i in 1:5) {
      cat(i, ": ", pride_and_prejudice$sentences[i], "\n", sep="")
    }
    

    How It Works

    The function operates in several stages:

    1. Package Management: The function checks for and installs any required packages based on the chosen sentence splitting method.

    2. Text Download: It uses gutenberg_download() from the gutenbergr package to download the specified text by its Gutenberg ID. The text comes with metadata like title and author.

    3. Text Preparation: The downloaded text is collapsed into a single string and cleaned to handle spacing issues.

    4. Sentence Splitting: The function offers four different methods for splitting text into sentences:

      • tokenizers: Uses the tokenize_sentences() function from the tokenizers package, which is designed specifically for natural language processing.

      • corpus: Uses the text_split() function from the corpus package, which provides more advanced text segmentation capabilities.

      • regex: Uses a regular expression approach to identify sentence boundaries (periods, question marks, or exclamation marks followed by a space and an uppercase letter).

      • basic: A simpler approach that replaces ending punctuation with a code word and then splits on that code.

    5. Post-processing: The function cleans the sentences by trimming whitespace and removing empty entries.

    6. Result Structure: Returns a list containing:

      • The metadata from the original download
      • The split sentences
      • The total sentence count
      • The method used for splitting

    Considerations for Sentence Splitting

    Each sentence splitting method has advantages and limitations:

    1. tokenizers: Generally works well for most texts and handles many edge cases, but requires the additional tokenizers package.

    2. corpus: Provides advanced functionality for text segmentation but requires the corpus package.

    3. regex: Works without additional packages but may struggle with abbreviations (like "Mr.", "Dr.", etc.) or quotations that cross sentence boundaries.

    4. basic: The simplest approach, but least accurate as it doesn't properly handle abbreviations or complex punctuation patterns.

    The most accurate results typically come from the specialized NLP packages (tokenizers or corpus), while the regex and basic approaches are more prone to errors with complex texts.

    If working with literary texts that have complex sentence structures, quotations, or dialogue, the tokenizers method is generally recommended for its balance of accuracy and speed.

  7. User

    Make another function that gets a chapter dataframe from the book dataframe

  8. Perplexity Assistant

    Function to Extract Chapters from Project Gutenberg Texts

    Here's a function that builds on the previous one to identify and extract chapters from a Project Gutenberg text:

    extract_chapters <- function(book_data, 
                                chapter_pattern = NULL,
                                include_chapter_titles = TRUE,
                                handle_preamble = TRUE) {
      
      # Load required packages
      required_packages <- c("dplyr", "stringr", "tidyr")
      new_packages <- required_packages[!required_packages %in% installed.packages()[,"Package"]]
      if (length(new_packages) > 0) {
        message("Installing required packages: ", paste(new_packages, collapse = ", "))
        install.packages(new_packages)
      }
      
      library(dplyr)
      library(stringr)
      library(tidyr)
      
      # Check if input is from our download_and_split_by_sentence function
      if (is.list(book_data) && "sentences" %in% names(book_data)) {
        sentences <- book_data$sentences
        metadata <- book_data$metadata
      } else if ("text" %in% names(book_data)) {
        # Input is a raw gutenbergr download
        sentences <- paste(book_data$text, collapse = " ")
        sentences <- unlist(str_split(sentences, "(?<=\\.)\\s+(?=[A-Z])"))
        sentences <- trimws(sentences)
        sentences <- sentences[sentences != ""]
        
        # Extract metadata
        metadata_cols <- setdiff(names(book_data), "text")
        metadata <- book_data %>%
          select(all_of(metadata_cols)) %>%
          distinct()
      } else {
        stop("Input must be either output from download_and_split_by_sentence function or a gutenbergr download dataframe")
      }
      
      # Determine default chapter pattern based on text content if not provided
      if (is.null(chapter_pattern)) {
        # Test a few common patterns against the first 100 sentences to auto-detect format
        sample_text <- paste(sentences[1:min(100, length(sentences))], collapse=" ")
        
        patterns <- list(
          standard = "(?i)\\b(chapter|chap\\.?)\\s+([0-9]+|[IVXLCDM]+)\\b",
          roman = "(?i)\\b(chapter|chap\\.?)\\s+([IVXLCDM]+)\\b",
          spelled = "(?i)\\b(chapter|chap\\.?)\\s+(one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve|thirteen|fourteen|fifteen|sixteen|seventeen|eighteen|nineteen|twenty)\\b",
          numbered = "(?i)\\b([0-9]+|[IVXLCDM]+)\\b"
        )
        
        # Check which pattern has the most matches
        matches <- sapply(patterns, function(p) length(str_extract_all(sample_text, p)[[1]]))
        
        if (max(matches) > 0) {
          pattern_name <- names(which.max(matches))
          chapter_pattern <- patterns[[pattern_name]]
          message("Auto-detected chapter pattern: ", pattern_name)
        } else {
          # Default to standard pattern if no matches found
          chapter_pattern <- patterns$standard
          message("No chapter pattern detected. Using standard pattern.")
        }
      }
      
      # Rebuild the full text with sentence markers for easier chapter detection
      full_text &lt;- paste(sentences, collapse = " <sentence_break> ")
      
      # Find all chapter markers
      chapter_markers &lt;- str_extract_all(full_text, chapter_pattern)[[1]]
      
      if (length(chapter_markers) == 0) {
        warning("No chapters detected with the specified pattern. Returning single chapter.")
        chapters_df &lt;- data.frame(
          chapter_number = 1,
          chapter_title = "Full Text",
          chapter_text = paste(sentences, collapse = " "),
          sentence_count = length(sentences),
          start_sentence = 1,
          end_sentence = length(sentences)
        )
        
        result &lt;- list(
          metadata = metadata,
          chapters = chapters_df,
          chapter_count = 1
        )
        
        return(result)
      }
      
      # Split text by chapter pattern
      chapter_splits &lt;- str_split(full_text, chapter_pattern)[[1]]
      
      # Process chapter text
      chapters &lt;- data.frame(
        chapter_marker = c("Preamble", chapter_markers),
        chapter_text = chapter_splits,
        stringsAsFactors = FALSE
      ) %&gt;%
        mutate(
          # Extract chapter numbers
          chapter_number = case_when(
            chapter_marker == "Preamble" ~ 0,
            grepl("(?i)chapter|chap", chapter_marker) ~ 
              as.numeric(factor(str_extract(chapter_marker, "(?&lt;=(chapter|chap\\.?)\\s+)[0-9IVXLCDM]+"),
                                levels = unique(str_extract(chapter_markers, "(?&lt;=(chapter|chap\\.?)\\s+)[0-9IVXLCDM]+")))),
            TRUE ~ row_number()
          ),
          
          # Create chapter titles if requested
          chapter_title = case_when(
            chapter_marker == "Preamble" ~ "Preamble",
            include_chapter_titles &amp; grepl(":", chapter_text) ~ 
              trimws(str_extract(chapter_text, "^.*?(?=\\.)")),
            TRUE ~ paste("Chapter", chapter_number)
          )
        )
      
      # Convert sentence-break markers back to sentences and count them
      chapters &lt;- chapters %&gt;%
        mutate(
          sentences = str_split(chapter_text, "<sentence_break>"),
          sentence_count = sapply(sentences, length),
          # Clean up the sentences
          sentences = lapply(sentences, trimws),
          sentences = lapply(sentences, function(s) s[s != ""])
        )
      
      # Handle the preamble (text before the first chapter)
      if (!handle_preamble &amp;&amp; nrow(chapters) &gt; 1) {
        chapters &lt;- chapters[-1, ]
      }
      
      # Calculate sentence indices
      chapters &lt;- chapters %&gt;%
        mutate(
          start_sentence = cumsum(c(1, head(sentence_count, -1))),
          end_sentence = cumsum(sentence_count)
        )
      
      # Final cleanup and structure
      chapters_df &lt;- chapters %&gt;%
        select(chapter_number, chapter_title, chapter_text, sentence_count, start_sentence, end_sentence)
      
      # Add sentences as a separate list column if needed
      chapters_df$sentences &lt;- chapters$sentences
      
      # Create the result structure
      result &lt;- list(
        metadata = metadata,
        chapters = chapters_df,
        chapter_count = nrow(chapters_df)
      )
      
      message("Successfully extracted ", result$chapter_count, " chapters")
      return(result)
    }
    

    Usage Examples

    Here's how to use this function with the previous function:

    # First download and split a book by sentences
    book_data &lt;- download_and_split_by_sentence(1342) # Pride and Prejudice
    
    # Then extract chapters
    pride_chapters &lt;- extract_chapters(book_data)
    
    # Alternative: provide your own chapter pattern
    moby_dick &lt;- download_and_split_by_sentence(2701) # Moby Dick
    moby_chapters &lt;- extract_chapters(moby_dick, 
                                     chapter_pattern = "(?i)chapter\\s+[0-9IVXLCDM]+")
    
    # Work with books that have named chapters
    david_copperfield &lt;- download_and_split_by_sentence(766) # David Copperfield
    copperfield_chapters &lt;- extract_chapters(david_copperfield, 
                                            include_chapter_titles = TRUE)
    
    # Display chapter information
    cat("Book:", pride_chapters$metadata$title, "by", pride_chapters$metadata$author, "\n")
    cat("Number of chapters:", pride_chapters$chapter_count, "\n\n")
    
    # Show chapter information
    print(select(pride_chapters$chapters, chapter_number, chapter_title, sentence_count))
    
    # Get text from a specific chapter
    chapter_3 &lt;- pride_chapters$chapters$chapter_text[pride_chapters$chapters$chapter_number == 3]
    cat("First 200 characters of Chapter 3:\n", substr(chapter_3, 1, 200), "...\n")
    
    # Analysis: Find the longest and shortest chapters
    chapter_lengths &lt;- pride_chapters$chapters %&gt;%
      mutate(word_count = stringr::str_count(chapter_text, "\\w+")) %&gt;%
      select(chapter_number, chapter_title, word_count, sentence_count)
    
    longest_chapter &lt;- chapter_lengths %&gt;% 
      filter(word_count == max(word_count))
    shortest_chapter &lt;- chapter_lengths %&gt;% 
      filter(word_count == min(word_count))
    
    cat("Longest chapter:", longest_chapter$chapter_title, "with", 
        longest_chapter$word_count, "words and", longest_chapter$sentence_count, "sentences\n")
    
    cat("Shortest chapter:", shortest_chapter$chapter_title, "with", 
        shortest_chapter$word_count, "words and", shortest_chapter$sentence_count, "sentences\n")
    

    How the Chapter Extraction Works

    Smart Pattern Detection

    The function uses several approaches to identify chapters:

    1. Auto-detection: If no pattern is provided, it samples the first portion of the text and tests different common chapter heading formats.

    2. Pattern Options:

      • Standard format: "Chapter 1", "Chapter 2", etc.
      • Roman numeral format: "Chapter I", "Chapter II", etc.
      • Spelled out format: "Chapter One", "Chapter Two", etc.
      • Simple numbered format: Just numbers or roman numerals
    3. Chapter Title Handling: The function can extract chapter titles when they follow patterns like "Chapter 1: The Beginning".

    Chapter Splitting Process

    The function:

    1. Rebuilds the text with sentence markers to maintain sentence boundaries during chapter splitting
    2. Identifies chapter markers throughout the text
    3. Splits the text at these markers to create chapter segments
    4. Processes each segment to extract chapter numbers, titles, and content
    5. Rebuilds the sentence structure within each chapter
    6. Calculates statistics like sentence count and position indices

    Special Case Handling

    1. Preamble Text: Text before the first chapter marker (preface, introduction, etc.) is handled as a "Chapter 0" or can be excluded.

    2. Single Chapter Books: If no chapters are detected, the function returns the entire text as a single chapter.

    3. Chapter Titles: The function attempts to extract meaningful chapter titles when available, or defaults to "Chapter X" otherwise.

    Benefits

    This function provides several benefits for analyzing literary texts:

    1. Structure Analysis: Break down books into their structural components for analysis

    2. Chapter-Level Metrics: Generate statistics about chapter length, complexity, and patterns

    3. Selective Processing: Work with specific chapters instead of entire texts

    4. Content Extraction: Easily pull out chapter titles and other structured elements

    5. Compatibility: Works seamlessly with the previous sentence-splitting function

    When combined with the previous sentence-splitting function, this creates a powerful toolkit for processing and analyzing Project Gutenberg texts at various structural levels.</sentence_break></sentence_break>

  9. User

    How do I finetune a tinyllama model

  10. Perplexity Assistant

    Fine-Tuning TinyLlama: A Comprehensive Guide

    TinyLlama is a compact but powerful language model trained on approximately 3 trillion tokens, making it suitable for edge devices and applications where computational resources are limited. Despite its small size (1.1B parameters), it can be effectively fine-tuned for specific tasks. Here's how to fine-tune TinyLlama on your custom dataset.

    Setup and Prerequisites

    Hardware Requirements

    • A GPU with at least 8GB VRAM (T4 on Google Colab works well)
    • 16GB+ system RAM

    Software Dependencies

    # Install required packages
    pip install transformers datasets accelerate bitsandbytes trl peft
    

    For faster fine-tuning with Unsloth:

    pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
    pip install --no-deps "xformers&lt;0.0.27" "trl&lt;0.9.0" peft accelerate bitsandbytes
    

    Approach 1: Fine-Tuning with Hugging Face Transformers

    1. Load Model and Tokenizer

    from transformers import AutoModelForCausalLM, AutoTokenizer
    import torch
    
    model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"  # Base chat model
    
    # Load tokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    tokenizer.pad_token = tokenizer.eos_token  # Set padding token to EOS token
    
    # Load model with quantization for memory efficiency
    from transformers import BitsAndBytesConfig
    
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.float16
    )
    
    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        quantization_config=bnb_config,
        device_map="auto"
    )
    

    2. Prepare the Dataset

    from datasets import load_dataset
    
    # Load your dataset (example using Alpaca)
    dataset = load_dataset("tatsu-lab/alpaca", split="train")
    
    # Define prompt template
    prompt_template = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
    
    ### Instruction:
    {}
    
    ### Input:
    {}
    
    ### Response:
    {}"""
    
    # Format the dataset
    def format_dataset(examples):
        texts = []
        for instruction, input_text, output in zip(
            examples["instruction"], 
            examples["input"], 
            examples["output"]
        ):
            # Format according to the template and add EOS token
            text = prompt_template.format(instruction, input_text, output) + tokenizer.eos_token
            texts.append(text)
        
        return {"text": texts}
    
    # Apply formatting and create train/test split
    formatted_dataset = dataset.map(format_dataset, batched=True)
    dataset_split = formatted_dataset.train_test_split(test_size=0.05)
    train_dataset = dataset_split["train"]
    eval_dataset = dataset_split["test"]
    

    3. Configure PEFT with LoRA

    from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
    
    # Prepare model for k-bit training if using quantization
    model = prepare_model_for_kbit_training(model)
    
    # Configure LoRA
    lora_config = LoraConfig(
        r=16,  # LoRA rank
        lora_alpha=32,
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj", 
                        "gate_proj", "up_proj", "down_proj"],
        lora_dropout=0.05,
        bias="none",
        task_type="CAUSAL_LM"
    )
    
    # Apply LoRA to model
    model = get_peft_model(model, lora_config)
    

    4. Set Up Training Arguments

    from transformers import TrainingArguments
    
    training_args = TrainingArguments(
        output_dir="./tinyllama-finetuned",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,  # Simulate larger batch size
        learning_rate=2e-4,
        num_train_epochs=1,
        weight_decay=0.1,
        warmup_ratio=0.1,
        logging_steps=1,
        evaluation_strategy="steps",
        eval_steps=100,
        save_strategy="steps",
        save_steps=200,
        fp16=True,  # Enable mixed precision
        optim="adamw_8bit",  # Use 8-bit Adam optimizer
        lr_scheduler_type="cosine",
        seed=42
    )
    

    5. Create SFT Trainer and Fine-Tune

    from trl import SFTTrainer
    
    trainer = SFTTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        dataset_text_field="text",
        max_seq_length=2048,
        packing=True  # Pack short sequences together
    )
    
    # Start training
    trainer.train()
    
    # Save the model
    trainer.save_model("./tinyllama-finetuned-final")
    

    Approach 2: Fine-Tuning with Unsloth (Faster)

    Unsloth provides optimized training for faster fine-tuning of small language models.

    1. Load Model with Unsloth

    from unsloth import FastLanguageModel
    import torch
    
    # Set parameters
    max_seq_length = 2048
    dtype = None  # Auto-detect (Float16 for T4)
    load_in_4bit = True
    
    # Load model and tokenizer
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name="unsloth/tinyllama-bnb-4bit",  # 4-bit quantized version
        max_seq_length=max_seq_length,
        dtype=dtype,
        load_in_4bit=load_in_4bit
    )
    

    2. Apply LoRA with Unsloth

    model = FastLanguageModel.get_peft_model(
        model,
        r=16,  # LoRA rank
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                       "gate_proj", "up_proj", "down_proj"],
        lora_alpha=16,
        lora_dropout=0,
        bias="none",
        use_gradient_checkpointing=True  # Save memory during training
    )
    

    3. Set Training Parameters and Train

    from trl import SFTTrainer
    from transformers import TrainingArguments
    from unsloth import is_bfloat16_supported
    
    # Training arguments
    training_args = TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_ratio=0.1,
        num_train_epochs=1,
        learning_rate=2e-4,
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=1,
        optim="adamw_8bit",
        weight_decay=0.1,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="outputs"
    )
    
    # Create trainer
    trainer = SFTTrainer(
        model=model,
        tokenizer=tokenizer,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        dataset_text_field="text",
        max_seq_length=max_seq_length,
        dataset_num_proc=2,
        packing=True,
        args=training_args
    )
    
    # Fine-tune
    trainer.train()
    

    4. Inference with the Fine-Tuned Model

    # Enable faster inference
    FastLanguageModel.for_inference(model)
    
    # Example prompt
    prompt = "Describe the color of the sky on a clear day"
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    
    # Generate response
    outputs = model.generate(**inputs, max_new_tokens=64, use_cache=True)
    response = tokenizer.decode(outputs)
    print(response)
    

    Example: Fine-Tuning on Color Descriptions

    Here's a practical example of fine-tuning TinyLlama to generate hex codes from color descriptions:

    # Load color dataset
    from datasets import load_dataset
    dataset = load_dataset("burkelibbey/colors")
    
    # Format dataset for training
    def format_prompts(examples):
        texts = []
        for description, color in zip(examples["description"], examples["color"]):
            # Format as user input and assistant response
            text = f"&lt;|user|&gt;\n{description}\n&lt;|assistant|&gt;\n{color}&lt;|endoftext|&gt;"
            texts.append(text)
        return {"text": texts}
    
    formatted_dataset = dataset.map(format_prompts, batched=True)
    

    Best Practices

    1. Always use LoRA - Full fine-tuning is impractical even for TinyLlama due to memory constraints[11].

    2. Quantization is your friend - Use 4-bit quantization with bitsandbytes to reduce memory usage[5][11].

    3. Gradient checkpointing - Enable this to save memory during training at the cost of slightly slower training[11].

    4. Target the right modules - Focus on attention layers (q_proj, k_proj, v_proj, o_proj) and feed-forward layers (gate_proj, up_proj, down_proj)[4].

    5. Start with small epochs - Begin with 1 epoch to see if your setup works, then increase for better results[6].

    6. Batch size management - Use smaller batch sizes with gradient accumulation steps to simulate larger batches[5].

    7. Monitor training losses - Watch for decreasing loss during training to ensure your model is learning effectively[1].

    By following these approaches, you can successfully fine-tune the TinyLlama model on your custom dataset, creating a specialized version that performs well on specific tasks while maintaining its efficiency for deployment on resource-constrained environments.

  11. User

    How can I finetune the tinyllama model for working with a specific dataset for sentence splitting

  12. Perplexity Assistant

    Fine-tuning TinyLlama for Sentence Splitting: A Comprehensive Guide

    Fine-tuning TinyLlama for sentence splitting requires careful dataset preparation, efficient training techniques, and proper evaluation. Despite its small size (1.1B parameters), TinyLlama can be effectively adapted to specialized text processing tasks like sentence splitting when properly configured.

    Dataset Preparation for Sentence Splitting

    Creating an Effective Training Dataset

    To fine-tune TinyLlama for sentence splitting, you'll need a high-quality dataset with clear examples of properly split sentences:

    import pandas as pd
    from datasets import Dataset
    
    # Example dataset creation for sentence splitting
    def create_sentence_splitting_dataset(texts_with_splits):
        dataset_entries = []
        
        for text in texts_with_splits:
            # Prepare examples with full text and properly split sentences
            original_text = ' '.join(text)  # Join to create unsplit text
            split_sentences = text  # List of properly split sentences
            
            dataset_entries.append({
                "input": original_text,
                "output": '\n'.join(split_sentences)  # Split sentences joined with newlines
            })
        
        df = pd.DataFrame(dataset_entries)
        return Dataset.from_pandas(df)
    

    For semantic-aware text splitting, which generally produces better results than simple rule-based splitting, you can leverage techniques like those recommended in search results:

    # Semantic-aware text chunking (referenced approach)
    def semantic_aware_splitting(text):
        # For training data preparation, use high-quality pre-split sentences
        # Either manually curated or processed with a strong NLP tool
        
        # Alternatively, you could use LlamaIndex's semantic chunking
        # from llama_index.node_parsers import SemanticSplitterNodeParser
        # parser = SemanticSplitterNodeParser()
        # chunks = parser.get_nodes_from_documents([text])
        
        # For simplicity, we'll use a basic approach here
        import re
        sentences = re.split(r'(?&lt;=[.!?])\s+', text)
        return sentences
    

    Formatting for TinyLlama

    TinyLlama responds best to very clean, structured formats. For sentence splitting tasks, a clear instruction format works well:

    def format_for_tinyllama(examples):
        formatted_examples = []
        
        for input_text, output_text in zip(examples["input"], examples["output"]):
            # Using the format that works well for TinyLlama
            formatted = f"### Instruction: Split the following text into proper sentences.\n\n{input_text}\n\n### Response: {output_text}"
            formatted_examples.append({"text": formatted})
        
        return {"text": formatted_examples}
    

    This clean format is important since TinyLlama "doesn't have the extra capacity to 'figure things out' from loose or unstructured text" as noted in one of the search results from Data Scientist's Diary.

    Model Configuration for Fine-tuning

    Setting Up the Environment

    # Install required packages
    !pip install transformers datasets accelerate bitsandbytes peft trl
    
    # For faster training with Unsloth (optional but recommended)
    !pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
    

    Loading the Model with Quantization

    Memory efficiency is crucial, especially for small models like TinyLlama:

    from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
    import torch
    
    # Configure quantization
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.float16
    )
    
    # Load model with quantization
    model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"  # Latest chat version
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    tokenizer.pad_token = tokenizer.eos_token
    
    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        quantization_config=bnb_config,
        device_map="auto"
    )
    

    Configuring LoRA - Essential for TinyLlama Fine-tuning

    As noted in the search results, "If you're fine-tuning TinyLlama without PEFT, you're basically fighting a dragon with a teaspoon." PEFT with LoRA is essential:

    from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
    
    # Prepare model for k-bit training
    model = prepare_model_for_kbit_training(model)
    
    # Configure LoRA - targeting key attention and feed-forward layers
    lora_config = LoraConfig(
        r=16,  # LoRA rank
        lora_alpha=32,
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj", 
                        "gate_proj", "up_proj", "down_proj"],
        lora_dropout=0.05,
        bias="none",
        task_type="CAUSAL_LM"
    )
    
    # Apply LoRA to model
    model = get_peft_model(model, lora_config)
    

    Training Configuration

    Tokenizing the Dataset

    def tokenize_dataset(dataset, tokenizer, max_length=512):
        def tokenize_function(examples):
            return tokenizer(examples["text"], truncation=True, max_length=max_length)
        
        tokenized_dataset = dataset.map(tokenize_function, batched=True)
        return tokenized_dataset
    
    # Apply tokenization
    tokenized_dataset = tokenize_dataset(formatted_dataset, tokenizer)
    
    # Create train/validation split
    dataset_split = tokenized_dataset.train_test_split(test_size=0.1)
    train_dataset = dataset_split["train"]
    eval_dataset = dataset_split["test"]
    

    Training Arguments

    from transformers import TrainingArguments
    
    training_args = TrainingArguments(
        output_dir="./tinyllama-sentence-splitter",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,  # Simulate larger batch size
        learning_rate=2e-4,
        num_train_epochs=3,  # Start with fewer epochs to prevent overfitting
        weight_decay=0.1,
        warmup_ratio=0.1,
        logging_steps=10,
        evaluation_strategy="steps",
        eval_steps=100,
        save_strategy="steps",
        save_steps=200,
        save_total_limit=2,  # Save disk space
        fp16=True,  # Enable mixed precision
        gradient_checkpointing=True,  # Critical for memory efficiency
        optim="adamw_8bit",
        lr_scheduler_type="cosine",
        seed=42
    )
    

    Creating the Trainer and Starting Training

    from trl import SFTTrainer
    
    trainer = SFTTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        dataset_text_field="text",
        max_seq_length=512,  # Adjust based on your text lengths
        packing=True  # Pack short sequences together for efficiency
    )
    
    # Start training
    trainer.train()
    
    # Save the fine-tuned model
    trainer.save_model("./tinyllama-sentence-splitter-final")
    

    Evaluation and Inference

    After training, evaluate your model on sentence splitting tasks:

    def evaluate_sentence_splitting(model, tokenizer, text):
        prompt = f"### Instruction: Split the following text into proper sentences.\n\n{text}\n\n### Response:"
        inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
        
        # Generate output
        with torch.no_grad():
            outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.1)
        
        response = tokenizer.decode(outputs, skip_special_tokens=True)
        
        # Extract only the response part
        if "### Response:" in response:
            response = response.split("### Response:")[1].strip()
        
        # Split by newlines to get individual sentences
        sentences = response.split('\n')
        return sentences
    
    # Example usage
    text = "This is a complex paragraph with multiple sentences It lacks proper punctuation The goal is to correctly split it"
    sentences = evaluate_sentence_splitting(model, tokenizer, text)
    print(sentences)
    

    Best Practices and Optimizations

    Memory Management

    For TinyLlama, memory management is critical even though it's smaller than models like Llama-2-7B:

    1. Always use 4-bit quantization for both training and inference
    2. Enable gradient checkpointing to save memory during training
    3. Use smaller batch sizes with gradient accumulation instead of large batches

    Dataset Quality Tips

    The quality of your sentence splitting dataset significantly affects performance:

    1. Include diverse text styles: Academic, conversational, literary, etc.
    2. Include challenging cases: Quotations, abbreviations, lists
    3. Balance sentence lengths: Include both short and long sentences
    4. Add noisy examples: Texts with missing punctuation that need proper splitting

    Model Performance Optimization

    1. Start with fewer epochs (2-3) and monitor validation loss to prevent overfitting
    2. Use data augmentation to improve generalization - slightly modify texts while preserving sentence structures
    3. Consider sentence-level evaluation metrics like F1 score on boundary detection

    Monitoring and Debugging

    Track your model's performance during training:

    # Custom evaluation function to monitor sentence splitting metrics
    def compute_metrics(eval_pred):
        # Calculate sentence boundary detection metrics
        # E.g., precision, recall, F1 score for boundary detection
        predictions, labels = eval_pred
        # ... custom metric calculation
        return {"f1_score": f1}
    

    Conclusion

    Fine-tuning TinyLlama for sentence splitting requires careful attention to dataset preparation, efficient training techniques, and proper evaluation. While TinyLlama is a smaller model, using parameter-efficient techniques like LoRA and 4-bit quantization makes fine-tuning feasible even with limited computational resources.

    By following this guide, you'll be able to create a specialized sentence-splitting model that can be deployed in production environments where larger models might be impractical due to resource constraints.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.