Write an R script that produces a dataframe of the most popular downloaded books on…
- User
Write an R script that produces a dataframe of the most popular downloaded books on Project Gutenberg with the gutenbergr package
Assistant
Creating a Dataframe of Popular Project Gutenberg Books with gutenbergr
The gutenbergr package provides a convenient way to access and download public domain works from Project Gutenberg. While it doesn't directly provide popularity metrics, we can create a script that generates a useful dataframe of notable books that are likely to be popular.
Here's an R script that produces a dataframe of books from Project Gutenberg that are likely to be popular or notable:
# Install and load required packages if (!require("gutenbergr")) { install.packages("gutenbergr") } if (!require("dplyr")) { install.packages("dplyr") } library(gutenbergr) library(dplyr) # Get latest metadata (updated as of September 14, 2024) gutenberg_metadata_date <- attr(gutenberg_metadata, "date_updated") cat("Using Project Gutenberg metadata updated on:", gutenberg_metadata_date, "\n") # Create a function to find popular books based on various criteria get_popular_books <- function(n = 50) { # Start with English language works that have text available popular_books <- gutenberg_works() %>% # Filter for works with recognized authors (not NULL) filter(!is.na(author)) %>% # Join with subjects to get more context left_join( gutenberg_subjects %>% filter(subject_type == "lcsh") %>% group_by(gutenberg_id) %>% summarize(subjects = paste(subject, collapse = " | ")), by = "gutenberg_id" ) # Extract classic literature by joining with authors dataset popular_classics <- popular_books %>% left_join(gutenberg_authors, by = c("gutenberg_author_id" = "gutenberg_author_id")) %>% # Find authors born before 1900 (likely classics) filter(!is.na(birthdate) & birthdate < 1900) %>% # Select fields for our final dataframe select(gutenberg_id, title, author, birthdate, deathdate, subjects) %>% # Get the top entries head(n) # Return our results return(popular_classics) } # Create a function to find the most downloaded books (based on available metadata) get_classic_fiction <- function(n = 25) { fiction_books <- gutenberg_works() %>% # Join with subjects inner_join( gutenberg_subjects %>% filter( subject_type == "lcsh", # Look for fiction categories grepl("Fiction", subject) | grepl("Novel", subject) | grepl("story", subject, ignore.case = TRUE) ), by = "gutenberg_id" ) %>% # Select unique books distinct(gutenberg_id, .keep_all = TRUE) %>% select(gutenberg_id, title, author, subject) %>% head(n) return(fiction_books) } # Create a function to find books by specific popular authors get_books_by_popular_authors <- function(n = 25) { # List of some popular authors popular_authors <- c( "Shakespeare, William", "Austen, Jane", "Dickens, Charles", "Doyle, Arthur Conan", "Twain, Mark", "Poe, Edgar Allan", "Wilde, Oscar" ) books_by_popular_authors <- gutenberg_works() %>% filter(author %in% popular_authors) %>% select(gutenberg_id, title, author) %>% # Sort by author then title arrange(author, title) %>% head(n) return(books_by_popular_authors) } # Generate our dataframes popular_classics <- get_popular_books(50) classic_fiction <- get_classic_fiction(25) popular_author_books <- get_books_by_popular_authors(25) # Print a summary cat("Found", nrow(popular_classics), "popular classic books\n") cat("Found", nrow(classic_fiction), "classic fiction works\n") cat("Found", nrow(popular_author_books), "books by popular authors\n") # Example of how to download one of these books if (interactive()) { # Get the ID of a popular book (e.g., Pride and Prejudice) pride_and_prejudice <- popular_author_books %>% filter(title == "Pride and Prejudice") if (nrow(pride_and_prejudice) > 0) { book_id <- pride_and_prejudice$gutenberg_id[1] # Download the book text book_text <- gutenberg_download(book_id) cat("Downloaded", nrow(book_text), "lines of text from book ID:", book_id, "\n") } } # Return our main dataframe popular_classicsHow the Script Works
-
Installing and Loading Packages: The script starts by ensuring the necessary packages are installed and loaded.
-
Checking Metadata Date: The script displays when the Project Gutenberg metadata was last updated, which as of writing is September 14, 2024[1][5].
-
Finding Popular Classic Books: The script creates a function that:
- Starts with English language works from the gutenberg_works() function
- Joins with subject data for more context
- Filters for authors from before 1900 to focus on classics
- Creates a comprehensive dataframe with book IDs, titles, authors, and subjects[2][8]
-
Finding Classic Fiction: A second function focuses specifically on fiction works by looking for fiction-related subjects.
-
Books by Popular Authors: The third function creates a dataframe of works by specific authors known to be popular, such as Shakespeare, Austen, and Dickens[10].
-
Downloading Example: The script includes an example of how to download the text of one of the identified popular books using gutenberg_download()[5].
Example Output
The resulting dataframe (popular_classics) includes columns for:
- gutenberg_id: The unique identifier for downloading the book
- title: The book's title
- author: The author's name
- birthdate: When the author was born
- deathdate: When the author died
- subjects: Library of Congress subject classifications
This script provides a useful starting point for identifying potentially popular or notable books in the Project Gutenberg collection. While it doesn't directly measure download counts (which would require scraping the Project Gutenberg website), it uses available metadata to identify books that are likely to be popular based on their classic status and authors.
-
- User
How could I use the pageviews package to identify trends in popular books people are looking into?
Assistant
Using the pageviews Package to Identify Book Popularity Trends
The pageviews R package provides access to Wikimedia's traffic data, which you can leverage to identify trends in book popularity by analyzing Wikipedia page traffic patterns. This approach allows you to see which books are receiving attention over time, potentially indicating rising or declining interest in specific titles, authors, or genres.
Setting Up Your Environment
First, install and load the necessary packages:
# Install packages if needed if (!require("pageviews")) { install.packages("pageviews") } if (!require("dplyr")) { install.packages("dplyr") } if (!require("ggplot2")) { install.packages("ggplot2") } # Load packages library(pageviews) library(dplyr) library(ggplot2)Tracking Individual Book Trends
You can analyze how interest in specific books changes over time by examining Wikipedia pageview data for those titles:
# Function to get pageviews for a list of books get_book_pageviews <- function(book_titles, start_date, end_date) { results <- data.frame() for (title in book_titles) { # Get pageviews for this book book_data <- article_pageviews( article = title, project = "en.wikipedia", start = pageview_timestamps(start_date), end = pageview_timestamps(end_date), granularity = "daily" ) # Add to our results results <- rbind(results, book_data) } return(results) } # Example: Track popular books popular_books <- c( "To_Kill_a_Mockingbird", "1984_(novel)", "The_Great_Gatsby", "Pride_and_Prejudice", "Harry_Potter_and_the_Philosopher's_Stone" ) # Get data for the past 3 months three_months_ago <- Sys.Date() - 90 book_trends <- get_book_pageviews(popular_books, three_months_ago, Sys.Date())Identifying Trending Books
You can discover which books are currently trending by examining the top articles in the literature category:
# Get top articles, then filter for books literature_trending <- top_articles( project = "en.wikipedia", platform = "all", start = Sys.Date() - 7 # Last week ) # Filter for book-related articles book_keywords <- c("novel", "book", "fiction", "author", "literature") trending_books <- literature_trending %>% filter(grepl(paste(book_keywords, collapse="|"), article, ignore.case=TRUE)) # Display top 10 trending book-related articles head(trending_books, 10)Comparing Book Categories or Genres
You can also compare different book categories to identify broader trends:
# Define lists of books by genre science_fiction <- c("Dune_(novel)", "Neuromancer", "The_Hitchhiker's_Guide_to_the_Galaxy") fantasy <- c("The_Lord_of_the_Rings", "A_Game_of_Thrones", "The_Name_of_the_Wind") classic_literature <- c("Moby-Dick", "War_and_Peace", "Jane_Eyre") # Get pageviews for each genre start_date <- as.Date("2025-01-01") # Beginning of the year sci_fi_views <- get_book_pageviews(science_fiction, start_date, Sys.Date()) fantasy_views <- get_book_pageviews(fantasy, start_date, Sys.Date()) classics_views <- get_book_pageviews(classic_literature, start_date, Sys.Date()) # Add genre column to each dataset sci_fi_views$genre <- "Science Fiction" fantasy_views$genre <- "Fantasy" classics_views$genre <- "Classic Literature" # Combine datasets genre_comparison <- rbind(sci_fi_views, fantasy_views, classics_views) # Summarize by genre and date genre_summary <- genre_comparison %>% group_by(date, genre) %>% summarize(total_views = sum(views))Visualizing Book Trends
Create visualizations to better understand the trends:
# Plot trends for individual books ggplot(book_trends, aes(x = date, y = views, color = article)) + geom_line() + theme_minimal() + labs(title = "Wikipedia Pageviews for Popular Books", x = "Date", y = "Number of Views", color = "Book") + theme(legend.position = "bottom") # Plot genre comparison ggplot(genre_summary, aes(x = date, y = total_views, color = genre)) + geom_line() + theme_minimal() + labs(title = "Comparison of Book Genre Popularity", x = "Date", y = "Total Pageviews", color = "Genre") + theme(legend.position = "bottom")Analyzing Seasonal Trends
Identify patterns in book popularity across different seasons or time periods:
# Function to aggregate monthly data get_seasonal_book_trends <- function(book_titles, years = 1) { start_date <- Sys.Date() - (365 * years) results <- data.frame() for (title in book_titles) { # Get data with monthly granularity book_data <- article_pageviews( article = title, project = "en.wikipedia", start = pageview_timestamps(start_date), end = pageview_timestamps(Sys.Date()), granularity = "monthly" ) results <- rbind(results, book_data) } # Extract month from date results$month <- format(results$date, "%m") return(results) } # Get seasonal data for popular books seasonal_trends <- get_seasonal_book_trends(popular_books) # Summarize by month monthly_summary <- seasonal_trends %>% group_by(month, article) %>% summarize(avg_views = mean(views))Advanced Analysis: Book Popularity Following Media Adaptations
Investigate how book pageviews change following movie or TV adaptations:
# Function to examine pageviews before and after an adaptation release analyze_adaptation_impact <- function(book_title, adaptation_date, window_days = 30) { # Convert to Date object adaptation_date <- as.Date(adaptation_date) # Get data for window before and after before_start <- adaptation_date - window_days after_end <- adaptation_date + window_days impact_data <- article_pageviews( article = book_title, project = "en.wikipedia", start = pageview_timestamps(before_start), end = pageview_timestamps(after_end), granularity = "daily" ) # Add a flag for before/after impact_data$period <- ifelse(impact_data$date < adaptation_date, "Before Adaptation", "After Adaptation") return(impact_data) } # Example: Analyze "Dune" before and after the 2024 movie release dune_impact <- analyze_adaptation_impact("Dune_(novel)", "2024-03-01") # Compare average views dune_summary <- dune_impact %>% group_by(period) %>% summarize(avg_daily_views = mean(views))Cross-Referencing with Bestseller Lists
While not directly available through pageviews, you could create a dataset that cross-references Wikipedia traffic with bestseller lists:
# Hypothetical bestseller list (you would need to obtain this data separately) bestsellers_2025 <- data.frame( title = c("New_Bestseller_1", "New_Bestseller_2", "New_Bestseller_3"), list_rank = c(1, 2, 3), stringsAsFactors = FALSE ) # Get pageviews for these bestsellers bestseller_views <- get_book_pageviews( bestsellers_2025$title, Sys.Date() - 30, Sys.Date() ) # Merge with bestseller ranks bestseller_analysis <- bestseller_views %>% left_join(bestsellers_2025, by = c("article" = "title"))Considerations and Limitations
-
Wikipedia as a proxy: Wikipedia pageviews are an indirect measure of interest rather than actual sales or readership.
-
Article naming conventions: You'll need to know the exact Wikipedia article title (with underscores instead of spaces).
-
External factors: Pageviews might spike due to media coverage, controversies, or adaptations rather than actual reading interest.
-
Historical data: For broader historical analysis, consider using
old_pageviews()for data from 2008-2016. -
API limitations: Be mindful of rate limits when making numerous requests.
By systematically analyzing Wikipedia pageview data, you can gain valuable insights into book popularity trends that can complement traditional publishing metrics and inform marketing strategies, inventory decisions, or research questions about literary interests.
-
- User
write me a function that downloads a project gutenberg text and then spilts the text by sentence
Assistant
Function to Download and Split Project Gutenberg Texts by Sentence
Here's a comprehensive R function that downloads texts from Project Gutenberg and splits them into sentences:
download_and_split_by_sentence <- function(gutenberg_id, method = "tokenizers", strip_headers = TRUE, meta_fields = c("title", "author")) { # Install required packages if not already installed required_packages <- c("gutenbergr", "stringr", "dplyr") # Add packages based on the method chosen if (method == "tokenizers") { required_packages <- c(required_packages, "tokenizers") } else if (method == "corpus") { required_packages <- c(required_packages, "corpus") } # Check and install missing packages new_packages <- required_packages[!required_packages %in% installed.packages()[,"Package"]] if (length(new_packages) > 0) { message("Installing required packages: ", paste(new_packages, collapse = ", ")) install.packages(new_packages) } # Load required packages library(gutenbergr) library(stringr) library(dplyr) # Download the text from Project Gutenberg message("Downloading text with ID: ", gutenberg_id) book_text <- gutenberg_download(gutenberg_id, strip = strip_headers, meta_fields = meta_fields) # Extract metadata metadata <- book_text %>% select(any_of(meta_fields)) %>% distinct() # Extract and collapse text full_text <- paste(book_text$text, collapse = " ") # Clean the text by handling multiple spaces and line breaks full_text <- str_replace_all(full_text, "\\s+", " ") message("Splitting text by sentences using method: ", method) # Split text into sentences based on chosen method sentences <- switch(method, "tokenizers" = { # Use tokenizers package library(tokenizers) unlist(tokenize_sentences(full_text)) }, "corpus" = { # Use corpus package library(corpus) unlist(text_split(full_text, units = "sentences")) }, "regex" = { # Use regex approach # This pattern looks for sentence boundaries: periods, question marks, or exclamation marks # followed by whitespace and an uppercase letter sentence_pattern <- "(?<=[.!?])\\s+(?=[A-Z])" str_split(full_text, sentence_pattern, simplify = FALSE)[[1]] }, "basic" = { # Simple approach replacing ending punctuation with a code word then splitting coded_text <- str_replace_all(full_text, "[.!?;]", "SENTENCEBREAK") sentences <- unlist(str_split(coded_text, "SENTENCEBREAK")) # Remove empty sentences and trim whitespace sentences <- sentences[sentences != ""] trimws(sentences) }, { # Default method if none of the above match warning("Unknown method '", method, "'. Using 'regex' method instead.") sentence_pattern <- "(?<=[.!?])\\s+(?=[A-Z])" str_split(full_text, sentence_pattern, simplify = FALSE)[[1]] } ) # Clean sentences by trimming whitespace sentences <- trimws(sentences) # Remove empty sentences sentences <- sentences[sentences != ""] # Create a result list containing both metadata and sentences result <- list( metadata = metadata, sentences = sentences, sentence_count = length(sentences), method = method ) message("Successfully split text into ", length(sentences), " sentences") return(result) }Usage Examples
Here are several ways to use this function:
# Example 1: Download and split "Pride and Prejudice" using the default tokenizers method pride_and_prejudice <- download_and_split_by_sentence(1342) # Example 2: Download "The Adventures of Sherlock Holmes" using the corpus package sherlock <- download_and_split_by_sentence(1661, method = "corpus") # Example 3: Download "Moby Dick" using regex approach moby_dick <- download_and_split_by_sentence(2701, method = "regex") # Example 4: Download "A Tale of Two Cities" using the basic approach tale_of_two_cities <- download_and_split_by_sentence(98, method = "basic") # Display information about the text cat("Title:", pride_and_prejudice$metadata$title, "\n") cat("Author:", pride_and_prejudice$metadata$author, "\n") cat("Number of sentences:", pride_and_prejudice$sentence_count, "\n") # Display the first 5 sentences cat("First 5 sentences:\n") for (i in 1:5) { cat(i, ": ", pride_and_prejudice$sentences[i], "\n", sep="") }How It Works
The function operates in several stages:
-
Package Management: The function checks for and installs any required packages based on the chosen sentence splitting method.
-
Text Download: It uses
gutenberg_download()from the gutenbergr package to download the specified text by its Gutenberg ID. The text comes with metadata like title and author. -
Text Preparation: The downloaded text is collapsed into a single string and cleaned to handle spacing issues.
-
Sentence Splitting: The function offers four different methods for splitting text into sentences:
-
tokenizers: Uses the
tokenize_sentences()function from the tokenizers package, which is designed specifically for natural language processing. -
corpus: Uses the
text_split()function from the corpus package, which provides more advanced text segmentation capabilities. -
regex: Uses a regular expression approach to identify sentence boundaries (periods, question marks, or exclamation marks followed by a space and an uppercase letter).
-
basic: A simpler approach that replaces ending punctuation with a code word and then splits on that code.
-
-
Post-processing: The function cleans the sentences by trimming whitespace and removing empty entries.
-
Result Structure: Returns a list containing:
- The metadata from the original download
- The split sentences
- The total sentence count
- The method used for splitting
Considerations for Sentence Splitting
Each sentence splitting method has advantages and limitations:
-
tokenizers: Generally works well for most texts and handles many edge cases, but requires the additional tokenizers package.
-
corpus: Provides advanced functionality for text segmentation but requires the corpus package.
-
regex: Works without additional packages but may struggle with abbreviations (like "Mr.", "Dr.", etc.) or quotations that cross sentence boundaries.
-
basic: The simplest approach, but least accurate as it doesn't properly handle abbreviations or complex punctuation patterns.
The most accurate results typically come from the specialized NLP packages (tokenizers or corpus), while the regex and basic approaches are more prone to errors with complex texts.
If working with literary texts that have complex sentence structures, quotations, or dialogue, the tokenizers method is generally recommended for its balance of accuracy and speed.
-
- User
Make another function that gets a chapter dataframe from the book dataframe
Assistant
Function to Extract Chapters from Project Gutenberg Texts
Here's a function that builds on the previous one to identify and extract chapters from a Project Gutenberg text:
extract_chapters <- function(book_data, chapter_pattern = NULL, include_chapter_titles = TRUE, handle_preamble = TRUE) { # Load required packages required_packages <- c("dplyr", "stringr", "tidyr") new_packages <- required_packages[!required_packages %in% installed.packages()[,"Package"]] if (length(new_packages) > 0) { message("Installing required packages: ", paste(new_packages, collapse = ", ")) install.packages(new_packages) } library(dplyr) library(stringr) library(tidyr) # Check if input is from our download_and_split_by_sentence function if (is.list(book_data) && "sentences" %in% names(book_data)) { sentences <- book_data$sentences metadata <- book_data$metadata } else if ("text" %in% names(book_data)) { # Input is a raw gutenbergr download sentences <- paste(book_data$text, collapse = " ") sentences <- unlist(str_split(sentences, "(?<=\\.)\\s+(?=[A-Z])")) sentences <- trimws(sentences) sentences <- sentences[sentences != ""] # Extract metadata metadata_cols <- setdiff(names(book_data), "text") metadata <- book_data %>% select(all_of(metadata_cols)) %>% distinct() } else { stop("Input must be either output from download_and_split_by_sentence function or a gutenbergr download dataframe") } # Determine default chapter pattern based on text content if not provided if (is.null(chapter_pattern)) { # Test a few common patterns against the first 100 sentences to auto-detect format sample_text <- paste(sentences[1:min(100, length(sentences))], collapse=" ") patterns <- list( standard = "(?i)\\b(chapter|chap\\.?)\\s+([0-9]+|[IVXLCDM]+)\\b", roman = "(?i)\\b(chapter|chap\\.?)\\s+([IVXLCDM]+)\\b", spelled = "(?i)\\b(chapter|chap\\.?)\\s+(one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve|thirteen|fourteen|fifteen|sixteen|seventeen|eighteen|nineteen|twenty)\\b", numbered = "(?i)\\b([0-9]+|[IVXLCDM]+)\\b" ) # Check which pattern has the most matches matches <- sapply(patterns, function(p) length(str_extract_all(sample_text, p)[[1]])) if (max(matches) > 0) { pattern_name <- names(which.max(matches)) chapter_pattern <- patterns[[pattern_name]] message("Auto-detected chapter pattern: ", pattern_name) } else { # Default to standard pattern if no matches found chapter_pattern <- patterns$standard message("No chapter pattern detected. Using standard pattern.") } } # Rebuild the full text with sentence markers for easier chapter detection full_text <- paste(sentences, collapse = " <sentence_break> ") # Find all chapter markers chapter_markers <- str_extract_all(full_text, chapter_pattern)[[1]] if (length(chapter_markers) == 0) { warning("No chapters detected with the specified pattern. Returning single chapter.") chapters_df <- data.frame( chapter_number = 1, chapter_title = "Full Text", chapter_text = paste(sentences, collapse = " "), sentence_count = length(sentences), start_sentence = 1, end_sentence = length(sentences) ) result <- list( metadata = metadata, chapters = chapters_df, chapter_count = 1 ) return(result) } # Split text by chapter pattern chapter_splits <- str_split(full_text, chapter_pattern)[[1]] # Process chapter text chapters <- data.frame( chapter_marker = c("Preamble", chapter_markers), chapter_text = chapter_splits, stringsAsFactors = FALSE ) %>% mutate( # Extract chapter numbers chapter_number = case_when( chapter_marker == "Preamble" ~ 0, grepl("(?i)chapter|chap", chapter_marker) ~ as.numeric(factor(str_extract(chapter_marker, "(?<=(chapter|chap\\.?)\\s+)[0-9IVXLCDM]+"), levels = unique(str_extract(chapter_markers, "(?<=(chapter|chap\\.?)\\s+)[0-9IVXLCDM]+")))), TRUE ~ row_number() ), # Create chapter titles if requested chapter_title = case_when( chapter_marker == "Preamble" ~ "Preamble", include_chapter_titles & grepl(":", chapter_text) ~ trimws(str_extract(chapter_text, "^.*?(?=\\.)")), TRUE ~ paste("Chapter", chapter_number) ) ) # Convert sentence-break markers back to sentences and count them chapters <- chapters %>% mutate( sentences = str_split(chapter_text, "<sentence_break>"), sentence_count = sapply(sentences, length), # Clean up the sentences sentences = lapply(sentences, trimws), sentences = lapply(sentences, function(s) s[s != ""]) ) # Handle the preamble (text before the first chapter) if (!handle_preamble && nrow(chapters) > 1) { chapters <- chapters[-1, ] } # Calculate sentence indices chapters <- chapters %>% mutate( start_sentence = cumsum(c(1, head(sentence_count, -1))), end_sentence = cumsum(sentence_count) ) # Final cleanup and structure chapters_df <- chapters %>% select(chapter_number, chapter_title, chapter_text, sentence_count, start_sentence, end_sentence) # Add sentences as a separate list column if needed chapters_df$sentences <- chapters$sentences # Create the result structure result <- list( metadata = metadata, chapters = chapters_df, chapter_count = nrow(chapters_df) ) message("Successfully extracted ", result$chapter_count, " chapters") return(result) }Usage Examples
Here's how to use this function with the previous function:
# First download and split a book by sentences book_data <- download_and_split_by_sentence(1342) # Pride and Prejudice # Then extract chapters pride_chapters <- extract_chapters(book_data) # Alternative: provide your own chapter pattern moby_dick <- download_and_split_by_sentence(2701) # Moby Dick moby_chapters <- extract_chapters(moby_dick, chapter_pattern = "(?i)chapter\\s+[0-9IVXLCDM]+") # Work with books that have named chapters david_copperfield <- download_and_split_by_sentence(766) # David Copperfield copperfield_chapters <- extract_chapters(david_copperfield, include_chapter_titles = TRUE) # Display chapter information cat("Book:", pride_chapters$metadata$title, "by", pride_chapters$metadata$author, "\n") cat("Number of chapters:", pride_chapters$chapter_count, "\n\n") # Show chapter information print(select(pride_chapters$chapters, chapter_number, chapter_title, sentence_count)) # Get text from a specific chapter chapter_3 <- pride_chapters$chapters$chapter_text[pride_chapters$chapters$chapter_number == 3] cat("First 200 characters of Chapter 3:\n", substr(chapter_3, 1, 200), "...\n") # Analysis: Find the longest and shortest chapters chapter_lengths <- pride_chapters$chapters %>% mutate(word_count = stringr::str_count(chapter_text, "\\w+")) %>% select(chapter_number, chapter_title, word_count, sentence_count) longest_chapter <- chapter_lengths %>% filter(word_count == max(word_count)) shortest_chapter <- chapter_lengths %>% filter(word_count == min(word_count)) cat("Longest chapter:", longest_chapter$chapter_title, "with", longest_chapter$word_count, "words and", longest_chapter$sentence_count, "sentences\n") cat("Shortest chapter:", shortest_chapter$chapter_title, "with", shortest_chapter$word_count, "words and", shortest_chapter$sentence_count, "sentences\n")How the Chapter Extraction Works
Smart Pattern Detection
The function uses several approaches to identify chapters:
-
Auto-detection: If no pattern is provided, it samples the first portion of the text and tests different common chapter heading formats.
-
Pattern Options:
- Standard format: "Chapter 1", "Chapter 2", etc.
- Roman numeral format: "Chapter I", "Chapter II", etc.
- Spelled out format: "Chapter One", "Chapter Two", etc.
- Simple numbered format: Just numbers or roman numerals
-
Chapter Title Handling: The function can extract chapter titles when they follow patterns like "Chapter 1: The Beginning".
Chapter Splitting Process
The function:
- Rebuilds the text with sentence markers to maintain sentence boundaries during chapter splitting
- Identifies chapter markers throughout the text
- Splits the text at these markers to create chapter segments
- Processes each segment to extract chapter numbers, titles, and content
- Rebuilds the sentence structure within each chapter
- Calculates statistics like sentence count and position indices
Special Case Handling
-
Preamble Text: Text before the first chapter marker (preface, introduction, etc.) is handled as a "Chapter 0" or can be excluded.
-
Single Chapter Books: If no chapters are detected, the function returns the entire text as a single chapter.
-
Chapter Titles: The function attempts to extract meaningful chapter titles when available, or defaults to "Chapter X" otherwise.
Benefits
This function provides several benefits for analyzing literary texts:
-
Structure Analysis: Break down books into their structural components for analysis
-
Chapter-Level Metrics: Generate statistics about chapter length, complexity, and patterns
-
Selective Processing: Work with specific chapters instead of entire texts
-
Content Extraction: Easily pull out chapter titles and other structured elements
-
Compatibility: Works seamlessly with the previous sentence-splitting function
When combined with the previous sentence-splitting function, this creates a powerful toolkit for processing and analyzing Project Gutenberg texts at various structural levels.</sentence_break></sentence_break>
-
- User
How do I finetune a tinyllama model
Assistant
Fine-Tuning TinyLlama: A Comprehensive Guide
TinyLlama is a compact but powerful language model trained on approximately 3 trillion tokens, making it suitable for edge devices and applications where computational resources are limited. Despite its small size (1.1B parameters), it can be effectively fine-tuned for specific tasks. Here's how to fine-tune TinyLlama on your custom dataset.
Setup and Prerequisites
Hardware Requirements
- A GPU with at least 8GB VRAM (T4 on Google Colab works well)
- 16GB+ system RAM
Software Dependencies
# Install required packages pip install transformers datasets accelerate bitsandbytes trl peftFor faster fine-tuning with Unsloth:
pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git" pip install --no-deps "xformers<0.0.27" "trl<0.9.0" peft accelerate bitsandbytesApproach 1: Fine-Tuning with Hugging Face Transformers
1. Load Model and Tokenizer
from transformers import AutoModelForCausalLM, AutoTokenizer import torch model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" # Base chat model # Load tokenizer tokenizer = AutoTokenizer.from_pretrained(model_id) tokenizer.pad_token = tokenizer.eos_token # Set padding token to EOS token # Load model with quantization for memory efficiency from transformers import BitsAndBytesConfig bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16 ) model = AutoModelForCausalLM.from_pretrained( model_id, quantization_config=bnb_config, device_map="auto" )2. Prepare the Dataset
from datasets import load_dataset # Load your dataset (example using Alpaca) dataset = load_dataset("tatsu-lab/alpaca", split="train") # Define prompt template prompt_template = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: {} ### Input: {} ### Response: {}""" # Format the dataset def format_dataset(examples): texts = [] for instruction, input_text, output in zip( examples["instruction"], examples["input"], examples["output"] ): # Format according to the template and add EOS token text = prompt_template.format(instruction, input_text, output) + tokenizer.eos_token texts.append(text) return {"text": texts} # Apply formatting and create train/test split formatted_dataset = dataset.map(format_dataset, batched=True) dataset_split = formatted_dataset.train_test_split(test_size=0.05) train_dataset = dataset_split["train"] eval_dataset = dataset_split["test"]3. Configure PEFT with LoRA
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training # Prepare model for k-bit training if using quantization model = prepare_model_for_kbit_training(model) # Configure LoRA lora_config = LoraConfig( r=16, # LoRA rank lora_alpha=32, target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], lora_dropout=0.05, bias="none", task_type="CAUSAL_LM" ) # Apply LoRA to model model = get_peft_model(model, lora_config)4. Set Up Training Arguments
from transformers import TrainingArguments training_args = TrainingArguments( output_dir="./tinyllama-finetuned", per_device_train_batch_size=4, gradient_accumulation_steps=4, # Simulate larger batch size learning_rate=2e-4, num_train_epochs=1, weight_decay=0.1, warmup_ratio=0.1, logging_steps=1, evaluation_strategy="steps", eval_steps=100, save_strategy="steps", save_steps=200, fp16=True, # Enable mixed precision optim="adamw_8bit", # Use 8-bit Adam optimizer lr_scheduler_type="cosine", seed=42 )5. Create SFT Trainer and Fine-Tune
from trl import SFTTrainer trainer = SFTTrainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset, dataset_text_field="text", max_seq_length=2048, packing=True # Pack short sequences together ) # Start training trainer.train() # Save the model trainer.save_model("./tinyllama-finetuned-final")Approach 2: Fine-Tuning with Unsloth (Faster)
Unsloth provides optimized training for faster fine-tuning of small language models.
1. Load Model with Unsloth
from unsloth import FastLanguageModel import torch # Set parameters max_seq_length = 2048 dtype = None # Auto-detect (Float16 for T4) load_in_4bit = True # Load model and tokenizer model, tokenizer = FastLanguageModel.from_pretrained( model_name="unsloth/tinyllama-bnb-4bit", # 4-bit quantized version max_seq_length=max_seq_length, dtype=dtype, load_in_4bit=load_in_4bit )2. Apply LoRA with Unsloth
model = FastLanguageModel.get_peft_model( model, r=16, # LoRA rank target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], lora_alpha=16, lora_dropout=0, bias="none", use_gradient_checkpointing=True # Save memory during training )3. Set Training Parameters and Train
from trl import SFTTrainer from transformers import TrainingArguments from unsloth import is_bfloat16_supported # Training arguments training_args = TrainingArguments( per_device_train_batch_size=2, gradient_accumulation_steps=4, warmup_ratio=0.1, num_train_epochs=1, learning_rate=2e-4, fp16=not is_bfloat16_supported(), bf16=is_bfloat16_supported(), logging_steps=1, optim="adamw_8bit", weight_decay=0.1, lr_scheduler_type="linear", seed=3407, output_dir="outputs" ) # Create trainer trainer = SFTTrainer( model=model, tokenizer=tokenizer, train_dataset=train_dataset, eval_dataset=eval_dataset, dataset_text_field="text", max_seq_length=max_seq_length, dataset_num_proc=2, packing=True, args=training_args ) # Fine-tune trainer.train()4. Inference with the Fine-Tuned Model
# Enable faster inference FastLanguageModel.for_inference(model) # Example prompt prompt = "Describe the color of the sky on a clear day" inputs = tokenizer(prompt, return_tensors="pt").to("cuda") # Generate response outputs = model.generate(**inputs, max_new_tokens=64, use_cache=True) response = tokenizer.decode(outputs) print(response)Example: Fine-Tuning on Color Descriptions
Here's a practical example of fine-tuning TinyLlama to generate hex codes from color descriptions:
# Load color dataset from datasets import load_dataset dataset = load_dataset("burkelibbey/colors") # Format dataset for training def format_prompts(examples): texts = [] for description, color in zip(examples["description"], examples["color"]): # Format as user input and assistant response text = f"<|user|>\n{description}\n<|assistant|>\n{color}<|endoftext|>" texts.append(text) return {"text": texts} formatted_dataset = dataset.map(format_prompts, batched=True)Best Practices
-
Always use LoRA - Full fine-tuning is impractical even for TinyLlama due to memory constraints[11].
-
Quantization is your friend - Use 4-bit quantization with bitsandbytes to reduce memory usage[5][11].
-
Gradient checkpointing - Enable this to save memory during training at the cost of slightly slower training[11].
-
Target the right modules - Focus on attention layers (q_proj, k_proj, v_proj, o_proj) and feed-forward layers (gate_proj, up_proj, down_proj)[4].
-
Start with small epochs - Begin with 1 epoch to see if your setup works, then increase for better results[6].
-
Batch size management - Use smaller batch sizes with gradient accumulation steps to simulate larger batches[5].
-
Monitor training losses - Watch for decreasing loss during training to ensure your model is learning effectively[1].
By following these approaches, you can successfully fine-tune the TinyLlama model on your custom dataset, creating a specialized version that performs well on specific tasks while maintaining its efficiency for deployment on resource-constrained environments.
- User
How can I finetune the tinyllama model for working with a specific dataset for sentence splitting
Assistant
Fine-tuning TinyLlama for Sentence Splitting: A Comprehensive Guide
Fine-tuning TinyLlama for sentence splitting requires careful dataset preparation, efficient training techniques, and proper evaluation. Despite its small size (1.1B parameters), TinyLlama can be effectively adapted to specialized text processing tasks like sentence splitting when properly configured.
Dataset Preparation for Sentence Splitting
Creating an Effective Training Dataset
To fine-tune TinyLlama for sentence splitting, you'll need a high-quality dataset with clear examples of properly split sentences:
import pandas as pd from datasets import Dataset # Example dataset creation for sentence splitting def create_sentence_splitting_dataset(texts_with_splits): dataset_entries = [] for text in texts_with_splits: # Prepare examples with full text and properly split sentences original_text = ' '.join(text) # Join to create unsplit text split_sentences = text # List of properly split sentences dataset_entries.append({ "input": original_text, "output": '\n'.join(split_sentences) # Split sentences joined with newlines }) df = pd.DataFrame(dataset_entries) return Dataset.from_pandas(df)For semantic-aware text splitting, which generally produces better results than simple rule-based splitting, you can leverage techniques like those recommended in search results:
# Semantic-aware text chunking (referenced approach) def semantic_aware_splitting(text): # For training data preparation, use high-quality pre-split sentences # Either manually curated or processed with a strong NLP tool # Alternatively, you could use LlamaIndex's semantic chunking # from llama_index.node_parsers import SemanticSplitterNodeParser # parser = SemanticSplitterNodeParser() # chunks = parser.get_nodes_from_documents([text]) # For simplicity, we'll use a basic approach here import re sentences = re.split(r'(?<=[.!?])\s+', text) return sentencesFormatting for TinyLlama
TinyLlama responds best to very clean, structured formats. For sentence splitting tasks, a clear instruction format works well:
def format_for_tinyllama(examples): formatted_examples = [] for input_text, output_text in zip(examples["input"], examples["output"]): # Using the format that works well for TinyLlama formatted = f"### Instruction: Split the following text into proper sentences.\n\n{input_text}\n\n### Response: {output_text}" formatted_examples.append({"text": formatted}) return {"text": formatted_examples}This clean format is important since TinyLlama "doesn't have the extra capacity to 'figure things out' from loose or unstructured text" as noted in one of the search results from Data Scientist's Diary.
Model Configuration for Fine-tuning
Setting Up the Environment
# Install required packages !pip install transformers datasets accelerate bitsandbytes peft trl # For faster training with Unsloth (optional but recommended) !pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"Loading the Model with Quantization
Memory efficiency is crucial, especially for small models like TinyLlama:
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig import torch # Configure quantization bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16 ) # Load model with quantization model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" # Latest chat version tokenizer = AutoTokenizer.from_pretrained(model_id) tokenizer.pad_token = tokenizer.eos_token model = AutoModelForCausalLM.from_pretrained( model_id, quantization_config=bnb_config, device_map="auto" )Configuring LoRA - Essential for TinyLlama Fine-tuning
As noted in the search results, "If you're fine-tuning TinyLlama without PEFT, you're basically fighting a dragon with a teaspoon." PEFT with LoRA is essential:
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training # Prepare model for k-bit training model = prepare_model_for_kbit_training(model) # Configure LoRA - targeting key attention and feed-forward layers lora_config = LoraConfig( r=16, # LoRA rank lora_alpha=32, target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], lora_dropout=0.05, bias="none", task_type="CAUSAL_LM" ) # Apply LoRA to model model = get_peft_model(model, lora_config)Training Configuration
Tokenizing the Dataset
def tokenize_dataset(dataset, tokenizer, max_length=512): def tokenize_function(examples): return tokenizer(examples["text"], truncation=True, max_length=max_length) tokenized_dataset = dataset.map(tokenize_function, batched=True) return tokenized_dataset # Apply tokenization tokenized_dataset = tokenize_dataset(formatted_dataset, tokenizer) # Create train/validation split dataset_split = tokenized_dataset.train_test_split(test_size=0.1) train_dataset = dataset_split["train"] eval_dataset = dataset_split["test"]Training Arguments
from transformers import TrainingArguments training_args = TrainingArguments( output_dir="./tinyllama-sentence-splitter", per_device_train_batch_size=4, gradient_accumulation_steps=4, # Simulate larger batch size learning_rate=2e-4, num_train_epochs=3, # Start with fewer epochs to prevent overfitting weight_decay=0.1, warmup_ratio=0.1, logging_steps=10, evaluation_strategy="steps", eval_steps=100, save_strategy="steps", save_steps=200, save_total_limit=2, # Save disk space fp16=True, # Enable mixed precision gradient_checkpointing=True, # Critical for memory efficiency optim="adamw_8bit", lr_scheduler_type="cosine", seed=42 )Creating the Trainer and Starting Training
from trl import SFTTrainer trainer = SFTTrainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset, dataset_text_field="text", max_seq_length=512, # Adjust based on your text lengths packing=True # Pack short sequences together for efficiency ) # Start training trainer.train() # Save the fine-tuned model trainer.save_model("./tinyllama-sentence-splitter-final")Evaluation and Inference
After training, evaluate your model on sentence splitting tasks:
def evaluate_sentence_splitting(model, tokenizer, text): prompt = f"### Instruction: Split the following text into proper sentences.\n\n{text}\n\n### Response:" inputs = tokenizer(prompt, return_tensors="pt").to("cuda") # Generate output with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.1) response = tokenizer.decode(outputs, skip_special_tokens=True) # Extract only the response part if "### Response:" in response: response = response.split("### Response:")[1].strip() # Split by newlines to get individual sentences sentences = response.split('\n') return sentences # Example usage text = "This is a complex paragraph with multiple sentences It lacks proper punctuation The goal is to correctly split it" sentences = evaluate_sentence_splitting(model, tokenizer, text) print(sentences)Best Practices and Optimizations
Memory Management
For TinyLlama, memory management is critical even though it's smaller than models like Llama-2-7B:
- Always use 4-bit quantization for both training and inference
- Enable gradient checkpointing to save memory during training
- Use smaller batch sizes with gradient accumulation instead of large batches
Dataset Quality Tips
The quality of your sentence splitting dataset significantly affects performance:
- Include diverse text styles: Academic, conversational, literary, etc.
- Include challenging cases: Quotations, abbreviations, lists
- Balance sentence lengths: Include both short and long sentences
- Add noisy examples: Texts with missing punctuation that need proper splitting
Model Performance Optimization
- Start with fewer epochs (2-3) and monitor validation loss to prevent overfitting
- Use data augmentation to improve generalization - slightly modify texts while preserving sentence structures
- Consider sentence-level evaluation metrics like F1 score on boundary detection
Monitoring and Debugging
Track your model's performance during training:
# Custom evaluation function to monitor sentence splitting metrics def compute_metrics(eval_pred): # Calculate sentence boundary detection metrics # E.g., precision, recall, F1 score for boundary detection predictions, labels = eval_pred # ... custom metric calculation return {"f1_score": f1}Conclusion
Fine-tuning TinyLlama for sentence splitting requires careful attention to dataset preparation, efficient training techniques, and proper evaluation. While TinyLlama is a smaller model, using parameter-efficient techniques like LoRA and 4-bit quantization makes fine-tuning feasible even with limited computational resources.
By following this guide, you'll be able to create a specialized sentence-splitting model that can be deployed in production environments where larger models might be impractical due to resource constraints.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.