library(rvest) create data frames for each section of data dataA <- data.frame(…
- User
library(rvest)
create data frames for each section of data
data_A <- data.frame( protocol_url_segment = character(), A.1 = character(), A.2 = character(), A.3 = character(), A.3.1 = character(), A.3.2 = character(), A.4.1 = character(), A.5.1 = character(), A.5.2 = character(), A.5.3 = character(), A.5.4 = character(), A.7 = character(), A.8 = character(), stringsAsFactors = FALSE )
[TRUNCATED] data_P <- data.frame( protocol_url_segment = character(), P.A = character(), P.B = character(), stringsAsFactors = FALSE )
specify the base URL of the protocol page
protocol_url_segment <- "2015-003589-10/GR" base_url <- "https://www.clinicaltrialsregister.eu/ctr-search/trial/" url <- paste0(base_url, protocol_url_segment)
read in the HTML content of the page
page <- read_html(url)
extract the tricell rows as a list of nodes
tricell_rows <- page %>% html_nodes("tr.tricell")
function to extract the text from row td elements and create a row vector
extract_td_text <- function(row) { td_elements <- html_children(row) td_texts <- sapply(td_elements, html_text) td_texts <- trimws(td_texts) return(as.vector(td_texts)) }
extract the text from the td elements in each row and store them in rows_list
rows_list <- lapply(tricell_rows, extract_td_text)
create a data frame from the rows_list
tricell_df <- do.call(rbind.data.frame, c(rows_list, stringsAsFactors = FALSE)) colnames(tricell_df) <- c("Column1", "Column2", "Column3")
[TRUNCATED]
data_list <- list(data_A, data_B, data_D, data_E, data_F, data_G, data_H, data_N, data_P) df_list <- list(df_A, df_B, df_D, df_E, df_F, df_G, df_H, df_N, df_P)
Loop through each pair of data_ and df_ dataframes
for (k in 1:length(data_list)) { data_df <- data_list[[k]] merge_df <- df_list[[k]]
if (nrow(merge_df) == 0) { next }
new_row <- data_df[1,] # Create a new row based on the structure of the data_ dataframe
Set all values in the new row to NA except the protocol_url_segment column
new_row[,-1] <- NA new_row$protocol_url_segment <- protocol_url_segment
Define the starting codes for new entries
starting_codes <- c("B.1.1", "D.1.2andD.1.3", "G.4.1")
entry_counter <- 1 # Initialize entry counter
Loop through each row in the df_ dataframe
for (i in 1:nrow(merge_df)) { code <- merge_df[i, 1] value <- merge_df[i, 3]
# Check if the code exists in the data_ dataframe if (code %in% colnames(data_df)) { # If the code is a starting code of a new entry and it is not the first row # then add the current new_row to the data_ dataframe and create a new new_row if (code %in% starting_codes && i > 1) { # Assign the entry counter value to the corresponding column if (k == 2) { # data_B new_row$sponsor_number <- entry_counter } else if (k == 3) { # data_D new_row$IMP_number <- entry_counter } else if (k == 6) { # data_G new_row$network_number <- entry_counter } data_list[[k]] <- rbind(data_list[[k]], new_row) new_row <- data_df[1,] new_row[,-1] <- NA new_row$protocol_url_segment <- protocol_url_segment entry_counter <- entry_counter + 1 # Increment the entry counter } # Insert the value at the corresponding column in the new row new_row[, code] <- value }}
Assign the entry counter value to the corresponding column in the last new_row
if (k == 2) { # data_B new_rowsponsor_number <- entry_counter } else if (k == 3) { # data_D new_rowIMP_number <- entry_counter } else if (k == 6) { # data_G new_row$network_number <- entry_counter }
Add the last new row to the data_ dataframe
data_list[[k]] <- rbind(data_list[[k]], new_row) }
Update the data_ dataframes with the merged values
data_A <- data_list[[1]] data_B <- data_list[[2]] data_D <- data_list[[3]] data_E <- data_list[[4]] data_F <- data_list[[5]] data_G <- data_list[[6]] data_H <- data_list[[7]] data_N <- data_list[[8]] data_P <- data_list[[9]]
This is working great! Great! How can i loop through multiple protocol_url_segment values and compile them in the data_ dataframes?
- Assistant
You can create a list of protocol_url_segments and then loop through them using a for loop. Modify the loop through each pair of data_ and df_ dataframes section with an outer loop which repeats the entire process for every protocol_url_segment. Finally, bind the results of each iteration together.
Please note that you cannot use "2015-003589-10/GR" as a protocol_url_segment because it contains a "/" which is not supported in URLs. I am using a dummy list of protocol_url_segments for demonstration purposes. Replace it with your actual list.
Here's the modified code:
# List of protocol_url_segments protocol_url_segments <- c("segment1", "segment2", "segment3") # Create empty data_ dataframes to store results for each protocol_url_segment result_data_A <- data_A result_data_B <- data_B result_data_D <- data_D result_data_E <- data_E result_data_F <- data_F result_data_G <- data_G result_data_H <- data_H result_data_N <- data_N result_data_P <- data_P # Loop through each protocol_url_segment for (url_segment in protocol_url_segments) { protocol_url_segment <- url_segment url <- paste0(base_url, protocol_url_segment) page <- read_html(url) tricell_rows <- page %>% html_nodes("tr.tricell") rows_list <- lapply(tricell_rows, extract_td_text) tricell_df <- do.call(rbind.data.frame, c(rows_list, stringsAsFactors = FALSE)) colnames(tricell_df) <- c("Column1", "Column2", "Column3") # ... [TRUNCATED] # Update the data_ dataframes with the merged values result_data_A <- rbind(result_data_A, data_list[[1]]) result_data_B <- rbind(result_data_B, data_list[[2]]) result_data_D <- rbind(result_data_D, data_list[[3]]) result_data_E <- rbind(result_data_E, data_list[[4]]) result_data_F <- rbind(result_data_F, data_list[[5]]) result_data_G <- rbind(result_data_G, data_list[[6]]) result_data_H <- rbind(result_data_H, data_list[[7]]) result_data_N <- rbind(result_data_N, data_list[[8]]) result_data_P <- rbind(result_data_P, data_list[[9]]) } # Remove the initial empty rows from the result_data_ dataframes result_data_A <- result_data_A[-1, ] result_data_B <- result_data_B[-1, ] result_data_D <- result_data_D[-1, ] result_data_E <- result_data_E[-1, ] result_data_F <- result_data_F[-1, ] result_data_G <- result_data_G[-1, ] result_data_H <- result_data_H[-1, ] result_data_N <- result_data_N[-1, ] result_data_P <- result_data_P[-1, ]Replace the
protocol_url_segmentslist with your actual list and the code will loop through all of them, scraping and storing the data in the result_data_ dataframes. - User
Help me make a shiny app with the following:
Title Instruction text Input box for a one line variable. Start button
The start button will start EUCTR_shiny.R script. But that script will need the inputted variable.
- Assistant
To create a simple Shiny app, you will need two separate R files:
ui.Randserver.R.ui.Rwill handle the app's user interface, andserver.Rwill handle the app's background processing.Here's a brief outline of a Shiny app structure that meets your requirements:
- Create a folder named
shiny_appin your working directory. - Inside the folder
shiny_app, create two R script files:ui.Randserver.R.
In the
ui.R, add the following code:library(shiny) # User interface definition shinyUI(fluidPage( # App title titlePanel("Your App Title"), # Instruction text p("Enter the variable and click Start to run the EUCTR_shiny script."), # Input box for a one-line variable textInput("input_var", "Enter a variable:"), # 'Start' button actionButton("start", "Start"), # Output console verbatimTextOutput("shiny_output") ))In the
server.R, add the following:library(shiny) source("EUCTR_shiny.R") shinyServer(function(input, output) { # When the Start button is pressed observeEvent(input$start, { # Call the EUCTR_shiny script function and pass the input variable output$shiny_output <- renderPrint({ EUCTR_function(input$input_var) }) }) })In your
EUCTR_shiny.R, create a function to wrap your script and make it accept the input variable. Replaceyour_variablein the script with theinput_varparameter:EUCTR_function <- function(input_var) { # Your EUCTR_shiny.R script # Replace "your_variable" with "input_var" in the script }Finally, run the Shiny app by opening an R session in the
shiny_appfolder and entering:library(shiny) runApp()This Shiny app will display a title, an instruction text, an input box for a one-line variable, and a 'Start' button. Once you click the 'Start' button, the
EUCTR_shiny.Rscript will run using the inputted variable from the user.For more functionality and customization, please refer to the Shiny documentation.
- Create a folder named