library(rvest) library(tidyverse) library(tidyr) library(openxlsx) library(readxl) Read…

You can modify the code to scrape only the data inside the specified table by adding the html_nodes() function with the appropriate CSS selector before extracting the data. In this case, you can use the '#adverseEventsSection' CSS selector to target the table ID 'adverseEventsSection'.

Here's the modified code:

library(rvest)
library(tidyverse)
library(tidyr)
library(openxlsx)
library(readxl)

# Read the EudraCT codes from the file (reads first column from first sheet)
eudract_codes <- read_excel("EUCTR_rvest_data/EUCTR_output.xlsx", sheet = 1, col_names = FALSE, skip = 1)[[1]]

# Remove duplicates
eudract_codes <- unique(eudract_codes)

# Define the variables to scrape
variables <- c("Reporting group title",
               "Reporting group description")

# Create an empty dataframe to store the cumulative data
cumulative_group_description <- data.frame(matrix(ncol = length(variables) + 1, nrow = 0))
colnames(cumulative_group_description) <- c("EudraCT_code", variables)

# Loop through each EudraCT code
for (eudract_code in eudract_codes) {
  
  # Construct the URL using the EudraCT code
  url <- paste0("https://www.clinicaltrialsregister.eu/ctr-search/trial/", eudract_code, "/results")
  
  # Read the HTML content of the trial results page
  page <- read_html(url)
  
  # Extract the adverse events table using the '#adverseEventsSection' CSS selector
  table_node <- page %>% html_nodes("#adverseEventsSection")
  
  # Extract the data for each variable within the table
  data_list <- lapply(variables, function(var) {
    values <- table_node %>% 
      html_nodes(paste0("td.labelColumn:contains('", var, "') + td.valueColumn")) %>%
      html_text(trim = T)
    return(values)
  })
  
  # Combine the data into a list with the EudraCT code and the variable values
  data_list <- c(list(eudract_code), data_list)
  
  # Find the max number of rows needed for this EudraCT code
  num_rows <- max(sapply(data_list, length))
  
  # Create temporary dataframe
  temp_df <- data.frame(matrix(ncol = length(variables) + 1, nrow = num_rows))
  colnames(temp_df) <- c("EudraCT_code", variables)
  
  # Populate temporary dataframe with extracted data
  for (i in 1:length(data_list)) {
    temp_df[[i]] <- rep(data_list[[i]], length.out = num_rows)
  }
  
  # Append temporary dataframe to summary dataframe
  cumulative_group_description <- rbind(cumulative_group_description, temp_df)
}

# Export cumulative dataframe to xlsx
write.xlsx(cumulative_group_description, "EUCTR_rvest_data/Group_Descriptions.xlsx", rowNames = FALSE)

Make sure to modify the path to your Excel file according to your file setup.

Top-p (nucleus sampling) and Temperature are two parameters used to control the output diversity and quality during text generation from a language model like GPT (Generative Pre-trained Transformer).

1. Top-p (nucleus sampling): This is a sampling strategy where the model selects the next token in a generated sequence from a narrowed down distribution rather than from the full distribution of potential tokens. The probability distribution is truncated, considering only the tokens with the highest probability values that sum up to p. For example, if we set the top-p parameter to 0.9, the model would only consider the tokens with the highest probability that together account for 90% of the total probability mass. This selective process filters out less probable tokens, which reduces the chances of generating less meaningful text. Top-p (nucleus sampling) introduces diversity into the generated text without sacrificing quality.

2. Temperature: This parameter controls the randomness in the output text. It scales the probability distribution of the tokens. A higher temperature (e.g., 1.0 or greater) yields more random outputs, resulting in a more diverse set of generated sequences. Conversely, a lower temperature (e.g., 0.1 or less) leads to more focused sampling, which means the generated text is more deterministic and influenced by the highest probability tokens. In other words, lower temperature values make the model generate more conservative text, while higher temperature values make it more creative but potentially less coherent.

Both parameters are critical when fine-tuning text generation models like GPT, and they can be adjusted depending on the use case and desired level of text diversity and quality.

When using GPT for coding assistance, you would want the generated code to be accurate, coherent, and contextually relevant. Therefore, it is important to balance the generation parameters properly. Here are some suggestions for adjusting Top-p (nucleus sampling) and Temperature:

1. Top-p (nucleus sampling): Since coding assistance requires more precise token generation, it is better to set the Top-p value relatively high, such as around 0.8 to 0.95. This ensures that the selected tokens come from the more probable part of the distribution, making the generated code more likely to be correct and relevant to the context.

2. Temperature: For coding assistance, you want the generated code to be focused and coherent. A low to moderate temperature value, such as 0.5 to 0.7, would be a good starting point. Lower temperature values lead to more deterministic outputs, while slightly higher values still allow for some diversity in generated code. If the generated code is too repetitive or bland, you can slightly increase the temperature to introduce more variability.

Please note that the optimal values for these parameters can depend on the specific model you're using and the context of your task. Thus, it is recommended to experiment with different values and observe the quality of the generated code. You can perform several iterations with different parameter values and choose the settings that yield the best results for your coding assistance task.

To include the "eudract_code" in the first column of the output data frame, you can make the following modifications to the code:

1. Add "EudraCT_code" to the list of variables.
2. Initialize the output data frame with an additional column for "EudraCT_code".
3. Assign the "eudract_code" value to the first column of the new row before appending it to the output data frame.

Here's the modified code:

# Define the variables to scrape
variables <- c("EudraCT_code", # Add EudraCT_code to the list of variables
               "EudraCT number",
               ...
               "Dictionary version"
               )

# Create empty data frame
output_df <- data.frame(matrix(ncol = length(variables), nrow = 0))
colnames(output_df) <- variables

# Loop through EudraCT codes
for (eudract_code in eudract_codes) {
  
  # Construct the URL using the EudraCT code
  url <- paste0("https://www.clinicaltrialsregister.eu/ctr-search/trial/", eudract_code, "/results")
  
  # Scrape webpage
  page <- read_html(url)
  
  # Extract values for each variable (excluding EudraCT_code)
  values <- sapply(variables[-1], function(var) { # Exclude the first variable (EudraCT_code) from the loop
    nodes <- page %>%
      html_nodes(paste0("td.labelColumn:contains('", var, "') + td.valueColumn"))
    if (length(nodes) == 0) {
      value <- NA
    } else {
      value <- nodes %>% html_text(trim = TRUE)
    }
    return(value)
  })
  
  # Add values to output data frame
  new_row <- data.frame(t(values), stringsAsFactors = FALSE)
  colnames(new_row) <- variables[-1] # Set column names excluding EudraCT_code
  new_row$EudraCT_code <- eudract_code # Add EudraCT_code to the first column of the new row
  output_df <- rbind(output_df, new_row)
  
}

This code will include the "eudract_code" in the first column of the output data frame, making it easier to track which EudraCT code corresponds to each row of data.

Sure, you can create the output data frame with an additional column for "EudraCT_code" without adding it to the list of variables. Here's the modified code:

# Define the variables to scrape
variables <- c("EudraCT number",
               ...
               "Dictionary version"
               )

# Create empty data frame with an additional column for "EudraCT_code"
output_df <- data.frame(matrix(ncol = length(variables) + 1, nrow = 0))
colnames(output_df) <- c("EudraCT_code", variables)  # Add "EudraCT_code" as the first column name

# Loop through EudraCT codes
for (eudract_code in eudract_codes) {
  
  # Construct the URL using the EudraCT code
  url <- paste0("https://www.clinicaltrialsregister.eu/ctr-search/trial/", eudract_code, "/results")
  
  # Scrape webpage
  page <- read_html(url)
  
  # Extract values for each variable
  values <- sapply(variables, function(var) {
    nodes <- page %>%
      html_nodes(paste0("td.labelColumn:contains('", var, "') + td.valueColumn"))
    if (length(nodes) == 0) {
      value <- NA
    } else {
      value <- nodes %>% html_text(trim = TRUE)
    }
    return(value)
  })
  
  # Add values to output data frame
  new_row <- data.frame(t(values), stringsAsFactors = FALSE)
  colnames(new_row) <- variables
  new_row$EudraCT_code <- eudract_code # Add EudraCT_code to the first column of the new row
  output_df <- rbind(output_df, new_row)
  
}

This code will create an output data frame with an additional column for "EudraCT_code" without adding it to the list of variables. The "eudract_code" will be listed in the first column of the output data frame.

Yes, you can use a counter to keep track of the number of iterations where the table with header 'Serious adverse events' is found or not found. Here's how you can modify the code to include a counter:

# Initialize counters
found_count <- 0
not_found_count <- 0

# Start the loop
for (i in 1:length(eudract_codes)) {
  
  # Your loop code here
  
  if (table_with_header_found) { # Replace this condition with the appropriate condition for your loop
    found_count <- found_count + 1
    # Process target_table_long as needed
  } else {
    not_found_count <- not_found_count + 1
  }
}

# Print the counters after the loop
cat("Number of tables with header 'Serious adverse events' found:", found_count, "\n")
cat("Number of tables with header 'Serious adverse events' not found:", not_found_count, "\n")

This code initializes two counters, found_count and not_found_count, before the loop starts. Inside the loop, it increments the appropriate counter based on whether the table with the header 'Serious adverse events' is found or not. After the loop, it prints the final counts for both cases.

You can use the txtProgressBar function from the base R package to create a progress bar for your loop. Here's how you can modify your code to include a progress bar:

# Initialize counters
found_count <- 0
not_found_count <- 0

# Create a progress bar
pb <- txtProgressBar(min = 0, max = length(eudract_codes), style = 3)

# Start the loop
for (i in 1:length(eudract_codes)) {
  
  # Your loop code here
  
  if (table_with_header_found) { # Replace this condition with the appropriate condition for your loop
    found_count <- found_count + 1
    # Process target_table_long as needed
  } else {
    not_found_count <- not_found_count + 1
  }
  
  # Update the progress bar
  setTxtProgressBar(pb, i)
}

# Close the progress bar
close(pb)

# Print the counters after the loop
cat("Number of tables with header 'Serious adverse events' found:", found_count, "\n")
cat("Number of tables with header 'Serious adverse events' not found:", not_found_count, "\n")

This code creates a progress bar using the txtProgressBar function before the loop starts. Inside the loop, it updates the progress bar using the setTxtProgressBar function. After the loop, it closes the progress bar using the close function and then prints the final counts for both cases.