Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0314· openaiHugging Face Suzhen/CodeChat

You are a helpful assistant, working in a R shiny app with a human. It's a web scrapper…

  1. User

    You are a helpful assistant, working in a R shiny app with a human. It's a web scrapper for EU Clinical Trials Register. Ask any clarifying questions before proceeding. Avoid repeating all the code when editing. Try instead to just tell what to add or change.

    Here's our code so far:

    driver.R:

    EUCTR_shiny_script <- function(selected_input, selected_data_sets, selected_option, selected_format, process_time) {

    library(rvest) library(tidyverse) library(tidyr) library(openxlsx) library(readxl)

    Create a new environment to store shared variables

    my_env <- new.env()

    Define variables

    EUCTR_version <- "0.2" process_time <- format(Sys.time(), "%Y%m%d_%H%M%S") filename <- paste0("EUCTR_", process_time, ".xlsx") filepath <- file.path("outputs", filename)

    Assign the variables to the environment

    assign("selected_input", selected_input, envir = my_env) assign("selected_data_sets", selected_data_sets, envir = my_env) assign("selected_option", selected_option, envir = my_env) assign("selected_format", selected_format, envir = my_env) assign("EUCTR_version", EUCTR_version, envir = my_env) assign("process_time", process_time, envir = my_env) assign("filename", filename, envir = my_env)
    assign("filepath", filepath, envir = my_env)

    withProgress( message = "Crawling pages...", value = 0, source('./crawler.R', local = my_env) )

    if ("protocol" %in% selected_data_sets) { withProgress( message = "Scraping protocol data...", value = 0.1, source('./protocol.R', local = my_env) ) }

    if ("results" %in% selected_data_sets) { withProgress( message = "Scraping results data...", value = 0.5, source('./results.R', local = my_env) ) withProgress( message = "Scraping subjects disposition...", value = 0.5, source('./disposition.R', local = my_env) ) }

    if ("endpoints" %in% selected_data_sets) { withProgress( message = "Scraping endpoints data...", value = 0.1, source('./endpoints.R', local = my_env) ) }

    withProgress( message = "Saving to file...", value = 0.92, source('./savexlsx.R', local = my_env) )

    Return a list containing the filepath and filename variables

    return(list(filepath = filepath, filename = filename))

    }


    How can we add a logging feature, logging errors, warnings, etc? Maybe using log4r?

  2. Assistant

    Yes, using log4r package is a good choice for implementing logging in your R shiny app. First, you will need to install and import log4r. Add the following line to your code:

    library(log4r)
    

    Next, create a logger object and configure it. Add the following lines of code after assigning the variables to the environment as this will include 'process_time' in the log file name:

    log_file_name <- paste0("EUCTR_log_", process_time, ".txt")
    log_file_path <- file.path("logs", log_file_name)
    logger <- create.logger(logfile = log_file_path, level = "INFO")
    
    assign("logger", logger, envir = my_env)
    

    Now that you have a Logger object, you can use it to log messages throughout the code. For example, use:

    loginfo(logger, "This is an INFO level log message")
    logwarn(logger, "This is a WARNING level log message")
    logerror(logger, "This is an ERROR level log message")
    

    Add appropriate log messages throughout your code by using these log functions provided by log4r. Be sure to replace the example messages with more descriptive messages that fit the context of your code.

    Remember to close the logger when you are done using it at the end of driver.R:

    close.logger(logger)
    

    Make sure to create a "logs" folder in your project directory where the log files will be stored. With these changes, your R shiny app will now log errors, warnings, and other information using log4r.

  3. User

    Warning: Error in loginfo: could not find function "loginfo"

  4. Assistant

    Apologies for the confusion. I provided the incorrect log4r function names. Instead of using loginfo, logwarn, and logerror, you should use the following log4r functions:

    info(logger, "This is an INFO level log message")
    warn(logger, "This is a WARNING level log message")
    error(logger, "This is an ERROR level log message")
    

    Make sure to replace the previous log functions (loginfo, logwarn, logerror) with the correct ones mentioned above.

    With these changes, the logging feature should now work as expected.

  5. User

    How can we implement a catcher to store any errors, warnings, etc?

  6. Assistant

    You can create a catcher using the tryCatch() function to handle errors, warnings, and other conditions, and then log them using the logger you set up earlier. Here's an example of how to implement a catcher:

    1. Wrap the main code inside tryCatch():
    tryCatch(
      {
        # Your main code here
      
      },
      error = function(err) {
        log.error(logger, sprintf("Error caught: %s", err))
        return(NULL)
      },
      warning = function(warn) {
        log.warn(logger, sprintf("Warning caught: %s", warn))
        return(NULL)
      },
      finally = {
        close.logger(logger)
      }
    )
    
    1. Update your error and warning logging functions to use log.error and log.warn:
    log.error(logger, "This is an ERROR level log message")
    log.warn(logger, "This is a WARNING level log message")
    
    1. Update the main code to have the catching mechanism:
    result <- tryCatch(
      {
        # Your main code here
      },
      error = function(err) {
        log.error(logger, sprintf("Error caught: %s", err$message))
        return(NULL)
      },
      warning = function(warn) {
        log.warn(logger, sprintf("Warning caught: %s", warn$message))
        return(NULL)
      },
      finally = {
        close.logger(logger)
      }
    )
    

    By using the tryCatch() function, you can effectively catch any errors, warnings, and other conditions that occur during the execution of your code and log them using the logger.

    Please remember to wrap the relevant sections of your code, like the source() calls or other individual functions where you want to catch errors or warnings using tryCatch() as shown in the example above.

  7. User

    Warning: Error in close.logger: could not find function "close.logger"

  8. Assistant

    Apologies for the mistake. I provided the wrong function for closing the logger in the log4r package. The correct function to close the logger is log4r::shutdown():

    log4r::shutdown()
    

    You can also remove the finally block in the previous example and simply put the log4r::shutdown() function at the end of your script:

    result <- tryCatch(
      {
        # Your main code here
      },
      error = function(err) {
        log.error(logger, sprintf("Error caught: %s", err$message))
        return(NULL)
      },
      warning = function(warn) {
        log.warn(logger, sprintf("Warning caught: %s", warn$message))
        return(NULL)
      }
    )
    
    log4r::shutdown()
    

    By calling log4r::shutdown() at the end of your script, the logger should close properly.

  9. User

    Warning: Error in : 'shutdown' is not an exported object from 'namespace:log4r'

  10. Assistant

    Apologies for the confusion earlier. I made a mistake; log4r package does not provide a specific function to shut down the logger. In that case, you don't need to call any shutdown() function. The logger will automatically save each log entry after it's written.

    Simply remove the log4r::shutdown() line from your code, and it should work without any issues.

  11. User

    EUCTR_shiny_script <- function(selected_input, selected_data_sets, selected_option, selected_format, process_time) {

    library(rvest) library(tidyverse) library(tidyr) library(openxlsx) library(readxl) library(log4r)

    Create a new environment to store shared variables

    my_env <- new.env()

    Define variables

    EUCTR_version <- "0.2" process_time <- format(Sys.time(), "%Y%m%d_%H%M%S") filename <- paste0("EUCTR_", process_time, ".xlsx") filepath <- file.path("outputs", filename)

    Assign the variables to the environment

    assign("selected_input", selected_input, envir = my_env) assign("selected_data_sets", selected_data_sets, envir = my_env) assign("selected_option", selected_option, envir = my_env) assign("selected_format", selected_format, envir = my_env) assign("EUCTR_version", EUCTR_version, envir = my_env) assign("process_time", process_time, envir = my_env) assign("filename", filename, envir = my_env)
    assign("filepath", filepath, envir = my_env)

    log_file_name <- paste0("EUCTR_log_", process_time, ".txt") log_file_path <- file.path("logs", log_file_name) logger <- create.logger(logfile = log_file_path, level = "INFO")

    assign("logger", logger, envir = my_env)

    info(logger, "This is an INFO level log message") warn(logger, "This is a WARNING level log message") error(logger, "This is an ERROR level log message")

    tryCatch({ withProgress( message = "Crawling pages...", value = 0, source('./crawler.R', local = my_env) )

    if ("protocol" %in% selected_data_sets) {
      withProgress(
        message = "Scraping protocol data...",
        value = 0.1,
        source('./protocol.R', local = my_env)
      )
    }
    
    if ("results" %in% selected_data_sets) {
      withProgress(
        message = "Scraping results data...",
        value = 0.5,
        source('./results.R', local = my_env)
      )
      withProgress(
        message = "Scraping subjects disposition...",
        value = 0.5,
        source('./disposition.R', local = my_env)
      )
    }
    
    if ("endpoints" %in% selected_data_sets) {
      withProgress(
        message = "Scraping endpoints data...",
        value = 0.1,
        source('./endpoints.R', local = my_env)
      )
    }
    
    withProgress(
      message = "Saving to file...",
      value = 0.92,
      source('./savexlsx.R', local = my_env)
    )
    
    # Return a list containing the filepath and filename variables
    return(list(filepath = filepath, filename = filename))
    

    }, error = function(err) { log.error(logger, sprintf("Error caught: %s", err)) return(NULL) }, warning = function(warn) { log.warn(logger, sprintf("Warning caught: %s", warn)) return(NULL) }, )

    }

    That's the code with the implementations you suggested. Maybe i did something wrong, but I got this message: Warning: Error in tryCatch: argument is missing, with no default

  12. Assistant

    Apologies for the confusion. My initial response didn't include the correct structure for error and warning messages. To properly catch errors and warnings, you need to use the conditionMessage() function on the respective err and warn objects. Here's the corrected code snippet:

      tryCatch({
        # Your main code here
      },
      error = function(err) {
        log.error(logger, sprintf("Error caught: %s", conditionMessage(err)))
        return(NULL)
      },
      warning = function(warn) {
        log.warn(logger, sprintf("Warning caught: %s", conditionMessage(warn)))
        return(NULL)
      }
      )
    

    Now, replace the tryCatch() block in your code with the corrected snippet above. The error message you encountered should no longer occur, and your code should work as expected.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.