Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0314· openaiHugging Face Suzhen/CodeChat

You're an helpful programmer assistant, working with a human user developing a web…

  1. User

    You're an helpful programmer assistant, working with a human user developing a web scrapper for a particular website in RStudio using rvest package. Ask any clarifying questions before proceeding if needed. Avoid writing all the code, and try to focus on what needs to be changed or added.

    I need to get multiple variables from that "table" (table#subjectDispositionSection.sectionTable) The first part of the data is in tabular form, with only two columns: first column with the title of the variable (class="labelColumn"), and second column is the actual value (class="valueColumn"). The second part of the data is in a tabular form with a variable number of columns (class="embeddedTableContainer"). We're focusing on the first part for now. Ask any clarifying questions before proceeding if needed.

    Here's the code so far:


    library(rvest)

    url <- "https://www.clinicaltrialsregister.eu/ctr-search/trial/2016-000172-19/results" # replace with the URL of the website you want to scrape webpage <- read_html(url)

    Extract the data using the provided CSS selector and classes:

    table_selector <- "table#subjectDispositionSection.sectionTable" label_selector <- ".labelColumn" value_selector <- ".valueColumn"

    label_nodes <- html_nodes(webpage, paste(table_selector, label_selector, sep = " ")) value_nodes <- lapply(label_nodes, function(node) { html_node(node, xpath = "following-sibling::*[1][contains(@class, 'valueColumn')]") })

    labels <- trimws(html_text(label_nodes)) values <- trimws(sapply(value_nodes, html_text))

    Combine the extracted labels and values into a data frame:

    data <- data.frame(Variable = labels, Value = values, stringsAsFactors = FALSE)


    THe problem is that the code so far is extracting the second part as well, but it is intended to get the first part only. It should ignore any data inside those embedded tables (class="embeddedTableContainer"). Here's the HTML code for one of those embedded tables to be ignored, in case it is helpful:


                <tbody><tr>
        <td class="header labelColumn">
            <div>Number of subjects in period 1            </div>
        </td>
                    <td class="valueColumn centeredValueColumn">Empagliflozin</td>
                    <td class="valueColumn centeredValueColumn">Placebo</td>
            </tr>
    
                <tr>
        <td class="labelColumn">
            <div>    Started</div>
        </td>
                    <td class="valueColumn centeredValueColumn"><div>22</div></td>
                    <td class="valueColumn centeredValueColumn"><div>22</div></td>
            </tr>
    
                            <tr>
        <td class="labelColumn">
            <div>    Completed</div>
        </td>
                    <td class="valueColumn centeredValueColumn"><div>22</div></td>
                    <td class="valueColumn centeredValueColumn"><div>22</div></td>
            </tr>
    
            
            
        
                </tbody></table>
            </div>
        </td>
    </tr>
    
  2. Assistant

    If you want to exclude the embedded tables, you can modify the label_nodes and value_nodes extraction lines to skip label and value columns inside the embedded tables. You can achieve this by changing the CSS selector to exclude the .embeddedTableContainer class using the :not pseudo-class.

    Change the label_nodes extraction line to:

    label_nodes <- html_nodes(webpage, paste(table_selector, ":not(.embeddedTableContainer)", label_selector, sep = " "))
    

    And the value_nodes extraction line to:

    value_nodes <- lapply(label_nodes, function(node) {
      html_node(node, xpath = "following-sibling::*[1][contains(@class, 'valueColumn') and not(ancestor::td[contains(@class, 'embeddedTableContainer')])]")
    })
    

    This will prevent the extraction of the label and value columns inside the embedded tables.

  3. User

    That didnt work. Two problems:

    .labelColumn elements that are inside a embedded table are still being scrped.
    no values were extracted. all is NA
    

    Not sure if it was clear, but .labelColumn elements can be inside embedded tables in any depth level. I want to ignore anything inside them. Also, this warnings appear: In xml_find_first.xml_node(x, make_selector(css, xpath)) : Invalid expression [1207]

  4. Assistant

    I apologize for the confusion. Based on the new information provided, I suggest using the following modification to the extraction lines to exclude the .embeddedTableContainer class and its children from the selection:

    First, change the label_nodes extraction line to:

    label_nodes <- html_nodes(webpage, paste(table_selector, label_selector, sep = " "))
    label_nodes <- label_nodes[!xpathSApply(label_nodes, 'boolean(ancestor::td[contains(@class, "embeddedTableContainer")])', xmlValue, T)]
    

    Second, change the value_nodes extraction line to:

    value_nodes <- lapply(label_nodes, function(node) {
      html_node(node, xpath = "following-sibling::*[1][contains(@class, 'valueColumn') and not(ancestor::td[contains(@class, 'embeddedTableContainer')])]")
    })
    

    This approach should filter out the .labelColumn elements inside the embeddedTableContainer and also extract the correct values.

  5. User

    Error in xpathSApply(label_nodes, "boolean(ancestor::td[contains(@class, "embeddedTableContainer")])", : could not find function "xpathSApply"

  6. Assistant

    I apologize for the confusion. The correct function to use is xpathApply. Please change the label_nodes extraction line to:

    label_nodes <- html_nodes(webpage, paste(table_selector, label_selector, sep = " "))
    label_nodes <- label_nodes[!xpathApply(label_nodes, 'boolean(ancestor::td[contains(@class, "embeddedTableContainer")])', xmlValue, T)]
    

    This should work as intended, filtering out the .labelColumn elements inside the embeddedTableContainer and extracting the correct values.

  7. User

    Error in UseMethod("xpathApply") : no applicable method for 'xpathApply' applied to an object of class "xml_nodeset"

  8. Assistant

    I apologize for the confusion once again. The correct function to use is xpathSApply from the XML package. Please make sure to load the XML package before executing the code:

    library(XML)
    

    Then, update the label_nodes extraction line to:

    label_nodes <- html_nodes(webpage, paste(table_selector, label_selector, sep = " "))
    label_nodes <- label_nodes[!sapply(label_nodes, function(node) { xpathSApply(node, 'boolean(ancestor::td[contains(@class, "embeddedTableContainer")])', xmlValue) }, USE.NAMES = FALSE)]
    

    This should work as intended, filtering out the .labelColumn elements inside the embeddedTableContainer and extracting the correct values.