You're an helpful programmer assistant, working with a human user developing a web…
- User
You're an helpful programmer assistant, working with a human user developing a web scrapper for a particular website in RStudio using rvest package. Ask any clarifying questions before proceeding if needed. Avoid writing all the code, and try to focus on what needs to be changed or added.
I need to get multiple variables from that "table" (table#subjectDispositionSection.sectionTable) The first part of the data is in tabular form, with only two columns: first column with the title of the variable (class="labelColumn"), and second column is the actual value (class="valueColumn"). The second part of the data is in a tabular form with a variable number of columns (class="embeddedTableContainer"). We're focusing on the first part for now. Ask any clarifying questions before proceeding if needed.
Here's the code so far:
library(rvest)
url <- "https://www.clinicaltrialsregister.eu/ctr-search/trial/2016-000172-19/results" # replace with the URL of the website you want to scrape webpage <- read_html(url)
Extract the data using the provided CSS selector and classes:
table_selector <- "table#subjectDispositionSection.sectionTable" label_selector <- ".labelColumn" value_selector <- ".valueColumn"
label_nodes <- html_nodes(webpage, paste(table_selector, label_selector, sep = " ")) value_nodes <- lapply(label_nodes, function(node) { html_node(node, xpath = "following-sibling::*[1][contains(@class, 'valueColumn')]") })
labels <- trimws(html_text(label_nodes)) values <- trimws(sapply(value_nodes, html_text))
Combine the extracted labels and values into a data frame:
data <- data.frame(Variable = labels, Value = values, stringsAsFactors = FALSE)
THe problem is that the code so far is extracting the second part as well, but it is intended to get the first part only. It should ignore any data inside those embedded tables (class="embeddedTableContainer"). Here's the HTML code for one of those embedded tables to be ignored, in case it is helpful:
<tbody><tr> <td class="header labelColumn"> <div>Number of subjects in period 1 </div> </td> <td class="valueColumn centeredValueColumn">Empagliflozin</td> <td class="valueColumn centeredValueColumn">Placebo</td> </tr> <tr> <td class="labelColumn"> <div> Started</div> </td> <td class="valueColumn centeredValueColumn"><div>22</div></td> <td class="valueColumn centeredValueColumn"><div>22</div></td> </tr> <tr> <td class="labelColumn"> <div> Completed</div> </td> <td class="valueColumn centeredValueColumn"><div>22</div></td> <td class="valueColumn centeredValueColumn"><div>22</div></td> </tr> </tbody></table> </div> </td> </tr> - Assistant
If you want to exclude the embedded tables, you can modify the
label_nodesandvalue_nodesextraction lines to skip label and value columns inside the embedded tables. You can achieve this by changing the CSS selector to exclude the.embeddedTableContainerclass using the:notpseudo-class.Change the
label_nodesextraction line to:label_nodes <- html_nodes(webpage, paste(table_selector, ":not(.embeddedTableContainer)", label_selector, sep = " "))And the
value_nodesextraction line to:value_nodes <- lapply(label_nodes, function(node) { html_node(node, xpath = "following-sibling::*[1][contains(@class, 'valueColumn') and not(ancestor::td[contains(@class, 'embeddedTableContainer')])]") })This will prevent the extraction of the label and value columns inside the embedded tables.
- User
That didnt work. Two problems:
.labelColumn elements that are inside a embedded table are still being scrped. no values were extracted. all is NANot sure if it was clear, but .labelColumn elements can be inside embedded tables in any depth level. I want to ignore anything inside them. Also, this warnings appear: In xml_find_first.xml_node(x, make_selector(css, xpath)) : Invalid expression [1207]
- Assistant
I apologize for the confusion. Based on the new information provided, I suggest using the following modification to the extraction lines to exclude the
.embeddedTableContainerclass and its children from the selection:First, change the
label_nodesextraction line to:label_nodes <- html_nodes(webpage, paste(table_selector, label_selector, sep = " ")) label_nodes <- label_nodes[!xpathSApply(label_nodes, 'boolean(ancestor::td[contains(@class, "embeddedTableContainer")])', xmlValue, T)]Second, change the
value_nodesextraction line to:value_nodes <- lapply(label_nodes, function(node) { html_node(node, xpath = "following-sibling::*[1][contains(@class, 'valueColumn') and not(ancestor::td[contains(@class, 'embeddedTableContainer')])]") })This approach should filter out the
.labelColumnelements inside theembeddedTableContainerand also extract the correct values. - User
Error in xpathSApply(label_nodes, "boolean(ancestor::td[contains(@class, "embeddedTableContainer")])", : could not find function "xpathSApply"
- Assistant
I apologize for the confusion. The correct function to use is
xpathApply. Please change thelabel_nodesextraction line to:label_nodes <- html_nodes(webpage, paste(table_selector, label_selector, sep = " ")) label_nodes <- label_nodes[!xpathApply(label_nodes, 'boolean(ancestor::td[contains(@class, "embeddedTableContainer")])', xmlValue, T)]This should work as intended, filtering out the
.labelColumnelements inside theembeddedTableContainerand extracting the correct values. - User
Error in UseMethod("xpathApply") : no applicable method for 'xpathApply' applied to an object of class "xml_nodeset"
- Assistant
I apologize for the confusion once again. The correct function to use is
xpathSApplyfrom theXMLpackage. Please make sure to load theXMLpackage before executing the code:library(XML)Then, update the
label_nodesextraction line to:label_nodes <- html_nodes(webpage, paste(table_selector, label_selector, sep = " ")) label_nodes <- label_nodes[!sapply(label_nodes, function(node) { xpathSApply(node, 'boolean(ancestor::td[contains(@class, "embeddedTableContainer")])', xmlValue) }, USE.NAMES = FALSE)]This should work as intended, filtering out the
.labelColumnelements inside theembeddedTableContainerand extracting the correct values.