gpt-3.5-turbo-0301· OpenAI
WildChat-4.8M (AI2)
使用java 爬取…
- User
- Assistant
抓取网站内容需要用到网络请求库和HTML解析库。常见的网络请求库有Java原生的URLConnection和第三方库Apache HttpClient,HTML解析库有Jsoup、XPath等。
以下是使用Java原生URLConnection和Jsoup库实现爬取的样例代码:
import java.io.IOException; import java.net.URL; import java.net.URLConnection; import java.util.ArrayList; import java.util.List; import org.jsoup.Jsoup; import org.jsoup.nodes.Document; import org.jsoup.nodes.Element; import org.jsoup.select.Elements; public class Crawler { public static void main(String[] args) throws IOException { String url = "http://va.rizhao.gov.cn/col/col81263/index.html"; String baseUrl = "http://va.rizhao.gov.cn"; List<Article> articles = new ArrayList<>(); URLConnection conn = new URL(url).openConnection(); Document doc = Jsoup.parse(conn.getInputStream(), null, baseUrl); Elements articleBlocks = doc.select("div.default_pgContainer li a"); for (Element articleBlock : articleBlocks) { String link = articleBlock.attr("href"); String articleUrl = baseUrl + link; URLConnection articleConn = new URL(articleUrl).openConnection(); Document articleDoc = Jsoup.parse(articleConn.getInputStream(), null, baseUrl); String title = articleDoc.select("div.ntcinfotitle h2").text(); String time = articleDoc.select("div.ntcinfotitle p").text(); Element articleContent = articleDoc.select("div.ntcinfocontent").first(); articleContent.select("img").forEach(img -> { String src = img.attr("src"); img.attr("src", baseUrl + src); }); String content = articleContent.html(); String source = articleDoc.select("div.ntcinfofrom span.infofrom").text(); Article article = new Article(title, time, content, source); articles.add(article); } for (Article article : articles) { System.out.println(article); } } private static class Article { String title; String time; String content; String source; public Article(String title, String time, String content, String source) { this.title = title; this.time = time; this.content = content; this.source = source; } @Override public String toString() { return title + "\n" + time + "\n" + content + "\n" + source + "\n"; } } }该代码会从目标网站获取信息,包括标题、时间、文章内容、信息来源,其中图片的相对地址会被改为绝对地址。
Jsoup库提供了丰富的语法,可以帮助我们很方便地提取网页内容。在上文中,我们使用了
doc.select(selector)方法来选取指定元素的内容,其中selector参数是一个CSS选择器。我们还用到了element.text()方法来获取元素文本,element.html()方法来获取元素包括标签在内的HTML代码。具体语法和方法的使用可以参考Jsoup官方文档。需要注意的是,在进行网络抓取时,考虑到目标网站的反爬虫机制,应该合理设置请求头信息,避免过于频繁的访问,控制抓取速度。
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.