Skip to content
TheChatArchiveThe Chat Archive
gpt-3.5-turbo-0301· OpenAIAi2 WildChat-4.8M (AI2)

使用java 爬取…

  1. User
  2. Assistant

    抓取网站内容需要用到网络请求库和HTML解析库。常见的网络请求库有Java原生的URLConnection和第三方库Apache HttpClient,HTML解析库有Jsoup、XPath等。

    以下是使用Java原生URLConnection和Jsoup库实现爬取的样例代码:

    import java.io.IOException;
    import java.net.URL;
    import java.net.URLConnection;
    import java.util.ArrayList;
    import java.util.List;
    import org.jsoup.Jsoup;
    import org.jsoup.nodes.Document;
    import org.jsoup.nodes.Element;
    import org.jsoup.select.Elements;
    
    public class Crawler {
    
        public static void main(String[] args) throws IOException {
            String url = "http://va.rizhao.gov.cn/col/col81263/index.html";
            String baseUrl = "http://va.rizhao.gov.cn";
    
            List<Article> articles = new ArrayList<>();
    
            URLConnection conn = new URL(url).openConnection();
    
            Document doc = Jsoup.parse(conn.getInputStream(), null, baseUrl);
    
            Elements articleBlocks = doc.select("div.default_pgContainer li a");
    
            for (Element articleBlock : articleBlocks) {
                String link = articleBlock.attr("href");
                String articleUrl = baseUrl + link;
    
                URLConnection articleConn = new URL(articleUrl).openConnection();
                Document articleDoc = Jsoup.parse(articleConn.getInputStream(), null, baseUrl);
    
                String title = articleDoc.select("div.ntcinfotitle h2").text();
                String time = articleDoc.select("div.ntcinfotitle p").text();
    
                Element articleContent = articleDoc.select("div.ntcinfocontent").first();
                articleContent.select("img").forEach(img -> {
                    String src = img.attr("src");
                    img.attr("src", baseUrl + src);
                });
                String content = articleContent.html();
    
                String source = articleDoc.select("div.ntcinfofrom span.infofrom").text();
    
                Article article = new Article(title, time, content, source);
                articles.add(article);
            }
    
            for (Article article : articles) {
                System.out.println(article);
            }
        }
    
        private static class Article {
            String title;
            String time;
            String content;
            String source;
    
            public Article(String title, String time, String content, String source) {
                this.title = title;
                this.time = time;
                this.content = content;
                this.source = source;
            }
    
            @Override
            public String toString() {
                return title + "\n" + time + "\n" + content + "\n" + source + "\n";
            }
        }
    }
    

    该代码会从目标网站获取信息,包括标题、时间、文章内容、信息来源,其中图片的相对地址会被改为绝对地址。

    Jsoup库提供了丰富的语法,可以帮助我们很方便地提取网页内容。在上文中,我们使用了doc.select(selector)方法来选取指定元素的内容,其中selector参数是一个CSS选择器。我们还用到了element.text()方法来获取元素文本,element.html()方法来获取元素包括标签在内的HTML代码。具体语法和方法的使用可以参考Jsoup官方文档。

    需要注意的是,在进行网络抓取时,考虑到目标网站的反爬虫机制,应该合理设置请求头信息,避免过于频繁的访问,控制抓取速度。

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.