摘要：工具和資料群高級爬蟲作者自建群，歡迎加入作者收集的爬蟲相關(guān)工具和資料簡介類是以命令行工具為參考，使用標(biāo)準(zhǔn)的實(shí)現(xiàn)的工具類。

QQ群 - Javascript高級爬蟲 - 作者自建群，歡迎加入！

awesome-java-crawler - 作者收集的爬蟲相關(guān)工具和資料

CUrl類是以命令行工具CUrl為參考，使用標(biāo)準(zhǔn)Java的HttpURLConnection實(shí)現(xiàn)的Http工具類。

項(xiàng)目地址: https://github.com/rockswang/...
中央倉庫: https://mvnrepository.com/art...

基于標(biāo)準(zhǔn)Java運(yùn)行庫的Http類實(shí)現(xiàn)，源碼兼容級別為1.6，適用性廣泛，可用于服務(wù)端、Android等Java環(huán)境

代碼精簡緊湊，僅一個(gè)1000余行的Java源文件，無任何外部依賴，可不用Maven直接源碼級重用

完全兼容CUrl命令行工具的常用開關(guān)，可直接作為命令行工具替代之

支持所有HTTP Method，支持多文件上傳

通過ThreadLocal解決了標(biāo)準(zhǔn)Java中Cookie只能全局保存的問題，可每線程獨(dú)立維護(hù)Cookie

可將線程中保存的Cookies序列化保存，方便建立Cookies池

支持HTTP認(rèn)證，支持HTTPS，可啟用/忽略證書安全

支持每連接代理，支持需認(rèn)證的HTTP/HTTPS代理

跳轉(zhuǎn)行為可控制，可獲取到每步跳轉(zhuǎn)的應(yīng)答頭信息

支持編程自定義應(yīng)答解析器

支持失敗重試，可編程自定義可重試異常

例1：POST表單提交

    public void httpPost() {
        CUrl curl = new CUrl("http://httpbin.org/post")
                .data("hello=world&foo=bar")
                .data("foo=overwrite");
        curl.exec();
        assertEquals(200, curl.getHttpCode());
    }

例2：通過Fiddler代理（抓包工具）訪問HTTPS站點(diǎn)

    public void insecureHttpsViaFiddler() {
        CUrl curl = new CUrl("https://httpbin.org/get")
                .proxy("127.0.0.1", 8888) // Use Fiddler to capture & parse HTTPS traffic
                .insecure();  // Ignore certificate check since it"s issued by Fiddler
        curl.exec();
        assertEquals(200, curl.getHttpCode());
    }

例3：上傳多個(gè)文件，一個(gè)內(nèi)存文件，一個(gè)物理文件

    public void uploadMultipleFiles() {
        CUrl.MemIO inMemFile = new CUrl.MemIO();
        try { inMemFile.getOutputStream().write("text file content blabla...".getBytes()); } catch (Exception ignored) {}
        CUrl curl = new CUrl("http://httpbin.org/post")
                .form("formItem", "value") // a plain form item
                .form("file", inMemFile)           // in-memory "file"
                .form("image", new CUrl.FileIO("D:	mpa2.png")); // A file in storage
        curl.exec();
        assertEquals(200, curl.getHttpCode());
    }

例4：模擬手機(jī)瀏覽器上的AJAX請求，添加自定義請求頭

    public void customUserAgentAndHeaders() {
        String mobileUserAgent = "Mozilla/5.0 (Linux; U; Android 8.0.0; zh-cn; KNT-AL10 Build/HUAWEIKNT-AL10) " 
                + "AppleWebKit/537.36 (KHTML, like Gecko) MQQBrowser/7.3 Chrome/37.0.0.0 Mobile Safari/537.36";
        Map fakeAjaxHeaders = new HashMap();
        fakeAjaxHeaders.put("X-Requested-With", "XMLHttpRequest");
        fakeAjaxHeaders.put("Referer", "http://somesite.com/fake_referer");
        CUrl curl = new CUrl("http://httpbin.org/get")
                .opt("-A", mobileUserAgent) // simulate a mobile browser
                .headers(fakeAjaxHeaders)   // simulate an AJAX request
                .header("X-Auth-Token: xxxxxxx"); // other custom header, this might be calculated elsewhere
        curl.exec();
        assertEquals(200, curl.getHttpCode());
    }

例5：多線程并發(fā)請求，線程間Cookies相互獨(dú)立

    public void threadSafeCookies() {
        final CountDownLatch count = new CountDownLatch(3);
        final CUrl[] curls = new CUrl[3];
        for (int i = 3; --i >= 0;) {
            final int idx = i;
            new Thread() {
                public void run() {
                    CUrl curl = curls[idx] = new CUrl("http://httpbin.org/get")
                            .cookie("thread" + idx + "=#" + idx);
                    curl.exec();
                    count.countDown();
                }
            }.start();
        }
        try { count.await(); } catch (Exception ignored) {} // make sure all requests are done
        assertEquals(200, curls[0].getHttpCode());
        assertEquals("thread0=#0", deepGet(curls[0].getStdout(jsonResolver, null), "headers.Cookie"));
        assertEquals("thread1=#1", deepGet(curls[1].getStdout(jsonResolver, null), "headers.Cookie"));
        assertEquals("thread2=#2", deepGet(curls[2].getStdout(jsonResolver, null), "headers.Cookie"));
    }

例6：編程自定義應(yīng)答解析器，使用JSoup解析HTML

    private CUrl.Resolver htmlResolver = new CUrl.Resolver() {
        @SuppressWarnings("unchecked")
        @Override
        public Document resolve(int httpCode, byte[] responseBody) throws Throwable {
            String html = new String(responseBody, "UTF-8");
            return Jsoup.parse(html);
        }
    };

    public void customResolver() {
        CUrl curl = new CUrl("http://httpbin.org/html");
        Document html = curl.exec(htmlResolver, null);
        assertEquals(200, curl.getHttpCode());
        assertEquals("Herman Melville - Moby-Dick", html.select("h1:first-child").text());
    }

例7：作為命令行工具使用，請求內(nèi)容參考例4

java -jar java-curl-1.2.0.jar https://httpbin.org/get ^
    -x 127.0.0.1:8888 -k ^
    -A "Mozilla/5.0 (Linux; U; Android 8.0.0; zh-cn; KNT-AL10 Build/HUAWEIKNT-AL10) AppleWebKit/537.36 (KHTML, like Gecko) MQQBrowser/7.3 Chrome/37.0.0.0 Mobile Safari/537.36" ^
    -H "Referer: http://somesite.com/fake_referer" ^
    -H "X-Requested-With: XMLHttpRequest" ^
    -H "X-Auth-Token: xxxxxxx"