问题

使用 java.net.URLConnection 常常出现在这里, Oracle教程也是简洁的.

该教程基本上只显示如何触发GET请求并读取响应.它不解释在任何地方如何使用它,以执行POST请求,设置请求标头,读取响应头,处理cookie,提交HTML表单,上传文件等.

那么,如何使用 java.net.URLConnection 启动和处理"高级"HTTP请求?



解决方法

首先是免责声明:发布的代码段都是基本示例.你需要处理像 NullPointerException , ArrayIndexOutOfBoundsException 这样简单的 IOException RuntimeException em>


Preparing

我们首先需要知道至少URL和字符集.这些参数是可选的,取决于功能要求.

String url = "http://example.com";
String charset = "UTF-8";  // Or in Java 7 and later, use the constant: java.nio.charset.StandardCharsets.UTF_8.name()
String param1 = "value1";
String param2 = "value2";
// ...

String query = String.format("param1=%s&param2=%s", 
     URLEncoder.encode(param1, charset), 
     URLEncoder.encode(param2, charset));

查询参数必须为 name = value 格式,并由& 连接.通常,您还可以使用指定的字符集 URL编码查询参数,使用 URLEncoder#encode() .

String#format()只是为了方便.我喜欢它,当我需要字符串连接运算符 + 两次以上.


Firing a HTTP GET request with (optionally) query parameters

这是一个简单的任务.这是默认的请求方法.

URLConnection connection = new URL(url + "?" + query).openConnection();
connection.setRequestProperty("Accept-Charset", charset);
InputStream response = connection.getInputStream();
// ...

任何查询字符串都应使用?连接到网址. 接受字符集 头可以暗示服务器参数所在的编码.如果您不发送任何查询字符串,那么您可以离开 Accept-Charset 头.如果您不需要设置任何标头,则甚至可以使用 URL#openStream() 快捷方式.

InputStream response = new URL(url).openStream();
// ...

无论如何,如果对方是 <代码> HttpServlet ,则其 doGet() 方法将被调用,参数将由 HttpServletRequest#getParameter() .

出于测试目的,您可以将响应正文打印到stdout,如下所示:

try (Scanner scanner = new Scanner(response)) {
    String responseBody = scanner.useDelimiter("\\A").next();
    System.out.println(responseBody);
}

Firing a HTTP POST request with query parameters

设置 URLConnection#setDoOutput() true 隐式地将请求方法设置为POST.标准的HTTP POST作为web表单做的是 application / x-www-form-urlencoded 类型,其中查询字符串被写入请求主体.

URLConnection connection = new URL(url).openConnection();
connection.setDoOutput(true); // Triggers POST.
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=" + charset);

try (OutputStream output = connection.getOutputStream()) {
    output.write(query.getBytes(charset));
}

InputStream response = connection.getInputStream();
// ...

注意:每当您要以程式化方式提交HTML表单时,不要忘记带任何&lt; input type ="hidden"&gt;的 name = value 元素添加到查询字符串中,当然还包括您想要的&lt; input type ="submit"&gt; 元素的 name = value 以"编程方式"(因为通常在服务器端使用,以区分是否按下按钮,如果是,则是哪个).

您还可以投射获取的 URLConnection HttpURLConnection ,并使用其 HttpURLConnection#setRequestMethod() .但如果您尝试使用连接输出,您仍然需要设置 URLConnection#setDoOutput() true

HttpURLConnection httpConnection = (HttpURLConnection) new URL(url).openConnection();
httpConnection.setRequestMethod("POST");
// ...

无论如何,如果对方是 <代码> HttpServlet ,则其 doPost() 方法将被调用,参数将由< ="http://docs.oracle.com/javaee/7/api/javax/servlet/ServletRequest.html#getParameter%28java.lang.String%29"rel ="noreferrer"> HttpServletRequest#getParameter() .


Actually firing the HTTP request

您可以使用 URLConnection#connect() ,但是当您想要获取有关HTTP响应的任何信息时,请求会自动触发,例如响应主体使用 URLConnection#getInputStream()< / code> 等.上面的例子正是这样,所以 connect()调用事实上是多余的.


Gathering HTTP response information

  1. HTTP response status:

    You need a HttpURLConnection here. Cast it first if necessary.

    int status = httpConnection.getResponseCode();
    
  2. HTTP response headers:

    for (Entry<String, List<String>> header : connection.getHeaderFields().entrySet()) {
        System.out.println(header.getKey() + "=" + header.getValue());
    }
    
  3. HTTP response encoding:

    When the Content-Type contains a charset parameter, then the response body is likely text based and we'd like to process the response body with the server-side specified character encoding then.

    String contentType = connection.getHeaderField("Content-Type");
    String charset = null;
    
    for (String param : contentType.replace(" ", "").split(";")) {
        if (param.startsWith("charset=")) {
            charset = param.split("=", 2)[1];
            break;
        }
    }
    
    if (charset != null) {
        try (BufferedReader reader = new BufferedReader(new InputStreamReader(response, charset))) {
            for (String line; (line = reader.readLine()) != null;) {
                // ... System.out.println(line) ?
            }
        }
    } else {
        // It's likely binary content, use InputStream/OutputStream.
    }
    

Maintaining the session

服务器端会话通常由Cookie支持.某些网络表单要求您登录和/或由会话跟踪.您可以使用 CookieHandler < / a> API来维护Cookie.您需要准备 CookieManager CookiePolicy ACCEPT_ALL < / code> ,然后再发送所有HTTP请求.

// First set the default cookie manager.
CookieHandler.setDefault(new CookieManager(null, CookiePolicy.ACCEPT_ALL));

// All the following subsequent URLConnections will use the same cookie manager.
URLConnection connection = new URL(url).openConnection();
// ...

connection = new URL(url).openConnection();
// ...

connection = new URL(url).openConnection();
// ...

请注意,这并不总是在所有情况下都能正常工作.如果它为你失败,那么最好是手动收集和设置cookie头.您基本上需要从登录或第一个 GET 请求的响应中获取所有 Set-Cookie 标头,然后将其传递到后续请求.

// Gather all cookies on the first request.
URLConnection connection = new URL(url).openConnection();
List<String> cookies = connection.getHeaderFields().get("Set-Cookie");
// ...

// Then use the same cookies on all subsequent requests.
connection = new URL(url).openConnection();
for (String cookie : cookies) {
    connection.addRequestProperty("Cookie", cookie.split(";", 2)[0]);
}
// ...

split(";",2)[0] 是摆脱与服务器端无关的cookie属性,如 expires 路径等.或者,您也可以使用 cookie.substring(0,cookie.indexOf(';'))而不是 split().< / p>


Streaming mode

HttpURLConnection < / a>将默认缓冲整个请求主体,然后才实际发送它,无论您是否使用自己设置固定的内容长度connection.setRequestProperty("Content-Length",contentLength ); .每当您同时发送大型POST请求(例如上传文件)时,这可能会导致 OutOfMemoryException .要避免这种情况,您需要设置 HttpURLConnection#setFixedLengthStreamingMode() .

httpConnection.setFixedLengthStreamingMode(contentLength);

但是如果内容长度事先不知道,那么您可以通过设置 HttpURLConnection#setChunkedStreamingMode() .这将设置HTTP Transfer-Encoding 头部到 chunked ,这将强制以块的形式发送请求正文.下面的示例将以1KB的大小块发送正文.

httpConnection.setChunkedStreamingMode(1024);

User-Agent

这可能发生,请求返回意外的响应,而它与一个真正的网络浏览器正常工作.服务器端可能阻止基于 User-代理 请求标头. URLConnection 将默认设置为 Java / 1.6.0_19 ,其中最后一部分显然是JRE版本.您可以覆盖此,如下所示:

connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"); // Do as if you're using Chrome 41 on Windows 7.

使用最近的浏览器中的User-Agent字符串.


Error handling

如果HTTP响应代码是 4nn (客户端错误)或 5nn (服务器错误),那么您可能想读取 HttpURLConnection#getErrorStream 以查看服务器是否发送了任何有用的错误信息.

InputStream error = ((HttpURLConnection) connection).getErrorStream();

如果HTTP响应代码为-1,则连接和响应处理出了问题. HttpURLConnection 实现是在旧的JRE有点bug,保持连接活着.您可能希望通过将 http.keepAlive 系统属性设置为 false 将其关闭.您可以通过以下方式在应用程序开头以编程方式执行此操作:

System.setProperty("http.keepAlive", "false");

Uploading files

您通常会使用 multipart / form-data 混合POST内容(二进制和字符数据)的编码.有关详情,请参阅 RFC2388 .

String param = "value";
File textFile = new File("/path/to/file.txt");
File binaryFile = new File("/path/to/file.bin");
String boundary = Long.toHexString(System.currentTimeMillis()); // Just generate some unique random value.
String CRLF = "\r\n"; // Line separator required by multipart/form-data.
URLConnection connection = new URL(url).openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);

try (
    OutputStream output = connection.getOutputStream();
    PrintWriter writer = new PrintWriter(new OutputStreamWriter(output, charset), true);
) {
    // Send normal param.
    writer.append("--" + boundary).append(CRLF);
    writer.append("Content-Disposition: form-data; name=\"param\"").append(CRLF);
    writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF);
    writer.append(CRLF).append(param).append(CRLF).flush();

    // Send text file.
    writer.append("--" + boundary).append(CRLF);
    writer.append("Content-Disposition: form-data; name=\"textFile\"; filename=\"" + textFile.getName() + "\"").append(CRLF);
    writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF); // Text file itself must be saved in this charset!
    writer.append(CRLF).flush();
    Files.copy(textFile.toPath(), output);
    output.flush(); // Important before continuing with writer!
    writer.append(CRLF).flush(); // CRLF is important! It indicates end of boundary.

    // Send binary file.
    writer.append("--" + boundary).append(CRLF);
    writer.append("Content-Disposition: form-data; name=\"binaryFile\"; filename=\"" + binaryFile.getName() + "\"").append(CRLF);
    writer.append("Content-Type: " + URLConnection.guessContentTypeFromName(binaryFile.getName())).append(CRLF);
    writer.append("Content-Transfer-Encoding: binary").append(CRLF);
    writer.append(CRLF).flush();
    Files.copy(binaryFile.toPath(), output);
    output.flush(); // Important before continuing with writer!
    writer.append(CRLF).flush(); // CRLF is important! It indicates end of boundary.

    // End of multipart/form-data.
    writer.append("--" + boundary + "--").append(CRLF).flush();
}

如果对方是 HttpServlet ,然后是 doPost() 方法将被调用,部件将可通过 HttpServletRequest#getPart()< / code> (注意,因此不是 getParameter()等等).然而, getPart()方法相对较新,它在Servlet 3.0(Glassfish 3,Tomcat 7等)中引入.在Servlet 3.0之前,最好的选择是使用 Apache Commons FileUpload 来解析 multipart / form-数据请求.另请参阅此答案,以了解FileUpload和Servelt 3.0的示例方法.


Dealing with untrusted or misconfigured HTTPS sites

有时您需要连接HTTPS网址,这可能是因为您在撰写网络刮板.在这种情况下,您可能在某些HTTPS网站上遇到了一个 javax.net.ssl.SSLException:不受信任的服务器证书,这些网站没有保持SSL证书最新,或者使用 java .security.cert.CertificateException:在某些配置错误的HTTPS网站上没有找到匹配[hostname]的主体备选DNS名称 javax.net.ssl.SSLProtocolException:handshake alert:unrecognized_name .

以下在您的web scraper类中的一次性运行 static 初始值设定器应该使 HttpsURLConnection 更宽松的那些HTTPS网站,因此不再抛出这些异常. / p>

static {
    TrustManager[] trustAllCertificates = new TrustManager[] {
        new X509TrustManager() {
            @Override
            public X509Certificate[] getAcceptedIssuers() {
                return null; // Not relevant.
            }
            @Override
            public void checkClientTrusted(X509Certificate[] certs, String authType) {
                // Do nothing. Just allow them all.
            }
            @Override
            public void checkServerTrusted(X509Certificate[] certs, String authType) {
                // Do nothing. Just allow them all.
            }
        }
    };

    HostnameVerifier trustAllHostnames = new HostnameVerifier() {
        @Override
        public boolean verify(String hostname, SSLSession session) {
            return true; // Just allow them all.
        }
    };

    try {
        System.setProperty("jsse.enableSNIExtension", "false");
        SSLContext sc = SSLContext.getInstance("SSL");
        sc.init(null, trustAllCertificates, new SecureRandom());
        HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
        HttpsURLConnection.setDefaultHostnameVerifier(trustAllHostnames);
    }
    catch (GeneralSecurityException e) {
        throw new ExceptionInInitializerError(e);
    }
}

Last words

Apache HttpComponents HttpClient 很方便更方便in this all:)


Parsing and extracting HTML

如果您只想从HTML解析和提取数据,则最好使用HTML解析器,如 Jsoup




相关问题推荐