问题描述
线上部署的代码，部署在测试环境突然抛出接口请求Read time out的异常。查看线上日志，接口请求正常。重新启动，部署测试环境代码都没有效果，接口还是必现Read time out。
原因分析：
1. 排查网络原因
直接在测试环境服务器上执行使用curl命令执行http请求，接口正常返回，所以排查网络，第三方服务的原因。
2. 网络抓包
解决这类问题，最简单的方法就是抓一个正常的http请求包，抓一个Read time out异常的http请求包，对比两个的报文。但是事与愿违，容器环境禁用tcpdump命令，也就抓不了包。
3. 本地代码debug
启动本地代码连接测试环境配置，再出发相同的请求，接口返回正常。
4. 调整日志等级
项目统一使用spring cloud体系微服务架构，将日志等级该为debug，日志会打印一些http请求信息。观查到报错前的http请求日志如下：
2022-12-05 09:47:21.209 DEBUG 1 --- [http-nio-8080-exec-44] s.n.www.protocol.http.HttpURLConnection : [TxId : xxx-1^1670143309480^7363 , SpanId : 1300305726613613085] sun.net.www.MessageHeader@11ef701319 pairs:
{GET /xxxx/audiorid?recordId=9776d4f1-8b89-488a-b00c-cd430f39c003&recordDate=1670146625251 HTTP/1.1: null}
{Accept: application/octet-stream, application/json, application/*+json, */*}
{System-Id: xxxapi}
{sign: xxxxxxxxxxxx}
{Sign-Type: RSA}
{Sign-Timestamp: 1670204841194}
{Content-Type: application/x-www-form-urlencoded}
{Content-Length: 135}
{request-source-app: xxxapi}
{Pinpoint-TraceID: xxx-1^1670143309480^7363}
{Pinpoint-SpanID: 3740963636577534541}
{Pinpoint-pSpanID: 1300305726613613085}
{Pinpoint-Flags: 0}{Pinpoint-pAppName: xxxapi}
{Pinpoint-pAppType: 1210}
{Pinpoint-Host: xxx.com}
{User-Agent: Java/1.8.0_231}
{Host: xxxxx.com}
{Connection: keep-alive}
因为本接口Method是GET，Content-Length应该为0，但是日志缺显示为135
5. 排查代码
http代码是基于restTemplate封装抽象统一实现的，以下是执行get请求的抽象方法：
public String executeGet(String systemName, String requestUrl, Map<String,Object> params, Map<String,Object> headerParams){
HttpEntity<MutiValueMap<String, String>> entity = RestHttpUtils.getEntity(params, headerParams);
return execute(systemName, requestUrl, HttpMethod.GET, entity, params);
}
其中装换entity有点问题
public static HttpEntity<MultiValueMap<String,String>> getEntity(Map<String,Object> params, Map<String,Object> headerParams){
MultiValueMap<String,String> mvm = getMultiValueMap(params);
HttpHeaders headers = getHeaders(headersParams);
return new HttpEntity<MultiValueMap<String,String>>(mvm, headers);
}
在HttpEntity中，param被赋值给body，get请求中的param应该是queryParam，不应该在body中。现在赋值到Body，那么Content-Length就会有值。
public HttpEntity(@Nullable T body, @Nullable MultiValueMap<String, String> headers) {
this.body = body;
this.headers = HttpHeaders.readOnlyHttpHeaders(headers != null ? headers : new HttpHeaders());
}
Content-Length在restTemplate#doExecute中进行赋值
protected <T> T doExecute(URI url, @Nullable HttpMethod method, @Nullable RequestCallback requestCallback,
@Nullable ResponseExtractor<T> responseExtractor) throws RestClientException {
Assert.notNull(url, "URI is required");
Assert.notNull(method, "HttpMethod is required");
ClientHttpResponse response = null;
try {
ClientHttpRequest request = createRequest(url, method);
if (requestCallback != null) {
requestCallback.doWithRequest(request);
}
response = request.execute();
handleResponse(url, method, response);
return (responseExtractor != null ? responseExtractor.extractData(response) : null);
}
catch (IOException ex) {
String resource = url.toString();
String query = url.getRawQuery();
resource = (query != null ? resource.substring(0, resource.indexOf('?')) : resource);
throw new ResourceAccessException("I/O error on " + method.name() +
" request for \"" + resource + "\": " + ex.getMessage(), ex);
}
finally {
if (response != null) {
response.close();
}
}
}
具体见requestCallback.doWithRequest(request)，有body有值，所以Content-Length 没有赋值为0，而是在converter中序列化body并赋值。
public void doWithRequest(ClientHttpRequest httpRequest) throws IOException {
super.doWithRequest(httpRequest);
Object requestBody = this.requestEntity.getBody();
if (requestBody == null) {
HttpHeaders httpHeaders = httpRequest.getHeaders();
HttpHeaders requestHeaders = this.requestEntity.getHeaders();
if (!requestHeaders.isEmpty()) {
requestHeaders.forEach((key, values) -> httpHeaders.put(key, new ArrayList<>(values)));
}
if (httpHeaders.getContentLength() < 0) {
httpHeaders.setContentLength(0L);
}
}
else {
Class<?> requestBodyClass = requestBody.getClass();
Type requestBodyType = (this.requestEntity instanceof RequestEntity ?
((RequestEntity<?>)this.requestEntity).getType() : requestBodyClass);
HttpHeaders httpHeaders = httpRequest.getHeaders();
HttpHeaders requestHeaders = this.requestEntity.getHeaders();
MediaType requestContentType = requestHeaders.getContentType();
for (HttpMessageConverter<?> messageConverter : getMessageConverters()) {
if (messageConverter instanceof GenericHttpMessageConverter) {
GenericHttpMessageConverter<Object> genericConverter =
(GenericHttpMessageConverter<Object>) messageConverter;
if (genericConverter.canWrite(requestBodyType, requestBodyClass, requestContentType)) {
if (!requestHeaders.isEmpty()) {
requestHeaders.forEach((key, values) -> httpHeaders.put(key, new ArrayList<>(values)));
}
logBody(requestBody, requestContentType, genericConverter);
genericConverter.write(requestBody, requestBodyType, requestContentType, httpRequest);
return;
}
}
else if (messageConverter.canWrite(requestBodyClass, requestContentType)) {
if (!requestHeaders.isEmpty()) {
requestHeaders.forEach((key, values) -> httpHeaders.put(key, new ArrayList<>(values)));
}
logBody(requestBody, requestContentType, messageConverter);
((HttpMessageConverter<Object>) messageConverter).write(
requestBody, requestContentType, httpRequest);
return;
}
}
String message = "No HttpMessageConverter for " + requestBodyClass.getName();
if (requestContentType != null) {
message += " and content type \"" + requestContentType + "\"";
}
throw new RestClientException(message);
}
}
body的数据类型是MultiValueMap<String,String，对应的converter是AllEncompassingFormHttpMessageConverter
public void write(MultiValueMap<String, ?> map, @Nullable MediaType contentType, HttpOutputMessage outputMessage)
throws IOException, HttpMessageNotWritableException {
if (isMultipart(map, contentType)) {
writeMultipart((MultiValueMap<String, Object>) map, contentType, outputMessage);
}
else {
writeForm((MultiValueMap<String, Object>) map, contentType, outputMessage);
}
}
private void writeForm(MultiValueMap<String, Object> formData, @Nullable MediaType contentType,
HttpOutputMessage outputMessage) throws IOException {
contentType = getFormContentType(contentType);
outputMessage.getHeaders().setContentType(contentType);
Charset charset = contentType.getCharset();
Assert.notNull(charset, "No charset"); // should never occur
byte[] bytes = serializeForm(formData, charset).getBytes(charset);
outputMessage.getHeaders().setContentLength(bytes.length);
if (outputMessage instanceof StreamingHttpOutputMessage) {
StreamingHttpOutputMessage streamingOutputMessage = (StreamingHttpOutputMessage) outputMessage;
streamingOutputMessage.setBody(outputStream -> StreamUtils.copy(bytes, outputStream));
}
else {
StreamUtils.copy(bytes, outputMessage.getBody());
}
}
虽然在这里进行赋值，但是代码一直以来缺正常运行。继续跟踪restTemplate代码。restTemplate第一层使用的是InterceptingClientHttpRequest，使用pipeline的模式，支持对http请求拓展。
protected final ClientHttpResponse executeInternal(HttpHeaders headers, byte[] bufferedOutput) throws IOException {
InterceptingRequestExecution requestExecution = new InterceptingRequestExecution();
return requestExecution.execute(this, bufferedOutput);
}
private class InterceptingRequestExecution implements ClientHttpRequestExecution {
private final Iterator<ClientHttpRequestInterceptor> iterator;
public InterceptingRequestExecution() {
this.iterator = interceptors.iterator();
}
@Override
public ClientHttpResponse execute(HttpRequest request, byte[] body) throws IOException {
if (this.iterator.hasNext()) {
ClientHttpRequestInterceptor nextInterceptor = this.iterator.next();
return nextInterceptor.intercept(request, body, this);
}
else {
HttpMethod method = request.getMethod();
Assert.state(method != null, "No standard HTTP method");
ClientHttpRequest delegate = requestFactory.createRequest(request.getURI(), method);
request.getHeaders().forEach((key, value) -> delegate.getHeaders().addAll(key, value));
if (body.length > 0) {
if (delegate instanceof StreamingHttpOutputMessage) {
StreamingHttpOutputMessage streamingOutputMessage = (StreamingHttpOutputMessage) delegate;
streamingOutputMessage.setBody(outputStream -> StreamUtils.copy(body, outputStream));
}
else {
StreamUtils.copy(body, delegate.getBody());
}
}
return delegate.execute();
}
}
}
真正执行http请求的是SimpleBufferingClientHttpRequest
@Override
protected ClientHttpResponse executeInternal(HttpHeaders headers, byte[] bufferedOutput) throws IOException {
addHeaders(this.connection, headers);
// JDK <1.8 doesn't support getOutputStream with HTTP DELETE
if (getMethod() == HttpMethod.DELETE && bufferedOutput.length == 0) {
this.connection.setDoOutput(false);
}
if (this.connection.getDoOutput() && this.outputStreaming) {
this.connection.setFixedLengthStreamingMode(bufferedOutput.length);
}
this.connection.connect();
if (this.connection.getDoOutput()) {
FileCopyUtils.copy(bufferedOutput, this.connection.getOutputStream());
}
else {
// Immediately trigger the request in a no-output scenario as well
this.connection.getResponseCode();
}
return new SimpleClientHttpResponse(this.connection);
}
static void addHeaders(HttpURLConnection connection, HttpHeaders headers) {
String method = connection.getRequestMethod();
if (method.equals("PUT") || method.equals("DELETE")) {
if (!StringUtils.hasText(headers.getFirst(HttpHeaders.ACCEPT))) {
// Avoid "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2"
// from HttpUrlConnection which prevents JSON error response details.
headers.set(HttpHeaders.ACCEPT, "*/*");
}
}
headers.forEach((headerName, headerValues) -> {
if (HttpHeaders.COOKIE.equalsIgnoreCase(headerName)) { // RFC 6265
String headerValue = StringUtils.collectionToDelimitedString(headerValues, "; ");
connection.setRequestProperty(headerName, headerValue);
}
else {
for (String headerValue : headerValues) {
String actualHeaderValue = headerValue != null ? headerValue : "";
connection.addRequestProperty(headerName, actualHeaderValue);
}
}
});
}
其中addHeaders(this.connection, headers)就是对header进行处理的地方，这里的connection就是HttpURLConnection
public synchronized void addRequestProperty(String var1, String var2) {
if (!this.connected && !this.connecting) {
if (var1 == null) {
throw new NullPointerException("key is null");
} else {
if (this.isExternalMessageHeaderAllowed(var1, var2)) {
this.requests.add(var1, var2);
if (!var1.equalsIgnoreCase("Content-Type")) {
this.userHeaders.add(var1, var2);
}
}
}
} else {
throw new IllegalStateException("Already connected");
}
}
private boolean isExternalMessageHeaderAllowed(String var1, String var2) {
this.checkMessageHeader(var1, var2);
return !this.isRestrictedHeader(var1, var2);
}
private boolean isRestrictedHeader(String var1, String var2) {
if (allowRestrictedHeaders) {
return false;
} else {
var1 = var1.toLowerCase();
if (restrictedHeaderSet.contains(var1)) {
return !var1.equals("connection") || !var2.equalsIgnoreCase("close");
} else {
return var1.startsWith("sec-");
}
}
}
注意这里有个isExternalMessageHeaderAllowed判断，条件成立的情况下，才往this.requests中添加header。
static {
String var0 = (String)AccessController.doPrivileged(new GetPropertyAction("http.agent"));
if (var0 == null) {
var0 = "Java/" + version;
} else {
var0 = var0 + " Java/" + version;
}
userAgent = var0;
String var1 = getNetProperty("jdk.http.auth.tunneling.disabledSchemes");
disabledTunnelingSchemes = schemesListToSet(var1);
var1 = getNetProperty("jdk.http.auth.proxying.disabledSchemes");
disabledProxyingSchemes = schemesListToSet(var1);
validateProxy = (Boolean)AccessController.doPrivileged(new GetBooleanAction("http.auth.digest.validateProxy"));
validateServer = (Boolean)AccessController.doPrivileged(new GetBooleanAction("http.auth.digest.validateServer"));
enableESBuffer = (Boolean)AccessController.doPrivileged(new GetBooleanAction("sun.net.http.errorstream.enableBuffering"));
timeout4ESBuffer = (Integer)AccessController.doPrivileged(new GetIntegerAction("sun.net.http.errorstream.timeout", 300));
if (timeout4ESBuffer <= 0) {
timeout4ESBuffer = 300;
}
bufSize4ES = (Integer)AccessController.doPrivileged(new GetIntegerAction("sun.net.http.errorstream.bufferSize", 4096));
if (bufSize4ES <= 0) {
bufSize4ES = 4096;
}
allowRestrictedHeaders = (Boolean)AccessController.doPrivileged(new GetBooleanAction("sun.net.http.allowRestrictedHeaders"));
if (!allowRestrictedHeaders) {
restrictedHeaderSet = new HashSet(restrictedHeaders.length);
for(int var2 = 0; var2 < restrictedHeaders.length; ++var2) {
restrictedHeaderSet.add(restrictedHeaders[var2].toLowerCase());
}
} else {
restrictedHeaderSet = null;
}
EXCLUDE_HEADERS = new String[]{"Proxy-Authorization", "Authorization"};
EXCLUDE_HEADERS2 = new String[]{"Proxy-Authorization", "Authorization", "Cookie", "Cookie2"};
logger = PlatformLogger.getLogger("sun.net.www.protocol.http.HttpURLConnection");
}
最为关键的就是allowRestrictedHeaders根据sun.net.http.allowRestrictedHeaders参数读取的，一开始在pinpoint上面看应用的参数没有设置这个值，所以又浪费了一些时间，最终使用jinfo命令看到测试环境这个配置项为true，而开发环境是false。
解决方案：
简单的解决方法就是启动的时候将sun.net.http.allowRestrictedHeaders配置为false，后面将queryParam赋值给body的代码去掉。
![[附源码]Python计算机毕业设计SSM健身房管理系统设计（程序+LW)](https://img-blog.csdnimg.cn/8cf92118c2ed464590bbf9236fbe8383.png)
![[附源码]Python计算机毕业设计Django体育馆场地预约管理系统](https://img-blog.csdnimg.cn/278d262b05b445e8a20b9c8af3dcbe77.png)
![[Cortex-M3]-3-分散加载文件解析（.sct）](https://img-blog.csdnimg.cn/b349d72b35d14e31bd6cc36ab99e80cd.png)
