问题现象
后台服务日志,大量报出如下异常,关键字:java.lang.OutOfMemoryError: Java heap space
,问题指向:o.a.c.h.Http11NioProtocol [DirectJDKLog.java:175] Failed to complete processing of a request
问题定位
线上OOM后,dump内存快照,通过MAT工具进行分析,如下图所示,有多个200MB的Http11OutputBuffer
对象存在,这很不合理。
排查相关配置后发现,server.max-http-header-size
被设置为200MB。
问题解决
将server.max-http-header-size
调整为1MB
即可。
问题分析
Tomcat在构建Http11OutputBuffer
对象时会通过maxHttpHeaderSize
来设置HTTP message header的大小,其默认为8KB,并通过ByteBuffer.allocate(headerBufferSize)
向内存申请,也就是说每接收到一个HTTP请求,都会向内存申请200MB的空间,所以,当请求并发上来以后,必然会导致OOM。
源码
org.apache.coyote.http11.AbstractHttp11Protocol
类,构建Processor
对象
@Override
protected Processor createProcessor() {
Http11Processor processor = new Http11Processor(this, adapter);
return processor;
}
Http11Processor
构建
public Http11Processor(AbstractHttp11Protocol<?> protocol, Adapter adapter) {
super(adapter);
this.protocol = protocol;
httpParser = new HttpParser(protocol.getRelaxedPathChars(),
protocol.getRelaxedQueryChars());
inputBuffer = new Http11InputBuffer(request, protocol.getMaxHttpHeaderSize(),
protocol.getRejectIllegalHeader(), httpParser);
request.setInputBuffer(inputBuffer);
// getMaxHttpHeaderSize默认为8KB,也可以通过server.max-http-header-size进行修改
outputBuffer = new Http11OutputBuffer(response, protocol.getMaxHttpHeaderSize());
response.setOutputBuffer(outputBuffer);
// Create and add the identity filters.
inputBuffer.addFilter(new IdentityInputFilter(protocol.getMaxSwallowSize()));
outputBuffer.addFilter(new IdentityOutputFilter());
// Create and add the chunked filters.
inputBuffer.addFilter(new ChunkedInputFilter(protocol.getMaxTrailerSize(),
protocol.getAllowedTrailerHeadersInternal(), protocol.getMaxExtensionSize(),
protocol.getMaxSwallowSize()));
outputBuffer.addFilter(new ChunkedOutputFilter());
// Create and add the void filters.
inputBuffer.addFilter(new VoidInputFilter());
outputBuffer.addFilter(new VoidOutputFilter());
// Create and add buffered input filter
inputBuffer.addFilter(new BufferedInputFilter());
// Create and add the gzip filters.
//inputBuffer.addFilter(new GzipInputFilter());
outputBuffer.addFilter(new GzipOutputFilter());
pluggableFilterIndex = inputBuffer.getFilters().length;
}
protected Http11OutputBuffer(Response response, int headerBufferSize) {
this.response = response;
// 内存申请
headerBuffer = ByteBuffer.allocate(headerBufferSize);
filterLibrary = new OutputFilter[0];
activeFilters = new OutputFilter[0];
lastActiveFilter = -1;
responseFinished = false;
outputStreamOutputBuffer = new SocketOutputBuffer();
}