JAVACPU占用过高、内存泄漏问题排查

news2024/11/20 6:16:24

📢📢📢📣📣📣
哈喽！大家好，我是「奇点」，江湖人称 singularity。刚工作几年，想和大家一同进步🤝🤝
一位上进心十足的【Java ToB端大厂领域博主】！😜😜😜
喜欢java和python，平时比较懒，能用程序解决的坚决不手动解决😜😜😜

✨ 如果有对【java】感兴趣的【小可爱】，欢迎关注我

❤️❤️❤️感谢各位大可爱小可爱！❤️❤️❤️
————————————————

如果觉得本文对你有帮助，欢迎点赞，欢迎关注我，如果有补充欢迎评论交流，我将努力创作更多更好的文章。
————————————————
版权声明：本文为CSDN博主「HiSiri666666」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/qq_29235677/article/details/126655356

问题起因：

java工程消费的内容延迟严重，发现负责消费的dispatcher 进程阻塞，且占用的CPU%很高。

开始按照https://www.cnblogs.com/paul8339/p/7464206.html 文中提到的方法排查。

一、追查

重点：

如果java进程在docker下执行，需要进入docker里边执行下面的步骤，否则会报错：

Unable to open socket file: target process not responding or HotSpot VM not loaded

第一步：先获取到需要排查的进程PID

查询前十个消耗CPU做多的进程命令：

ps aux|head -1;ps aux|grep -v PID|sort -rn -k +3|head

获取java进程的资源消费情况：

ps -ef | grep java

第二步：获取线程信息，找到耗CPU的线程

ps -mp pid -o THREAD,tid,time | sort -rn 6

第三步：打印线程的堆栈信息

先将需要的线程ID转换为16进制格式

printf "%x\n" tid

jstack pid |grep tid -A 30

jstack 1024 |grep tid -A 30

踩坑1:

我们的工程运行在docker环境下，需要进入docker下执行前两步，否则在执行jstack的时候报错：

Error attaching to process: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
at sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:411)
at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)
at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
at sun.tools.jstack.JStack.main(JStack.java:106)

解决办法：

执行 docker ps -a 找到需要进入的docker CONTAINER ID

执行 docker exec -it CONTAINER ID /bin/bash 进入docker

踩坑2:

linux没有默认开启trace debug功能，执行jstack的时候提示：

Can't attach to the process

解决办法：

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

参考https://www.cnblogs.com/wscit/p/6803354.html

执行jstack后发现好多线程都是BLOCKED状态，堆栈信息如下：

Thread 171: (state = BLOCKED)

- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)

- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)

- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)

- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)

- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1074 (Compiled frame)

- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Compiled frame)

- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)

- java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)

--

- org.apache.commons.httpclient.HttpMethodBase.readStatusLine(org.apache.commons.httpclient.HttpState, org.apache.commons.httpclient.HttpConnection) @bci=36, line=1973 (Compiled frame)

- org.apache.commons.httpclient.HttpMethodBase.readResponse(org.apache.commons.httpclient.HttpState, org.apache.commons.httpclient.HttpConnection) @bci=21, line=1735 (Compiled frame)

- org.apache.commons.httpclient.HttpMethodBase.execute(org.apache.commons.httpclient.HttpState, org.apache.commons.httpclient.HttpConnection) @bci=68, line=1098 (Compiled frame)

- org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(org.apache.commons.httpclient.HttpMethod) @bci=135, line=398 (Compiled frame)

- org.apache.commons.httpclient.HttpMethodDirector.executeMethod(org.apache.commons.httpclient.HttpMethod) @bci=288, line=171 (Compiled frame)

- org.apache.commons.httpclient.HttpClient.executeMethod(org.apache.commons.httpclient.HostConfiguration, org.apache.commons.httpclient.HttpMethod, org.apache.commons.httpclient.HttpState) @bci=114, line=397 (Compiled frame)

- org.apache.commons.httpclient.HttpClient.executeMethod(org.apache.commons.httpclient.HttpMethod) @bci=14, line=323 (Compiled frame)

- com.weibo.qa.util.RequestUtil.execHttpMethod(org.apache.commons.httpclient.HttpMethod, int) @bci=38, line=361 (Compiled frame)

- com.weibo.qa.util.RequestUtil.getHttpRequest(java.util.Map, java.util.Map, java.lang.String, int) @bci=18, line=319 (Compiled frame)

- com.weibo.qa.util.RequestUtil.getHttpRequest(java.util.Map, java.util.Map, java.lang.String) @bci=4, line=340 (Compiled frame)

- com.weibo.qa.util.RequestUtil.getHttpRequest(java.util.Map, java.lang.String) @bci=11, line=272 (Compiled frame)

- com.weibo.qa.terminator.service.FriendService.getUserInforBatch(java.lang.Long[]) @bci=255, line=126 (Compiled frame)

二、分析

分析上面的信息可以看到，是调用了HttpClient之后线程锁死。

看工程的log日志，发现较多的warning：

WARN org.apache.commons.httpclient.HttpMethodBase - Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.

参考https://blog.csdn.net/xiaoshuji/article/details/71077619

提到HttpClient建议使用InputStream getResponseBodyAsStream()代替byte[]getResponseBody()。对于返回结果很大或无法预知的情况，就需要使用InputStreamgetResponseBodyAsStream()，避免byte[] getResponseBody()可能带来的内存的耗尽问题。