目录

基本信息

问题现象

问题原因

排查过程

解决方案

---

• 基本信息

  客户名称：xx集团

  产品名称：ATS 3.0微服务单体
  版本号：3.1.14.X
  问题分类：环境问题

  环境类型：K8S

• 问题现象

        单体组织计划的新增/修改功能，多次点击，有时能成功，有时会报失败，失败的时候报"网络请求发生错误！"。其他的功能都没问题。
在白天客户使用系统高峰期，失败的次数会很多，晚上客户不用系统的时候，报错会减少，甚至不报错。

接口报网络问题的时候，经观察也有几种现象：

一种是保存按钮点完马上报出来网络异常，

一种是点完按钮会一直等待，像是超时后再报的网络异常。

• 问题原因

        k8s内部网络问题，导致不同node之间网络通信问题，ui和ingress的nginx服务器报502，反应到页面上就是"网络请求发生错误！。

• 排查过程

        f12查看报错时候的接口返回，发现马上报错的接口返回也有不同，有的是1.14.2版本nginx返回的，有的是1.19.1版本返回的。这两个版本分别是ui和ingress的nginx版本，所以报错位置也有不同。

        1. 根据日志分析

         看ui的nginx的日志，发现确实有502的日志，

看gateway的日志，并没有发现报错，都正常转发出去了请求
看budget的日志，也没有发现报错
因为这个接口用了特殊的sheet组件，发请求的数据量较大，所以猜测是因为nginx配置问题。搜104的解决方案，加上了这些配置：

client_body_buffer_size 50m;
client_header_timeout 120;
client_body_timeout 120;

发现还是不行。

然后猜测是budget处理该请求时间过长，导致budget资源用光，然后请求后打jstack，马上报错的情况，budget没有处理请求的线程。当请求卡住的时候，发现budget中有线程卡在了spring mvc的getParamter方法上。

        因为金地的部署都是集群部署，每个服务都有两个pod，为了排查方便，晚上客户不用的时候，把ui，gateway，budget都改成了一台。然后再次追踪日志，发现页面上直接报错的请求没走到gateway，直接在ui里就返回了。

        根据上面的观察，猜测可能是k8s的网络问题导致的，如果发的包不完整可能会导致一些参数没传完，也就是budget一直卡在获取参数的代码。如果nginx直接网络出问题，就可能不会再转发到gateway了。这种情况只能抓包观察了

         2. 抓包分析

容器中抓包的方法

1.进入容器bash

cat /sys/class/net/eth0/iflink

2.到容器对应的node中，根据上面的到的网卡号获取网卡

ip a | grep 50

找以50开头的，后面@之前的这一串就是这个pod的网卡。

3.拿到该容器的网卡编号可以对该网卡进行抓包

nohup tcpdump -i cali894e62c8009 -w pod-xx.cap &

用curl发请求

这里介绍一个技巧，将f12中请求复制为curl命令，可以到pod中执行。实现的效果是使用域名或pod内部ip的方式从一个pod发到目标的pod。
例如：在bankaccount的pod中发请求，发给ingress或者ui

拿到的脚本：

curl 'https://ats.gemdale.com/api?uc=UC_BUDGET_BU_SINGLEREDACT_EDIT' \
 -H 'Connection: keep-alive' \
 -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"' \
 -H 'luna-operation-menu-code: ms_budgetSingleOrganization' \
 -H 'current-user-industry-code: default' \
 -H 'FrontBackInteractionEncryptUniqueId: f85b7c59b1a349438fbb5767b1ca2472' \
 -H 'current-user-tenant-code: ATS' \
-H 'Authorization: auth:access:4863A38BBA514569CD4D4611B05977ED$$E8F88B188700BE9B971763882CCB674C$$2f2b512247' \
 -H 'Content-Type: application/x-www-form-urlencoded' \
 -H 'Accept: application/json, text/plain, /' \
 -H 'sec-ch-ua-mobile: ?0' \
 -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36' \
 -H 'sec-ch-ua-platform: "Windows"' \
 -H 'Origin: https://ats.gemdale.com' \
 -H 'Sec-Fetch-Site: same-origin' \
 -H 'Sec-Fetch-Mode: cors' \
 -H 'Sec-Fetch-Dest: empty' \
 -H 'Referer: https://ats.gemdale.com/home' \
 -H 'Accept-Language: zh-CN,zh;q=0.9' \
 -H 'Cookie: a_u_k=8BA59D8C6F232A9132A125FAB1C2814A9491477598102D1EAD05E270F4DBFE0FBBD33A07E702513EEB186DFB5B5F951D46388608105E31E22CF0D58B7662A7B00C6B0D44167F73C3CCF61B94C2D63BEF35533367E85F10A259C0FFB8B586C810F6A280671D756E5DFD68AC7500A62D3E6BAB847815713810E96E37F69101BDF5CFA3ECB78F67E9BCFE6ED9BB71B5C248CBF1DB684C6945005DC9970458E7EC6C8240B12FD497CB2D091280DBAFC00D35907C3A1F499D8C518FF50DFF01401BC23F6F63FA3CD3DB722839A919F105B6A985CD8DCAE8CC6E5472ED960C7D5325736ABA53E5C588F79F48E6FFB94044E50E555F0B2A95A84BD0DAAE16898B30221E89C192991D4CE8E8E23E7AC599F001E058535E871D86E28102E500F0A9318AE4DE6BC2A34376FF58FBC6A6BA694891D0; JSESSIONID=49AAD0B4D00AFB3C54750B391760E487' \
 -data-raw 'urid=575241ccd16c4176af8f7cb7c2a55cd0&orgid=6000001&person=ATS%E7%AE%A1%E7%90%86%E5%91%98&unit=1&currencyid=1&projectitemid=&budgetcalendarid=ad56bf5fd9d14438bf611b1e317df44a&istrade=1&isconsolidate=0&code=1&description=&excelData=%5B%5B%22%E8%AE%A1%E5%88%92%E9%A1%B9%E7%9B%AE%22%2C%222022%E5%B9%B4%E7%AC%AC6%E6%9C%88%22%2C%22%22%2C%22%22%2C%22%22%2C%222022%E5%B9%B4%E7%AC%AC7%E6%9C%88%22%2C%222022%E5%B9%B4%E7%AC%AC8%E6%9C%88%22%2C%22%E5%90%88%E8%AE%A1%22%5D%2C%5B%22%22%2C%22%E4%B8%8A%E6%9C%9F%E5%AE%9E%E9%99%85%E6%95%B0%22%2C%22%E4%B8%8A%E6%9C%9F%E8%AE%A1%E5%88%92%E5%80%BC%22%2C%22%E4%B8%8A%E6%9C%9F%E5%AE%9E%E9%99%85%E5%80%BC%28%E6%89%8B%E5%B7%A5%29%22%2C%22%E8%AE%A1%E5%88%92%E5%80%BC%22%2C%22%E8%AE%A1%E5%88%92%E5%80%BC%22%2C%22%E8%AE%A1%E5%88%92%E5%80%BC%22%2C%22%22%5D%2C%5B%22527f5912131245769b286247b2b04291%22%2C%22%22%2C%22-%22%2C%22-%22%2C%220.00%22%2C%22-31.00%22%2C%22-13.00%22%2C%22-%22%5D%2C%5B%221%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%221.00%22%2C%220.00%22%2C%220.00%22%2C%221.00%22%5D%2C%5B%221.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%221.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%221.00%22%2C%220.00%22%2C%220.00%22%2C%221.00%22%5D%2C%5B%222%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%22-27.00%22%2C%2218.00%22%2C%2298.00%22%2C%2289.00%22%5D%2C%5B%222.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%2228.00%22%2C%2228.00%22%2C%22105.00%22%2C%22161.00%22%5D%2C%5B%222.1.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%221.00%22%2C%221.00%22%2C%2212.00%22%2C%2214.00%22%5D%2C%5B%222.1.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%222.00%22%2C%222.00%22%2C%2213.00%22%2C%2217.00%22%5D%2C%5B%222.1.3%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%223.00%22%2C%223.00%22%2C%2214.00%22%2C%2220.00%22%5D%2C%5B%222.1.4%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%224.00%22%2C%224.00%22%2C%2215.00%22%2C%2223.00%22%5D%2C%5B%222.1.5%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%225.00%22%2C%225.00%22%2C%2216.00%22%2C%2226.00%22%5D%2C%5B%222.1.6%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%226.00%22%2C%226.00%22%2C%2217.00%22%2C%2229.00%22%5D%2C%5B%222.1.7%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%227.00%22%2C%227.00%22%2C%2218.00%22%2C%2232.00%22%5D%2C%5B%222.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%22-55.00%22%2C%22-10.00%22%2C%22-7.00%22%2C%22-72.00%22%5D%2C%5B%222.2.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%221.00%22%2C%220.00%22%2C%220.00%22%2C%221.00%22%5D%2C%5B%222.2.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%222.00%22%2C%220.00%22%2C%223.00%22%2C%225.00%22%5D%2C%5B%222.2.3%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%223.00%22%2C%220.00%22%2C%224.00%22%2C%227.00%22%5D%2C%5B%222.2.4%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%224.00%22%2C%220.00%22%2C%220.00%22%2C%224.00%22%5D%2C%5B%222.2.5%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%225.00%22%2C%220.00%22%2C%220.00%22%2C%225.00%22%5D%2C%5B%222.2.6%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%226.00%22%2C%220.00%22%2C%220.00%22%2C%226.00%22%5D%2C%5B%222.2.7%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%227.00%22%2C%225.00%22%2C%220.00%22%2C%2212.00%22%5D%2C%5B%222.2.8%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%228.00%22%2C%220.00%22%2C%220.00%22%2C%228.00%22%5D%2C%5B%222.2.9%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%229.00%22%2C%225.00%22%2C%220.00%22%2C%2214.00%22%5D%2C%5B%222.2.10%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%2210.00%22%2C%220.00%22%2C%220.00%22%2C%2210.00%22%5D%2C%5B%223%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%22-8.00%22%2C%22-4.00%22%2C%220.00%22%2C%22-12.00%22%5D%2C%5B%223.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%221.00%22%2C%225.00%22%2C%225.00%22%2C%2211.00%22%5D%2C%5B%223.1.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%223.1.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%221.00%22%2C%220.00%22%2C%225.00%22%2C%226.00%22%5D%2C%5B%223.1.3%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%225.00%22%2C%220.00%22%2C%225.00%22%5D%2C%5B%223.1.4%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%223.1.5%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%223.1.6%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%223.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%22-9.00%22%2C%22-9.00%22%2C%22-5.00%22%2C%22-23.00%22%5D%2C%5B%223.2.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%223.2.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%225.00%22%2C%220.00%22%2C%220.00%22%2C%225.00%22%5D%2C%5B%223.2.3%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%225.00%22%2C%225.00%22%2C%2210.00%22%5D%2C%5B%223.2.4%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%223.2.5%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%224.00%22%2C%224.00%22%2C%220.00%22%2C%228.00%22%5D%2C%5B%223.2.6%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%223.2.7%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%223.2.8%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%223.2.9%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%224%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%224.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%224.00%22%2C%224.00%22%2C%224.00%22%2C%2212.00%22%5D%2C%5B%224.1.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%224.1.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%224.00%22%2C%224.00%22%2C%220.00%22%2C%228.00%22%5D%2C%5B%224.1.3%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%224.00%22%2C%224.00%22%5D%2C%5B%224.1.4%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%224.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%224.00%22%2C%224.00%22%2C%224.00%22%2C%2212.00%22%5D%2C%5B%224.2.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%224.2.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%224.2.3%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%224.00%22%2C%220.00%22%2C%224.00%22%2C%228.00%22%5D%2C%5B%224.2.4%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%224.00%22%2C%220.00%22%2C%224.00%22%5D%2C%5B%224.2.5%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%224.2.6%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%225%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%224.00%22%2C%224.00%22%2C%224.00%22%2C%2212.00%22%5D%2C%5B%225.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%224.00%22%2C%224.00%22%2C%224.00%22%2C%2212.00%22%5D%2C%5B%225.1.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%225.1.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%224.00%22%2C%224.00%22%2C%228.00%22%5D%2C%5B%225.1.3%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%224.00%22%2C%220.00%22%2C%220.00%22%2C%224.00%22%5D%2C%5B%225.1.4%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%225.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%225.2.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%225.2.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%225.2.3%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%225.2.4%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%225.2.5%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%226%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%226.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%226.1.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%226.1.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%226.1.3%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%226.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%226.2.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%226.2.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%226.2.3%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%227%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%227.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%227.1.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%227.1.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%227.1.3%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%227.1.4%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%227.1.5%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%227.1.6%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%227.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%227.2.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%227.2.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%227.2.3%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%227.2.4%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%228%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%22-30.00%22%2C%2218.00%22%2C%22102.00%22%2C%2290.00%22%5D%2C%5B%228.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%22-31.00%22%2C%2218.00%22%2C%22102.00%22%2C%2289.00%22%5D%2C%5B%228.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%221.00%22%2C%220.00%22%2C%220.00%22%2C%221.00%22%5D%2C%5B%228.1.1%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%228.1.2%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%229%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%228.00%22%2C%224.00%22%2C%220.00%22%2C%2212.00%22%5D%2C%5B%2210%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%22-23.00%22%2C%2222.00%22%2C%22102.00%22%2C%22101.00%22%5D%2C%5B%2211%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%2231.00%22%2C%22-18.00%22%2C%22-102.00%22%2C%22-89.00%22%5D%2C%5B%2291%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%2292%22%2C%220.00%22%2C%22-%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%2C%220.00%22%5D%2C%5B%223f1c9999d2de4df28ffda0f38fb89bb7%22%2C%22-%22%2C%22-%22%2C%22-%22%2C%22-31.00%22%2C%22-13.00%22%2C%2289.00%22%2C%22-%22%5D%5D&catalogname=T_BD_BUDGET&attachmentIds=&templateid=fbe7aa88acf445f2a7e446f3272c0cff&budgetorgid=6000001&beginbalance=0.00&endbalance=-31.00&calendarid=ad56bf5fd9d14438bf611b1e317df44a&budgetDetailDatas=&deptid=&isorgbudget=1&isSendApprove=0&currencybeginbalance=0&draftsbeginbalance=0&editstartcalendarid=ad56bf5fd9d14438bf611b1e317df44a&editendcalendarid=79671f6d6d774576a077e08d8098390a&editstartdate=2022-06-01%2000%3A00%3A00.0&editenddate=2022-08-31%2000%3A00%3A00.0&controlstartcalendarid=ad56bf5fd9d14438bf611b1e317df44a&controlendcalendarid=ad56bf5fd9d14438bf611b1e317df44a&controlstartdate=2022-06-01%2000%3A00%3A00.0&controlenddate=2022-06-30%2000%3A00%3A00.0&virtualtype=&budgettitles=' \
 --compressed

可以通过更改curl后面的域名地址，发给ui，curl '  http://ui/api?uc=UC_BUDGET_BU_SINGLEREDACT_EDIT​http://ui/api?uc=UC_BUDGET_BU_SINGLEREDACT_EDIT

抓包结果分析

第一次抓包

        通过上面curl的方式发送请求，在bankaccount的pod中分别发给ingress和ui发请求。
每个pod的ip分别是：

ingress:10.36.13.205
ui:192.168.40.70
gateway:192.168.159.255
budget:192.168.159.249
发送pod：192.168.159.246

每个pod的节点部署情况是：

        对ui,gateway,budget抓包，在bankaccount上对ingress发请求，发了5次，记录成功失败时间，有一次失败时间是16:49:00。查看抓包结果：

ui：

        看到ingress发给ui的http包已经收到了，10.105.185.4是gateway集群的ip，192.168.40.70发给10.105.185.4的是ui发给gateway的。然后收到了一个RST。后面就是gateway回复给ingress [10.36.13.205], ingress在返回给ui[192.168.40.70]的包，502 BadGateway。

gateway:

gateway也只收到了两个ui发过来的包。没凑成一个完整的http包。

        整个过程就是 ingress发送给ui一个http包用了3个TCP报文段，但是ui只发给了gateway两个，就受到了RST包，不再发送后面的包了，导致gateway拿不到一个完整的http包，就返回502bad gateway了。

按照这个流程，问题可能出现在了ui发给gateway的k8s网络中间层这个过程上。

第二次抓包

        然后把ui里的nginx.conf的配置，从路由到gateway:8080改成了192.169.159.255:8080，期望是能跳过gateway cluster这一层，然后改完重启，ui这个pod跑到了a04上，情况是这样。

再请求，发现报错只有1.19.1的的nginx报出来的502了。需要再次抓包。

从bankaccount的pod发请求给ingress，报错都是

各个pod的ip情况如下：

ingress:10.36.13.205
ui:192.168.159.208
gateway:192.168.159.255
budget:192.168.159.249
发送pod：192.168.159.246

ingress:

ui:

在看gateway的抓包结果，发现这个时间段是没有对应的包的，说明这个包没走到gateway。
可以看到ingress给ui发了5个TCP报文段组成的http包，到ui这边只收到了两个TCP报文段，然后发送了RST包，不在接收后面的包了，ingress接收到RST包后，也不再重发接受ui失败的包，造成ui接收不到完整的http包。

流程如图：

        但是这里有一点需要注意，ingress发给ui的包，ip就是ui这个pod的，也没经过转换。这里和上一次抓包的时候，发给gateway的包ip转换了有一些不同。

基于上述分析，猜测可能pod之间的调用如果跨node，就会有包丢失造成connection reset的问题。

验证跨node的调用

        想法就是挑选a04和a05之间的pod互相发请求，看结果。
挑选的四个pod如下：

验证的流程是： homepage →  ingress(a05 → a05); bankaccount → homepage(a04 → a05); homepage → budget(a05->a04) ; budget → bankaccount( a04 → a04)
根据上面的流程发数次请求后，结果是：
成功：a05->a05 a04->a04
connection reset: a05->a04 a04->a05
和这个想法是一致的。

结论

        k8s的内部网络问题，跨node之间发送比较大的请求（一个TCP报文段发不完）的情况，可能会因为k8s网络中间层繁忙而转换ip失败，导致接收http包的一方接收到未经过ip转换的TCP报文段而不能识别此包，会发送RST包给发送方，发送http包的一方收到RST包后不会再发送没发完的http包，也不会重发之前接受失败的包，会把tcp连接释放掉，造成丢包的问题。反应到页面上就是nginx报502，页面提示"网络异常“。

• 解决方案

1. 因为xx集团的k8s环境问题比较多，三个从节点a04,a05,a06里面Evicted 的pod有一万多个，目前是把这些pod删掉了，问题还没复现过，还需要继续跟踪。
   
   参考资料：
   https://kubernetes.io/blog/2019/03/29/kube-proxy-subtleties-debugging-an-intermittent-connection-reset/​
   https://maimai.cn/article/detail?fid=1716626083&efid=exAQgZXTj-4Tzk8rKYSjiA​
   https://blog.frognew.com/2018/12/kubernetes-ipvs-long-connection-optimize.html​