18.2 k8s-apiserver监控源码解读

news2024/12/26 10:34:19

本节重点介绍 :

  • k8s代码库和模块地址

    • 下载 apiserver源码
  • apiserver中监控源码阅读

k8s源码地址分布

k8s代码库

  • 访问github上k8s仓库,readme中给出了k8s 模块的代码地址
  • 举例图片
  • image.png

组件仓库列表 地址

Repositories currently staged here:

  • k8s.io/api
  • k8s.io/apiextensions-apiserver
  • k8s.io/apimachinery
  • k8s.io/apiserver
  • k8s.io/cli-runtime
  • k8s.io/client-go
  • k8s.io/cloud-provider
  • k8s.io/cluster-bootstrap
  • k8s.io/code-generator
  • k8s.io/component-base
  • k8s.io/controller-manager
  • k8s.io/cri-api
  • k8s.io/csi-api
  • k8s.io/csi-translation-lib
  • k8s.io/kube-aggregator
  • k8s.io/kube-controller-manager
  • k8s.io/kube-proxy
  • k8s.io/kube-scheduler
  • k8s.io/kubectl
  • k8s.io/kubelet
  • k8s.io/legacy-cloud-providers
  • k8s.io/metrics
  • k8s.io/mount-utils
  • k8s.io/pod-security-admission
  • k8s.io/sample-apiserver
  • k8s.io/sample-cli-plugin
  • k8s.io/sample-controller

下载 apiserver 源码

go get -d  k8s.io/apiserver

分析apiserver 监控源码

以qps指标 apiserver_request_total为例

定位源码位置

  • 在源码目录全文搜索apiserver_request_total,选择.go文件
  • 举例图片
  • image.png
  • 发现位于 D:\go_path\pkg\mod\k8s.io\apiserver@v0.22.1\pkg\endpoints\metrics\metrics.go
	requestCounter = compbasemetrics.NewCounterVec(
		&compbasemetrics.CounterOpts{
			Name:           "apiserver_request_total",
			Help:           "Counter of apiserver requests broken out for each verb, dry run value, group, version, resource, scope, component, and HTTP response code.",
			StabilityLevel: compbasemetrics.STABLE,
		},
		[]string{"verb", "dry_run", "group", "version", "resource", "subresource", "scope", "component", "code"},
	)
  • 分析,在这里定义了指标,并且指定了相关的标签,可以和我们在prometheus中查询到的结果匹配上

apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="apiregistration.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="apiservices", scope="cluster", verb="WATCH", version="v1beta1"}
9014
apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="apps", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="daemonsets", scope="cluster", verb="WATCH", version="v1"}
8863
apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="apps", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="deployments", scope="cluster", verb="WATCH", version="v1"}
8830
apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="crd.projectcalico.org", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="bgpconfigurations", scope="cluster", verb="WATCH", version="v1"}
5575
apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="crd.projectcalico.org", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="bgppeers", scope="cluster", verb="WATCH", version="v1"}
2791
apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="crd.projectcalico.org", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="blockaffinities", scope="cluster", verb="WATCH", version="v1"}
2942
apiserver_request_total{code="0", component="apiserver", contentType="application/json", group="crd.projectcalico.org", instance="172.20.70.205:6443", job="kubernetes-apiservers", resource="clusterinformations", scope="cluster", verb="WATCH", version="v1"}
2770

处理请求的metric 函数 MonitorRequest

  • 代码位置 D:\go_path\pkg\mod\k8s.io\apiserver@v0.22.1\pkg\endpoints\metrics\metrics.go
  • 源码如下
// MonitorRequest handles standard transformations for client and the reported verb and then invokes Monitor to record
// a request. verb must be uppercase to be backwards compatible with existing monitoring tooling.
func MonitorRequest(req *http.Request, verb, group, version, resource, subresource, scope, component string, deprecated bool, removedRelease string, httpCode, respSize int, elapsed time.Duration) {
	// We don't use verb from <requestInfo>, as this may be propagated from
	// InstrumentRouteFunc which is registered in installer.go with predefined
	// list of verbs (different than those translated to RequestInfo).
	// However, we need to tweak it e.g. to differentiate GET from LIST.
	reportedVerb := cleanVerb(CanonicalVerb(strings.ToUpper(req.Method), scope), verb, req)

	dryRun := cleanDryRun(req.URL)
	elapsedSeconds := elapsed.Seconds()
	requestCounter.WithContext(req.Context()).WithLabelValues(reportedVerb, dryRun, group, version, resource, subresource, scope, component, codeToString(httpCode)).Inc()
	// MonitorRequest happens after authentication, so we can trust the username given by the request
	info, ok := request.UserFrom(req.Context())
	if ok && info.GetName() == user.APIServerUser {
		apiSelfRequestCounter.WithContext(req.Context()).WithLabelValues(reportedVerb, resource, subresource).Inc()
	}
	if deprecated {
		deprecatedRequestGauge.WithContext(req.Context()).WithLabelValues(group, version, resource, subresource, removedRelease).Set(1)
		audit.AddAuditAnnotation(req.Context(), deprecatedAnnotationKey, "true")
		if len(removedRelease) > 0 {
			audit.AddAuditAnnotation(req.Context(), removedReleaseAnnotationKey, removedRelease)
		}
	}
	requestLatencies.WithContext(req.Context()).WithLabelValues(reportedVerb, dryRun, group, version, resource, subresource, scope, component).Observe(elapsedSeconds)
	// We are only interested in response sizes of read requests.
	if verb == "GET" || verb == "LIST" {
		responseSizes.WithContext(req.Context()).WithLabelValues(reportedVerb, group, version, resource, subresource, scope, component).Observe(float64(respSize))
	}
}

  • 其中 WithLabelValues设置metric的值 ,Inc()代表counter +1
requestCounter.WithContext(req.Context()).WithLabelValues(reportedVerb, dryRun, group, version, resource, subresource, scope, component, codeToString(httpCode)).Inc()

  • 这个是k8s在prometheus sdk上做的封装,位置 D:\go_path\pkg\mod\k8s.io\component-base@v0.22.1\metrics\counter.go
func (v *CounterVec) WithLabelValues(lvs ...string) CounterMetric {
	if !v.IsCreated() {
		return noop // return no-op counter
	}
	if v.LabelValueAllowLists != nil {
		v.LabelValueAllowLists.ConstrainToAllowedList(v.originalLabels, lvs)
	}
	return v.CounterVec.WithLabelValues(lvs...)
}

自己请求的计数
  • 如果userName是 system:apiserver ,那么把代表自身请求的metric apiserver_selfrequest_total +1
	info, ok := request.UserFrom(req.Context())
	if ok && info.GetName() == user.APIServerUser {
		apiSelfRequestCounter.WithContext(req.Context()).WithLabelValues(reportedVerb, resource, subresource).Inc()
	}
要被废弃的api被请求
  • metric丢弃的设置,apiserver_requested_deprecated_apis代表要被废弃的api被请求了,打印信息
	if deprecated {
		deprecatedRequestGauge.WithContext(req.Context()).WithLabelValues(group, version, resource, subresource, removedRelease).Set(1)
		audit.AddAuditAnnotation(req.Context(), deprecatedAnnotationKey, "true")
		if len(removedRelease) > 0 {
			audit.AddAuditAnnotation(req.Context(), removedReleaseAnnotationKey, removedRelease)
		}
	}
  • prometheus查询结果
apiserver_requested_deprecated_apis{group="apiregistration.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="apiservices", version="v1beta1"}
1
apiserver_requested_deprecated_apis{group="authorization.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="subjectaccessreviews", version="v1beta1"}
1
apiserver_requested_deprecated_apis{group="certificates.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="certificatesigningrequests", version="v1beta1"}
1
apiserver_requested_deprecated_apis{group="extensions", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="ingresses", version="v1beta1"}
1
apiserver_requested_deprecated_apis{group="networking.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="ingressclasses", version="v1beta1"}
1
apiserver_requested_deprecated_apis{group="networking.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="ingresses", version="v1beta1"}
1
apiserver_requested_deprecated_apis{group="scheduling.k8s.io", instance="172.20.70.205:6443", job="kubernetes-apiservers", removed_release="1.22", resource="priorityclasses", version="v1beta1"}
1

设置请求延迟值
  • 代码如下
requestLatencies.WithContext(req.Context()).WithLabelValues(reportedVerb, dryRun, group, version, resource, subresource, scope, component).Observe(elapsedSeconds)

  • requestLatencies定义了从0.05到60秒的bucket
	requestLatencies = compbasemetrics.NewHistogramVec(
		&compbasemetrics.HistogramOpts{
			Name: "apiserver_request_duration_seconds",
			Help: "Response latency distribution in seconds for each verb, dry run value, group, version, resource, subresource, scope and component.",
			// This metric is used for verifying api call latencies SLO,
			// as well as tracking regressions in this aspects.
			// Thus we customize buckets significantly, to empower both usecases.
			Buckets: []float64{0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
				1.25, 1.5, 1.75, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30, 40, 50, 60},
			StabilityLevel: compbasemetrics.STABLE,
		},
		[]string{"verb", "dry_run", "group", "version", "resource", "subresource", "scope", "component"},
	)
  • 分位值查询
histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{job="kubernetes-apiservers"}[5m])) by (verb, le))

  • 结果图片image.png
设置 http响应的大小histogram
  • 设置代码
	if verb == "GET" || verb == "LIST" {
		responseSizes.WithContext(req.Context()).WithLabelValues(reportedVerb, group, version, resource, subresource, scope, component).Observe(float64(respSize))
	}
  • responseSizes定义了 1KB 到1GB的大小的bucket
	responseSizes = compbasemetrics.NewHistogramVec(
		&compbasemetrics.HistogramOpts{
			Name: "apiserver_response_sizes",
			Help: "Response size distribution in bytes for each group, version, verb, resource, subresource, scope and component.",
			// Use buckets ranging from 1000 bytes (1KB) to 10^9 bytes (1GB).
			Buckets:        compbasemetrics.ExponentialBuckets(1000, 10.0, 7),
			StabilityLevel: compbasemetrics.ALPHA,
		},
		[]string{"verb", "group", "version", "resource", "subresource", "scope", "component"},
	)

追踪MonitorRequest调用链

InstrumentRouteFunc在prometheus的HandlerFunc基础上封装了k8s的go-restful

// InstrumentRouteFunc works like Prometheus' InstrumentHandlerFunc but wraps
// the go-restful RouteFunction instead of a HandlerFunc plus some Kubernetes endpoint specific information.
func InstrumentRouteFunc(verb, group, version, resource, subresource, scope, component string, deprecated bool, removedRelease string, routeFunc restful.RouteFunction) restful.RouteFunction {
	return restful.RouteFunction(func(req *restful.Request, response *restful.Response) {
		requestReceivedTimestamp, ok := request.ReceivedTimestampFrom(req.Request.Context())
		if !ok {
			requestReceivedTimestamp = time.Now()
		}

		delegate := &ResponseWriterDelegator{ResponseWriter: response.ResponseWriter}

		//lint:file-ignore SA1019 Keep supporting deprecated http.CloseNotifier
		_, cn := response.ResponseWriter.(http.CloseNotifier)
		_, fl := response.ResponseWriter.(http.Flusher)
		_, hj := response.ResponseWriter.(http.Hijacker)
		var rw http.ResponseWriter
		if cn && fl && hj {
			rw = &fancyResponseWriterDelegator{delegate}
		} else {
			rw = delegate
		}
		response.ResponseWriter = rw

		routeFunc(req, response)

		MonitorRequest(req.Request, verb, group, version, resource, subresource, scope, component, deprecated, removedRelease, delegate.Status(), delegate.ContentLength(), time.Since(requestReceivedTimestamp))
	})
}

registerResourceHandlers 中根据不同的verb处理

  • 位置 D:\go_path\pkg\mod\k8s.io\apiserver@v0.22.1\pkg\endpoints\installer.go
  • LIST的verb代码如下
		case "LIST": // List all resources of a kind.
			doc := "list objects of kind " + kind
			if isSubresource {
				doc = "list " + subresource + " of objects of kind " + kind
			}
			handler := metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, deprecated, removedRelease, restfulListResource(lister, watcher, reqScope, false, a.minRequestTimeout))
			handler = utilwarning.AddWarningsHandler(handler, warnings)
			route := ws.GET(action.Path).To(handler).
				Doc(doc).
				Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
				Operation("list"+namespaced+kind+strings.Title(subresource)+operationSuffix).
				Produces(append(storageMeta.ProducesMIMETypes(action.Verb), allMediaTypes...)...).
				Returns(http.StatusOK, "OK", versionedList).
				Writes(versionedList)
  • 上层被APIInstaller调用
// Install handlers for API resources.
func (a *APIInstaller) Install() ([]metav1.APIResource, []*storageversion.ResourceInfo, *restful.WebService, []error) {
	var apiResources []metav1.APIResource
	var resourceInfos []*storageversion.ResourceInfo
	var errors []error
	ws := a.newWebService()

	// Register the paths in a deterministic (sorted) order to get a deterministic swagger spec.
	paths := make([]string, len(a.group.Storage))
	var i int = 0
	for path := range a.group.Storage {
		paths[i] = path
		i++
	}
	sort.Strings(paths)
	for _, path := range paths {
		apiResource, resourceInfo, err := a.registerResourceHandlers(path, a.group.Storage[path], ws)
		if err != nil {
			errors = append(errors, fmt.Errorf("error in registering resource: %s, %v", path, err))
		}
		if apiResource != nil {
			apiResources = append(apiResources, *apiResource)
		}
		if resourceInfo != nil {
			resourceInfos = append(resourceInfos, resourceInfo)
		}
	}
	return apiResources, resourceInfos, ws, errors
}

本节重点总结 :

  • k8s代码库和模块地址

    • 下载 apiserver源码
  • apiserver中监控源码阅读

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.coloradmin.cn/o/2164342.html

如若内容造成侵权/违法违规/事实不符,请联系多彩编程网进行投诉反馈,一经查实,立即删除!

相关文章

Maven配置、下载教程(非常详细)

maven下载地址 Maven – 发行说明 – Maven 3.6.1 (apache.org) 1.配置settings.xml 下载完maven之后&#xff0c;保存在电脑中并解压 打开maven文件 -->conf-->settings.xml 使用记事本方式打开 打开之后找到这个地方&#xff0c;在电脑中创建一个文件夹(repository)…

起重机防摇摆技术如何达标-武汉正向科技

武汉正向科技防摇摆控制器 主要技术参数 1、防摇摆精度&#xff1a; 0.4 2、行车到达目标位置偏差位置偏差&#xff1a; 25mm 3、通讯方式&#xff1a;PROFINET / PROFIBUS / RS232 / RS422 / RS485&#xff1b; 4、消除载荷的摇摆达 96% 以上&#xff1b; 5、技术先进…

图像面积计算一般方法及MATLAB实现

一、引言 在数字图像处理中&#xff0c;经常需要获取感兴趣区域的面积属性&#xff0c;下面给出图像处理的一般步骤。 1.读入的彩色图像 2.将彩色图像转化为灰度图像 3.灰度图像转化为二值图像 4.区域标记 5.对每个区域的面积进行计算和显示 二、程序代码 %面积计算 cle…

- 串口通信

USART串口通信 目录 USART串口通信 回顾 USART串口通信 1、通信分类与作用 2、串口通信的相关参数&#xff08;重点&#xff09; 3、位协议层 -- RS232协议 4、STM32F103 中的串口外设 5、调试串口编程 -- &#xff08;1&#xff09;串口初始化&#xff1a;时钟、IO、…

【Git入门】使用 Git 进行项目管理:Word Count 程序开发与托管

在软件开发过程中&#xff0c;版本控制工具是不可或缺的。Git 作为一款强大的分布式版本控制工具&#xff0c;为开发者提供了高效的代码管理和协作方式。本博客将介绍如何下载安装 Git 版本管理工具&#xff0c;并使用 Git 和 GitHub 平台进行一个名为 Word Count 的项目开发与…

【鸿蒙HarmonyOS NEXT】数据存储之关系型数据库RDS

【鸿蒙HarmonyOS NEXT】数据存储之关系型数据库RDS 一、环境说明二、关系型数据库RDS介绍三、示例代码加以说明四、小结 一、环境说明 DevEco Studio 版本&#xff1a; API版本&#xff1a;以12为主 二、关系型数据库RDS介绍 1. RDS关系型数据库简介&#xff1a; 关系型数…

VMware提供虚拟硬盘并使得Oracle Linux集群共享块设备并绑定raw设备。

一、Vmware操作 nodeA : 1、创建SCSI控制器:类型为物理。 添加新磁盘。 类型为独立-持久。 nodeB: 新增磁盘,但是选择node A刚才创建的磁盘。 类型:独立-持久。 二、OS层操作 两台都要做绑定。 详细步骤 1. 创建 raw 设备节点 首先,确保 /dev/raw 目录存在。如果不…

5分钟精通Excel在go中的使用

一些简单操作可以在官方文档中找到&#xff0c;应该足够无经验的朋友们入门 介绍 - 《Excelize v2.2 中文文档》 - 书栈网 BookStack 这里贴一个中文版的链接&#xff08;以excelize库为例&#xff0c;相对其他库来说&#xff0c;体验很不错&#xff09;&#xff0c;不过要注…

沉浸式艺术创作:FLUX.1模型下的Java开发者体验之旅

文章目录 1 FLUX.1革命2 应用落地3 部署ComfyUI4 部署FLUX.15 导入工作流6 新的挑战 1 FLUX.1革命 FLUX.1&#xff1a;图像生成的新纪元 在人工智能的图像生成领域&#xff0c;FLUX.1模型的出现标志着一个新的时代。 由黑森林实验室&#xff08;Black Forest Labs&#xff0…

Android 车载应用开发指南 - CarService 详解(下)

车载应用正在改变人们的出行体验。从导航到娱乐、从安全到信息服务&#xff0c;车载应用的开发已成为汽车智能化发展的重要组成部分。而对于开发者来说&#xff0c;如何将自己的应用程序无缝集成到车载系统中&#xff0c;利用汽车的硬件和服务能力&#xff0c;是一个极具挑战性…

数据报文解析

数据报文解析 报文介绍 如下图所示&#xff0c;每一层把上传的协议包当作数据部分&#xff0c;加上自己的协议头部&#xff0c;组成自己的协议包。 一般说法&#xff1a;默认以太网协议包&#xff08;网络层从IP头部开始计算&#xff09;最大传输单元&#xff08;MTU&#x…

记一次sql查询优化

记一次sql查询优化 前言 这是我在这个网站整理的笔记,有错误的地方请指出&#xff0c;关注我&#xff0c;接下来还会持续更新。 作者&#xff1a;神的孩子都在歌唱 今天测试环境发现一个问题&#xff0c;就是测试同事在测试的时候&#xff0c;发现cpu一直居高不下&#xff0c;然…

CNN网络训练WISDM数据集:模型仿真及可视化分析

卷积神经网络&#xff08;CNN&#xff09;因其强大的特征提取能力和深度学习架构而备受推崇&#xff0c;CNN在处理图像数据时展现出的卓越性能&#xff0c;使其成为解决各种视觉识别任务的首选工具。WISDM数据集是一个广泛用于运动估计研究的基准数据集&#xff0c;它包含了多个…

EmguCV学习笔记 VB.Net 11.9 姿势识别 OpenPose

版权声明&#xff1a;本文为博主原创文章&#xff0c;转载请在显著位置标明本文出处以及作者网名&#xff0c;未经作者允许不得用于商业目的。 EmguCV是一个基于OpenCV的开源免费的跨平台计算机视觉库,它向C#和VB.NET开发者提供了OpenCV库的大部分功能。 教程VB.net版本请访问…

AMEYA360代理:兆易创新GD32A7系列全新一代车规级MCU介绍

兆易创新GigaDevice(股票代码 603986)今日宣布&#xff0c;重磅推出全新一代车规级MCU GD32A7系列。与上一代采用Arm Cortex-M4/M33的产品相比&#xff0c;GD32A7系列搭载了超高性能Arm Cortex-M7内核&#xff0c;提供GD32A71x/GD32A72x/GD32A74x等多款型号供用户选择。该系列产…

git 生成和查看密钥

项目场景&#xff1a; 在前端项目开发中&#xff0c;经常会用到git。一般的小公司很少去设置git令牌或者密钥&#xff1b;而在一些大公司&#xff0c;会用到这个。今天主要整理下git如何生成和查看密钥。 密钥 1、生成密钥 cat ~/.ssh/id_rsa.pub 2、查看密钥 ssh-keygen…

电容笔最建议买哪一款?2024百元价位性价比首选榜单,速速码住!

现在电容笔已经成为我们日常学习、工作和创作中不可或缺的辅助工具。无论是记笔记、做习题&#xff0c;还是进行绘画、设计&#xff0c;电容笔都发挥着关键作用。然而&#xff0c;市场上的电容笔品牌和款式繁多&#xff0c;价格也从几十元到上千元不等&#xff0c;这让消费者在…

OSError: [Errno 16] Device or resource busy: ‘.nfs*‘报错解决办法

目录 1 项目场景&问题描述&#xff1a;2 原因分析&#xff1a;2.1 问题背景&#xff1a; 3 解决方案&#xff1a;3.1 创建存放临时文件的目录3.2 使用该目录3.2.1 设置环境变量 TMPDIR3.2.2 运行时设置&#xff08;推荐&#xff09;3.2.3 代码中设置 4 总结 1 项目场景&…

Redis 键值对数据库学习

目录 一、介绍 二、安装以及连接 三、设置连接密码 四、连接报错 五、redis 操作字符串以及过期时间 六、 redis 列表操作 七、redis 集合操作 八、hash 哈希操作 九、redis 发布和订阅操作 十、RDB和AOF的两种数据持久化机制 十一、 其他机器连接redis 十二、 pyt…

【Linux】解锁文件描述符奥秘,高效缓存区的实战技巧

fd和缓冲区 1. 文件描述符fd1.1. 概念与本质1.2. 打开文件的管理1.3. 一切皆文件的理解1.4. 分配规则1.5. 重定向的本质1.5.1. dup2 2. FILE中的缓冲区2.1. 概念2.2. 存在的原因2.3. 类型(刷新方案)2.4. 存放的位置2.4.1. 代码证明、现象解释 2.5. 模拟C标准库中的方法 1. 文件…