一、原文地址
滴滴开源Super-jacoco:java代码覆盖率收集平台 - 掘金
二、背景
我要使用Super-jacoco,对手工测试,进行代码覆盖率的统计。
为什么使用Super-jacoco,而不是直接使用jacoco,因为Super-jacoco解决了增量覆盖率的问题
三、Super-jacoco介绍
Super-Jacoco是基于Jacoco、git二次开发打造的一站式JAVA代码全量/diff覆盖率收集平台,能够低成本、无侵入的收集代码覆盖率数据;Super-Jacoco除了支持JVM运行时间段的覆盖率收集外;还能够和环境无缝对接,收集服务端自定义时间段代码全量/增量覆盖率;并提供可视化的html覆盖率报表,协助覆盖率分析,支撑精准测试落地。
我这里只统计手工测试用例,代码覆盖率的数据。
没有对单元测试代码覆盖率做说明。
四、Super-jacoco原理
4.1 整体流程
为了支持增量覆盖率收集,我们需要做两件事情:**1)**获取不同版本代码diff文件;**2)**对jacoco进行二次开发,使其支持增量方法列表参数。
4.2 获取增量代码
主要流程:拉取master(参照分支)和feature(提测分支)代码,再通过JGit对两个分支源码进行比对,获取增量代码。以下为部分代码片段:
4.3. jacoco 二次改造,支持增量方法列表参数
JaCoCo 对 exec 的解析主要是在 Analyzer 类的 analyzeClass(final byte[] source) 方法。这里面会调用 createAnalyzingVisitor 方法,生成一个用于解析的 ASM 类访问器,继续跟代码,发现对方法级别的探针计算逻辑是在 ClassProbesAdapter 类的 visitMethod 方法里面。所以我们只需要改造 visitMethod 方法,使它只对提取出的每个类的新增或变更方法做解析,非指定类和方法不做处理。改造后的核心代码片段如下:
4.4. 执行
只需要在执行的mvn命令中加入-Djacoco.diffFile=变更方法列表,即可收集变更方法的代码覆盖率。如果不传入-Djacoco.diffFile或者Djacoco.diffFile参数为空,则默认收集全量覆盖率。
4.5. 报告输出
覆盖率报告如下图,在图中是某个 service 的实现类,在最新的代码中有23个方法,但是只会对变更或新增的5个方法进行覆盖率统计与显示:
五、特性
-
通用:既支持单元测试覆盖率收集,也支持手工测试覆盖率收集;既支持全量覆盖率收集,也支持diff覆盖率收集;
-
无侵入:采用on-the-fly模式,无需对开发代码做任何改造,即可收集覆盖率数据;
-
高可用:分布式架构,任务机可无限扩展,避免任务机down机或者任务过多时出现性能瓶颈;
-
可视化:提供html格式的覆盖率报告,可读性高。
六、架构
七、super-jacoco具体应用
7.1 环境准备
7.1.1 MySQL数据库准备
MySQL 5.x:
1、需要先下载安装 MySQL 5.x的数据库。
MySQL 8.x:
如果你的数据库为MySQL 8.x,需要修改项目中的驱动。
1、修改pom.xml文件
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.didichuxing.chefuqa</groupId>
<artifactId>super-jacoco</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>super-jacoco</name>
<description>project for Spring cloud eureka client</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.2.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<entity.target.dir>src/main/java/</entity.target.dir>
<dao.resources.dir>src/main/resources/</dao.resources.dir>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>1.8</java.version>
<spring-cloud.version>Finchley.RELEASE</spring-cloud.version>
<mybatis-spring-boot>1.2.0</mybatis-spring-boot>
<mysql-connector>8.0.26</mysql-connector>
<JAVA_HOME>/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home</JAVA_HOME>
</properties>
<dependencies>
<dependency>
<groupId>org.jodd</groupId>
<artifactId>jodd-core</artifactId>
<version>5.1.6</version>
</dependency>
<dependency>
<groupId>dom4j</groupId>
<artifactId>dom4j</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>org.jacoco</groupId>
<artifactId>org.jacoco.core</artifactId>
<version>0.8.4</version>
</dependency>
<dependency>
<groupId>org.ow2.asm</groupId>
<artifactId>asm</artifactId>
<version>7.1</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context-support</artifactId>
<version>5.1.4.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mybatis.spring.boot</groupId>
<artifactId>mybatis-spring-boot-starter</artifactId>
<version>${mybatis-spring-boot}</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.16.16</version>
</dependency>
<!-- MySQL 连接驱动依赖 -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>${mysql-connector}</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-pool2</artifactId>
<version>2.5.0</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.13</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.9.13</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>15.0</version>
</dependency>
<dependency>
<groupId>net.sf.json-lib</groupId>
<artifactId>json-lib</artifactId>
<version>2.4</version>
<classifier>jdk15</classifier>
</dependency>
<dependency>
<groupId>com.jayway.jsonpath</groupId>
<artifactId>json-path</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.sharegov</groupId>
<artifactId>mjson</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.3</version>
</dependency>
<!-- pinyin-->
<dependency>
<groupId>com.belerweb</groupId>
<artifactId>pinyin4j</artifactId>
<version>2.5.1</version>
</dependency>
<dependency>
<groupId>org.jdom</groupId>
<artifactId>jdom2</artifactId>
<version>2.0.6</version>
</dependency>
<!-- ssh的依赖 -->
<dependency>
<groupId>ch.ethz.ganymed</groupId>
<artifactId>ganymed-ssh2</artifactId>
<version>build210</version>
</dependency>
<dependency>
<groupId>org.tmatesoft.svnkit</groupId>
<artifactId>svnkit</artifactId>
<version>1.8.7</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.5</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
<version>4.2.3</version>
</dependency>
<dependency>
<groupId>io.rest-assured</groupId>
<artifactId>rest-assured</artifactId>
<version>3.1.1</version>
</dependency>
<dependency>
<groupId>org.eclipse.jgit</groupId>
<artifactId>org.eclipse.jgit</artifactId>
<version>4.9.0.201710071750-r</version>
</dependency>
<dependency>
<groupId>com.github.javaparser</groupId>
<artifactId>javaparser-core</artifactId>
<version>3.4.4</version>
</dependency>
<!-- Swagger2核心包 -->
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>2.7.0</version>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>2.7.0</version>
</dependency>
<dependency>
<!-- jsoup HTML parser library @ http://jsoup.org/ -->
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.10.2</version>
</dependency>
<dependency>
<groupId>tk.mybatis</groupId>
<artifactId>mapper-spring-boot-starter</artifactId>
<version>2.0.2</version>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<finalName>super-jacoco</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
<compilerArguments>
<bootclasspath>${java.home}/lib/rt.jar:${java.home}/lib/jce.jar</bootclasspath>
</compilerArguments>
</configuration>
</plugin>
<plugin>
<groupId>org.mybatis.generator</groupId>
<artifactId>mybatis-generator-maven-plugin</artifactId>
<version>1.3.7</version>
<configuration>
<verbose>true</verbose>
<overwrite>true</overwrite>
<configurationFile>
${basedir}/src/main/resources/tk-mybatis-autogen.xml
</configurationFile>
<!--允许移动生成的文件 -->
<verbose>true</verbose>
<overwrite>true</overwrite>
</configuration>
<dependencies>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.26</version>
</dependency>
<dependency>
<groupId>tk.mybatis</groupId>
<artifactId>mapper</artifactId>
<version>4.1.5</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</project>
2、application.properties修改数据库配置驱动
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
application.properties整体代码:
spring.application.name=super-jacoco
server.port=8899
# 数据源配置
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.session.store-type=none
# Mybatis 配置
mybatis.typeAliasesPackage=com.xiaoju.basetech.entity
mybatis.mapper-locations=classpath*:mapper/*.xml
spring.resources.static-locations=file:${user.home}/report
spring.mvc.static-path-pattern=/**
spring.http.encoding.force=true
# 以下信息需要手动配置
spring.datasource.url=jdbc:mysql://127.0.0.1:3306/super-jacoco?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull&allowMultiQueries=true&serverTimezone=Asia/Shanghai&useSSL=false
spring.datasource.username=root
spring.datasource.password=123456
#git username & password
gitlab.username=
gitlab.password=
7.2. 数据库安装和初始化
7.1.1 安装mysql数据库,创建数据库后执行sql/db.sql文件中的建表SQL
CREATE DATABASE `super-jacoco` DEFAULT CHARACTER SET utf8 COLLATE utf8_bin ;
CREATE TABLE `diff_coverage_report` (
`id` int(10) NOT NULL AUTO_INCREMENT,
`job_record_uuid` varchar(80) NOT NULL COMMENT '请求唯一标识码',
`request_status` int(10) NOT NULL COMMENT '请求执行状态,1=下载代码成功,2=生成diffmethod成功,3=生成报告成功,-1=执行出错',
`giturl` varchar(80) NOT NULL COMMENT 'git 地址',
`now_version` varchar(80) NOT NULL COMMENT '本次提交的commidId',
`base_version` varchar(80) NOT NULL COMMENT '比较的基准commitId',
`diffmethod` mediumtext COMMENT '增量代码的diff方法集合',
`type` int(11) NOT NULL DEFAULT '0' COMMENT '2=增量代码覆盖率,1=全量覆盖率',
`report_url` varchar(300) NOT NULL DEFAULT '' COMMENT '覆盖率报告url',
`line_coverage` double(5,2) NOT NULL DEFAULT '-1.00' COMMENT '行覆盖率',
`branch_coverage` double(5,2) NOT NULL DEFAULT '-1.00' COMMENT '分支覆盖率',
`err_msg` varchar(1000) NOT NULL DEFAULT '' COMMENT '错误信息',
`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
`update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
`sub_module` varchar(255) NOT NULL DEFAULT '' COMMENT '子项目目录名称',
`from` int(10) NOT NULL DEFAULT '0' COMMENT '1=单元测试,2=环境部署1=单元测试,2=hu',
`now_local_path` varchar(500) NOT NULL DEFAULT '',
`base_local_path` varchar(500) NOT NULL DEFAULT '',
`log_file` varchar(255) NOT NULL DEFAULT '',
PRIMARY KEY (`job_record_uuid`),
KEY `id` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8 COMMENT='增量代码覆盖率';
CREATE TABLE `diff_deploy_info` (
`id` int(10) NOT NULL AUTO_INCREMENT,
`job_record_uuid` varchar(80) NOT NULL COMMENT '请求唯一标识码',
`address` varchar(15) NOT NULL COMMENT 'HOST',
`port` int(10) NOT NULL COMMENT '端口',
`code_path` varchar(1000) NOT NULL DEFAULT '' COMMENT 'nowVersion代码目录',
`child_modules` varchar(1000) NOT NULL DEFAULT '' COMMENT '项目子模块名称',
PRIMARY KEY (`job_record_uuid`),
KEY `id` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8 COMMENT='服务部署地址';
7.3 super-jacoco 项目下载
项目地址:
https://github.com/didi/super-jacoco
尽量使用git命令,直接下载:
git clone https://github.com/didi/super-jacoco.git
7.2. 编译打包
1、更改配置文件(数据库配置与gitlab配置)
更改application.properties文件中配置
spring.datasource.url=jdbc:mysql://IP:端口/数据库名?useUnicode=true&characterEncoding=utf8
spring.datasource.username=
pring.datasource.password=
gitlab.username=
gitlab.password=
2、打包
执行mvn package -Dmaven.test.skip=true生成super-jacoco.jar
7.3. 部署
启动super-jacoco.jar 服务
执行命令
nohup java -jar super-jacoco.jar &
默认端口为8899
可以8899端口,程序是否启动成功
lsof -i:8899
7.4. 覆盖率收集接口
7.4.1 单测覆盖率接口
1)启动覆盖率收集
URL:/cov/triggerUnitCover
调用方法:POST
参数(body方式传入):{"uuid":"uuid","type":1,"gitUrl":"git@git","subModule":"","baseVersion":"master","nowVersion":"feature","envType":"-Ptest"}
返回:{"code":200,"data":true,"msg":"msg"}
备注:
2)获取覆盖率结果
URL:/cov/getUnitCoverResult
调用方法:GET
参数:uuid(String)
返回:
{"code":200,"data":{"coverStatus":1,"errMsg":"msg","lineCoverage":100.0,"branchCoverage":100.0,"logFile":"file content","reportUrl":"http://"},"msg":"msg"}
备注:
7.4.2 环境覆盖率接口
1)启动覆盖率收集URL:/cov/triggerEnvCov调用方法:POST参数(body方式传入):{"uuid":"uuid","type":1,"gitUrl":"git@git","subModule":"","baseVersion":"master","nowVersion":"feature","address":"127.0.0.1","port":"8088"}返回:{"code":200,"data":true,"msg":"msg"}备注:IP和port为模块部署服务器的IP和端口,在dump jacoco.exec时使用,需要提前把org.jacoco.agent-0.8.5-runtime.jar包拷贝到服务器:/home/xxx/目录,服务启动时需要添加启动参数:-javaagent:/home/xxx/org.jacoco.agent-0.8.5-runtime.jar=includes=*,output=tcpserver,address=*,port=18513 2)获取覆盖率结果URL:/cov/getEnvCoverResult调用方法:GET参数:uuid(String)返回:{"code":200,"data":{"coverStatus":1,"errMsg":"msg","lineCoverage":100.0,"branchCoverage":100.0,"logFile":"file content","reportUrl":"http://"},"msg":"msg"}备注:
八、GitHub项目地址
GitHub - didi/super-jacoco