Gerrit | 重磅! 2.x 版本升级到 3.x 版本----转

Gerrit | 重磅! 2.x 版本升级到 3.x 版本

为什么要做版本升级？

2.x known bugs

重大问题不一一列举，这里仅仅是举几个例子：

安全或权限问题：普通用户能看到敏感数据，例如看到其他用户的 hashed api 密码，

问题传送门：https://bugs.chromium.org/p/gerrit/issues/detail?id=13514；

再例如已经设置为失效的账号，还可以登录 gerrit ；
升级时索引建立失败：重建索引未完成，插件就 load 了，索引建立过程 invoked，https://bugs.chromium.org/p/gerrit/issues/detail?id=6472；

3.0 的重大更新

ReviewDB 寿终正寝，NoteDB 上台
UI 界面全面更新，GWT 被 PolyGerrit 取代
支持 REST-API
其他重大更新，等等

新版本大概长这样

升级 Migration 实操记录

目前待升级的 gerrit 版本为 2.13.9，根据官方文档，必须确保每个中间版本都被升级，所以升级路线暂定为：

2.13 -- 2.14 -- 2.15 -- 2.16
到了 2.16 后将review DB 迁移到NoteDB
迁移 2.16 到 3.0

注意：v2.16是唯一包含和支持 reviewDB 到 NoteDB 的关键版本

Reference Doc：https://gitenterprise.me/2019/05/20/gerrit-v3-0-is-here/

提前测试

测试原版 gerrit 升级路径，不携带任何数据，为的是快速探索成功路径

升级路径：2.13.9 -- 2.15.21 -- 2.16.25 -- 3.2.3

s0：升级前，确认 gerrit 的 replication 和 HA 的插件和配置文件，被禁用(可移走或重命名)

s1:起点版本 2.13.9

$ java -jar gerrit-2.13.9.war init --batch --dev -d gerrit_test

s2:升级到 2.15.21

$ java -jar gerrit-2.15.21.war init --batch --dev -d gerrit_test

$ java -jar gerrit-2.15.21.war reindex --index groups -d gerrit_test

$ java -jar gerrit-2.15.21.war reindex --index accounts -d gerrit_test

$ java -jar gerrit-2.15.21.war reindex --index changes -d gerrit_test

执行以上三步，gerrit 服务才能启动成功

其中第三步 reindex，耗时较长，可以先 skip ，升级到 3.x 再做

s3:升级到 2.16.25

$ java -jar gerrit-2.16.25.war init --batch --dev -d gerrit_test

直接启动 gerrit 可以成功

s4:升级 3.x 之前，将 jdk 11 下载解压，并加入到环境变量中，source

$ java -version

openjdk version "11.0.9.1" 2020-11-04

OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.9.1+1)

OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.9.1+1, mixed mode)

s5:在2.16.25 上进行数据库 migrate

$ java -jar gerrit-2.16.25.war migrate-to-note-db -d gerrit_test

可以加上 --reindex false 选择跳过 reindex

s6:升级到 3.2.3

$ java -jar gerrit-3.2.3.war init --batch --dev -d gerrit_test

s7:进行 reindex

$ java -jar gerrit-3.2.3.war reindex --index accounts -d gerrit_test

$ java -jar gerrit-3.2.3.war reindex --index changes -d gerrit_test

启动 gerrit 服务，启动 OK, awsome！！

验证一下：

$ curl localhost:29418

SSH-2.0-GerritCodeReview_3.2.3 (APACHE-SSHD-2.4.0)

Unsupported protocol version: GET / HTTP/1.1

到此基本的操作步骤都涵盖在上面了，分割线以下，是我的实际操作日志，仅供参考。

如果有任何问题，可私信咨询。

实际 Tips

升级前的准备

检查 gerrit.config

[cache "diff"]

timeout = 200ms # 减少timeout 提升整体 reindex 速度

导入数据

进入 MySQL ： set GLOBAL max_connections=1000;

本地搭建 mysql，并导入数据

mysql -u username -p < ...sql脚本.sql ，

例如

$ mysql -u gerrit_rw -p --verbose < c3-inf-gerrit-db01.bj.sql

拷贝线上的 gerrit 目录并停掉 gerrit 的 HA 和 replication，然后修改 gerrit.config 的 database 为本地数据库配置

[database]

type = mysql

hostname = localhost

database = xxx

username = x

password = x

port = 3306

useSSL = false

[auth] type = DEVELOPMENT_BECOME_ANY_ACCOUNT

同步线上代码

$ ssh -p 29418 gerrit@localhost replication start --url 目标地址 --all

2.13 --> 2.14

#停止 gerrit 服务

java -jar gerrit-2.14.22.war init --batch --dev -d ${gerrit-site}

Get an error：
Exception in thread "main" java.nio.file.NoSuchFileException: gerrit/ha/1/gerrit/shared
解决方式：去掉或重命名etc目录下的 ha config 和 replication config ，去掉 plugins 文件下的 ha 和 replication 的 jar 包
这一步也可以在执行 升级命令 之前完成

继续上面的 java 命令

$ java -jar gerrit-2.14.22.war init --batch --dev -d ${gerrit-site}

Upgrading schema to 130 ...

Upgrading schema to 131 ...

Upgrading schema to 132 ...

迁移正常的打印信息为：

Migrating 5263 repositories ...

Migration completed: 3 repositories updated:

All-Projects platform/packages/apps/ContactsCommon platform/packages/apps/OneAccountSdk

> Done (44.769 s)

Migrating data to schema 131 ...

Migrating 5263 repositories ...

Migration completed: 16 repositories updated:

... ... ...

Migrating data to schema 141 ...

> Done (0.022 s)

Migrating data to schema 142 ...

> Done (137.173 s)

Execute the following SQL to drop unused objects:

... ... ...

Initialized /home/work/gerrit

报错：直接启动 gerrit 服务会失败：No index versions ready;

解决方式：进行 reindex 操作

$ java -jar gerrit-2.14.22.war reindex --index groups -d ${gerrit-site}

... ... ...

Finishing 5 disk cache updates

启动 gerrit 服务，查看升级后的版本

$ curl localhost:29418

SSH-2.0-GerritCodeReview_2.14.22 (SSHD-CORE-1.4.0)

Unsupported protocol version: GET / HTTP/1.1

或者查看一下新UI 左下角的版本信息（2.14.22）

继续下一步：2.14 --> 2.15

migration to 2.15 与上述内容基本一致

$ java -jar gerrit-2.15.21.war init --batch --dev -d ${gerrit-site}

Upgrading schema to 143 ...

Upgrading schema to 144 ...

... ... ...

Initialized /home/work/gerrit

这一步想偷懒不做 reindex，但是失败了

No index versions for index 'changes' ready;

run java -jar /home/work/gerrit/bin/gerrit.war reindex --index changes

先升级到 3.2.3 再做 reindex

下一步：2.15 --> 2.16

$ java -jar gerrit-2.16.25.war init --batch --dev -d ${gerrit-site}

$ java -jar gerrit-2.16.25.war reindex --index projects -d ${gerrit-site}

$ java -jar gerrit-2.16.25.war reindex --index groups -d ${gerrit-site}

$ java -jar gerrit-2.16.25.war reindex --index changes -d ${gerrit-site}

注意这一步可能会耗时很久，取决于Gerrit上change的数量大小

启动 gerrit 服务

$ curl localhost:29418

SSH-2.0-GerritCodeReview_2.16.25 (SSHD-CORE-2.0.0)

Unsupported protocol version: GET / HTTP/1.1

继续下一步：2.16 --> 3.0

出现错误ERROR com.google.gerrit.pgm.init.BaseInit : Couldn't upgrade schema. Expected if slave and read-only database

com.google.gerrit.exceptions.StorageException: You appear to be upgrading from a 2.x site, but the NoteDb change migration was not completed. See documentation:

https://gerrit-review.googlesource.com/Documentation/note-db.html#migration

根据文档，必须先进行 NoteDB 迁移

$ java -jar gerrit-2.16.25.war migrate-to-note-db -d ${gerrit-site}

这一步报错了： failed on some changes ,see log

于是需要把失败的 changid 的相关信息从数据库里删除掉；

# 举一个changeid 的例子

use database_name;

delete from patch_sets where change_id=183663;

delete from patch_set_approvals where change_id=183663;

delete from patch_comments where change_id=183663;

delete from changes where change_id=183663;

delete from change_messages where change_id=183663;

delete from change_id where s=183663;

... ... ...

执行完后，继续进行 migrate

为了提升效（偷）率（懒），写了个自动化脚本

接受命令行输入参数的脚本

#!/bin/bash 
#
#********************************************************************
#Author:                奈文摩尔
#ID:                    xxxxx
#Created:               2021-02-02
#Modified:              2021-02-04  10:27:27
#Email:                 
#Description:           Upgrade gerrit from 2.x to 3.x
#Copyright (C):         2021 All rights reserved
#********************************************************************
# run as work 
# Input Paras :
# 1:gerrit folder (local path)
# 2:dumped sql script (local path)
# 3:remote mysql host/port/user/password
``` Example:  
bash gerrit_upgrate.sh -g /home/work/gerrit -d /home/work/gerrit/c3-inf-gerrit-db01.bj.sql -h 10.172.28.40 -p 3315 -u root -w 123456
```
set -e
if [ ! -n "$1" ] ;then
    echo "invalid input , bash xxx  -help to get demo"
 exit -1
fi
if [[ $1 = "--help" ]] || [[ $1 = "-h" ]]
then
    echo "
    ----------------------------------------------------
    -g  gerrit folder path
    -d  dump sql script path
    -h  mysql host name
    -p  mysql host port
    -u  mysql login username
    -w  mysql username password
 
    Example:  
    bash gerrit_upgrate.sh -g /home/work/gerrit -d /home/work/gerrit/c3-inf-gerrit-db01.bj.sql -h 10.172.28.40 -p 3315 -u root -w 123456
    -----------------------------------------------------
 "
 exit 0
fi
while getopts "g:d:h:p:u:w:" opt; do
  case $opt in
    g)
        GERRIT_SITE=$OPTARG
        echo "GERRIT_SITE is $GERRIT_SITE" ;;
    d)
        DUMP_SQL=$OPTARG
        echo "DUMP_SQL is $DUMP_SQL" ;;
    h)
        DB_HOST=$OPTARG
        echo "DB_HOST is $DB_HOST" ;;
    p)
        DB_PORT=$OPTARG
        echo "DB_PORT is $DB_PORT" ;;
    u)
        DB_USER=$OPTARG
        echo "DB_USER is $DB_USER" ;;
    w)
        DB_PWD=$OPTARG
        echo "DB_PWD is $DB_PWD" ;;
    \?)
        echo "invalid arg" ;;
  esac
done
# step 0 : modify gerrit.config
cd ${GERRIT_SITE}/etc/
if [ -f gerrit.config ]; then
    mv gerrit.config gerrit-bak.config
fi
if [ -f secure.config ]; then
    mv secure.config secure-bak.config
fi
wget https://staging-cnbj2-fds.api.zzz.net/files/gerrit.config
wget https://staging-cnbj2-fds.api.zzz.net/files/secure.config
JDBC_STR="jdbc:mysql://${DB_HOST}:${DB_PORT}/reviewmiui?autoReconnect=true&characterEncoding=UTF-8&useSSL=false"
sed -i "0,/hostname =/{s/hostname =.*/hostname = ${DB_HOST}/}" gerrit.config
sed -i "0,/username =/{s/username =.*/username = ${DB_USER}/}" gerrit.config
sed -i "0,/password =/{s/password =.*/password = ${DB_PWD}/}" gerrit.config
sed -i "0,/port =/{s/port =.*/port = ${DB_PORT}/}" gerrit.config
sed -i "/url = /c\        url = jdbc:mysql://${DB_HOST}:${DB_PORT}/reviewmiui?autoReconnect=true&characterEncoding=UTF-8&useSSL=false" gerrit.config
sed -i "0,/password =/{s/password =.*/password = ${DB_PWD}/}" secure.config
# sed -i "s/hostname =/    hostname = ${DB_HOST}/g" gerrit.config
# sed -i "s/username =/    username = ${DB_USER}/g" gerrit.config
# sed -i "s/password =/    password = ${DB_PWD}/g" gerrit.config
# sed -i "s/port =/    port = ${DB_PORT}/g" gerrit.config
# sed -i "s/url =/    url = ${JDBC_STR}/g" gerrit.config
# sed -i "s/password =*/    password = ${DB_PWD}/g" secure.config
# step 1 : delete replication /  hA config
RPC_CONFIG=${GERRIT_SITE}/etc/replication.config
HA_CONFIG=${GERRIT_SITE}/etc/high-availability.config
rm -rf ${RPC_CONFIG}
rm -rf ${HA_CONFIG}
# delete confirm
if [ -f "${RPC_CONFIG}"] || [ -f "${HA_CONFIG}" ];
then
    echo "config file delete failed: ${RPC_CONFIG} or ${HA_CONFIG}"
 exit -1
fi
# step 2 : download pkg
cd ${GERRIT_SITE}
cd ..
rm -rf gerrit-*.war
wget https://staging-cnbj2-fds.api.zzzzz.net/files/gerrit-2.15.21.war
wget https://staging-cnbj2-fds.api.zzzzz.net/files/gerrit-2.16.25.war
wget https://staging-cnbj2-fds.api.zzzzz.net/files/gerrit-3.2.3.war
if [ ! -f "delete_dirty_changeids.sql" ]; then
    wget https://staging-cnbj2-fds.api.zzzzz.net/files/delete_dirty_changeids.sql
fi
# step 3 : Loading dump data into DB
echo "-------------------- split line ---------------------------"
echo "start loading data :$(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
mysql -h${DB_HOST}  -P${DB_PORT} -u${DB_USER} -p${DB_PWD} --verbose < ${DUMP_SQL}
mysql -h${DB_HOST}  -P${DB_PORT} -u${DB_USER} -p${DB_PWD} --verbose < delete_dirty_changeids.sql
# step 4 : upgrate from 2.13  --> 2.15
echo "start 2.15 :$(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
cd ${GERRIT_SITE}/lib 
wget https:///mysql-connector-java-5.1.43.jar
mkdir -p ${GERRIT_SITE}/ha/1/gerrit/shared
cd -
java -jar gerrit-2.15.21.war init --batch --dev -d ${GERRIT_SITE}
# step 5 : upgrate from 2.15 --> 2.16
echo "start 2.16 :$(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
cd ${GERRIT_SITE}/lib 
wget https://staging-cnbj2-fds.api.zzzzz.net/files/mysql-connector-java-5.1.48.jar
cd -
java -jar gerrit-2.16.25.war init --batch --dev -d ${GERRIT_SITE}
# step 6 : migrate on 2.16
echo "start migrate :$(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
rm -rf ${GERRIT_SITE}/etc/notedb.config
nohup java -jar gerrit-2.16.25.war migrate-to-note-db --reindex false -d ${GERRIT_SITE} > mi.log 2>&1 &
while true
do
    echo "Waiting for NoteDB Migration complete: $(date "+%Y-%m-%d %H:%M:%S")"
 if test ! -z "$(tail -n 20 mi.log | grep 'NoteDB Migration complete.')"; then
        echo "NoteDB Migration complete: $(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
 break
    fi
 if test ! -z "$(tail -n 20 mi.log | grep 'fatal: Migrating primary storage for some changes failed, see log')"; then
        echo "fatal: Migrating primary storage for some changes failed, see log: $(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
 exit -1
    fi
 sleep 1m
done
# step 7 : upgrate from 2.16  --> 3.2.3
echo "start 3.2.3 : $(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
java -jar gerrit-3.2.3.war init --batch --dev -d ${GERRIT_SITE}
# step 8 : reindex on 3.2.3
echo "Reindex accounts : $(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
java -jar gerrit-3.2.3.war reindex --index accounts -d ${GERRIT_SITE}
echo "Reindex changes : $(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
nohup java -jar gerrit-3.2.3.war reindex --index changes -d ${GERRIT_SITE} > "reindex-$(date "+%Y-%m-%d-%H-%M-%S").log" 2>&1 &

可固定参数的脚本：

#!/bin/bash
set -e

export GERRIT_SITE=/home/work/hdd1/gerrit_odm
cd ${GERRIT_SITE}/lib
rm -rf ${GERRIT_SITE}/lib/*
wget https://staging-cnbj2-fds.api.xiaomi.net/files/mysql-connector-java-5.1.43.jar

cd /home/work/hdd1
echo "start 2.15 :$(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
java -jar gerrit-2.15.21.war init --batch --dev -d ${GERRIT_SITE}


cd ${GERRIT_SITE}/lib
rm -rf ${GERRIT_SITE}/lib/*
wget https://staging-cnbj2-fds.api.xiaomi.net/files/mysql-connector-java-5.1.48.jar

cd -
echo "start 2.16 :$(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
java -jar gerrit-2.16.25.war init --batch --dev -d ${GERRIT_SITE}

echo "start migrate :$(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
nohup java -jar gerrit-2.16.25.war migrate-to-note-db --reindex false -d ${GERRIT_SITE} > mi.log 2>&1 &

while true
do
    echo "Waiting for NoteDB Migration complete: $(date "+%Y-%m-%d %H:%M:%S")"
    if test ! -z "$(tail -n 20 mi.log | grep 'NoteDB Migration complete.')"; then
        echo "NoteDB Migration complete: $(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
        break
    fi
    if test ! -z "$(tail -n 20 mi.log | grep 'fatal: Migrating primary storage for some changes failed, see log')"; then
        echo "fatal: Migrating primary storage for some changes failed, see log: $(date "+%Y-%m-%d %H:%M:%S")" >> upgrade_time.log
        exit -1
    fi
    sleep 1m
done

编辑于 2021-05-13 11:47