doris docker部署和本地化部署 1.2.4.1版本

news2024/9/29 19:24:02

写在前面

以下操作语句按顺序执行即可，注意切换目录的命令一定记得执行，如果需要改动的地方会有${}注释，其余不需要任何改动，默认安装版本为1.12.4（稳定版）

本地化部署

下载

# 创建目录
mkdir /data/soft/doris-1.2.4
cd /data/soft/doris-1.2.4
# 下载FE、BE、解压、更名
wget https://archive.apache.org/dist/doris/1.2/1.2.4.1-rc01/apache-doris-fe-1.2.4.1-bin-x86_64.tar.xz
wget https://archive.apache.org/dist/doris/1.2/1.2.4.1-rc01/apache-doris-be-1.2.4.1-bin-x86_64.tar.xz
tar -xvf apache-doris-fe-1.2.4.1-bin-x86_64.tar.xz
tar -xvf apache-doris-be-1.2.4.1-bin-x86_64.tar.xz
mv apache-doris-fe-1.2.4.1-bin-x86_64 fe
mv apache-doris-be-1.2.4.1-bin-x86_64 be

修改FE配置文件

# 修改FE配置文件
vi /data/soft/doris-1.2.4/fe/conf/fe.conf

# 添加DORIS_HOME
DORIS_HOME=/data/soft/doris-1.2.4/fe
# 打开注释
meta_dir = ${DORIS_HOME}/doris-meta
# 打开注释，并修改
priority_networks = ${fe的ip}/24

启动FE

# 启动后检查 http://${feip}:8030,正常访问代表安装成功，否则去掉--daemon 查看日志
/data/soft/doris-1.2.4/fe/bin/start-fe.sh --daemon

修改BE配置文件

# 修改BE配置文件
vi /data/soft/doris-1.2.4/be/conf/be.conf

# 添加DORIS_HOME
DORIS_HOME=/data/soft/doris-1.2.4/be
# 打开注释，并修改
priority_networks = ${be的ip}/24
# 打开注释
storage_root_path = ${DORIS_HOME}/storage 

 BE UDF依赖 

# 下载udf依赖
cd /data/soft/doris-1.2.4/fe/conf/lib
wget  https://archive.apache.org/dist/doris/1.2/1.2.4.1-rc01/apache-doris-dependencies-1.2.4.1-bin-x86_64.tar.xz
# 解压
tar -xvf apache-doris-dependencies-1.2.4.1-bin-x86_64.tar.xz
# 替换文件
mv apache-doris-dependencies-1.2.4.1-bin-x86_64/* ./

启动BE

/data/soft/doris-1.2.4/fe/bin/start-be.sh --daemon

注册BE

# 进入一台有mysql的服务器，端口9093，账号root，默认密码是空
mysql -h ${fe的ip} -P9030 -uroot 
# 注册be
ALTER SYSTEM ADD BACKEND "${be的ip}:9050";
# 查看be Alive属性如果是true就ok了
show PROC '/backends';

MySQL [(none)]> show PROC '/backends';
+-----------+-----------------+---------------+----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+-----------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+
| BackendId | Cluster         | IP            | HostName | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | RemoteUsedCapacity | Tag                      | ErrMsg | Version               | Status                                                                                                                        | HeartbeatFailureCounter | NodeRole |
+-----------+-----------------+---------------+----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+-----------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+
| 10003     | default_cluster | 172.16.10.142 | hdp04    | 9050          | 9060   | 8041     | 8060     | 2023-06-15 08:42:32 | 2023-06-15 08:42:45 | true  | false                | false                 | 0         | 0.000            | 713.640 GB    | 899.561 GB    | 20.67 % | 20.67 %        | 0.000              | {"location" : "default"} |        | doris-1.2.4-1-Unknown | {"lastSuccessReportTabletsTime":"2023-06-15 08:42:39","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
+-----------+-----------------+---------------+----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+-----------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+

##################以下是Docker部署################

镜像编译

准备工作

1、确认/opt/目录是否还有空间，至少需要10G，这个只是编译镜像的临时文件，会自动删除的，所以就安装在这个目录下，如果修改了，以下涉及opt的路径都要同步更改，非常不建议

df -h /opt

2、切换到root用户

sudo su root

3、创建文件夹

mkdir -p ~/docker-build/be/resource
mkdir -p ~/docker-build/fe/resource

下载编译好的包

fe安装包

cd ~/docker-build/fe/resource
# 下载fe编译包
wget https://archive.apache.org/dist/doris/1.2/1.2.4.1-rc01/apache-doris-fe-1.2.4.1-bin-x86_64.tar.xz
# 解压xz
xz -d apache-doris-fe-1.2.4.1-bin-x86_64.tar.xz
# 压缩为gz，最后会产出一个 apache-doris-fe-1.2.4.1-bin-x86_64.tar.gz
gizp apache-doris-fe-1.2.4.1-bin-x86_64.tar

be安装包

cd ~/docker-build/be/resource
# 下载be编译包
wget https://archive.apache.org/dist/doris/1.2/1.2.4.1-rc01/apache-doris-be-1.2.4.1-bin-x86_64.tar.xz
# 解压xz
xz -d apache-doris-be-1.2.4.1-bin-x86_64.tar.xz
# 压缩为gz，最后会产出一个 apache-doris-be-1.2.4.1-bin-x86_64.tar.gz
gizp apache-doris-be-1.2.4.1-bin-x86_64.tar

编辑初始化文件

init_fe.sh

cd ~/docker-build/fe/resource
# 输入以下文件内容
vi init_fe.sh

init_fe.sh内容，不需要任何改动

#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

set -eo pipefail
shopt -s nullglob

DORIS_HOME="/opt/apache-doris"

# Obtain necessary and basic information to complete initialization

# logging functions
# usage: doris_[note|warn|error] $log_meg
#    ie: doris_warn "task may fe risky!"
#   out: 2023-01-08T19:08:16+08:00 [Warn] [Entrypoint]: task may fe risky!
doris_log() {
    local type="$1"
    shift
    # accept argument string or stdin
    local text="$*"
    if [ "$#" -eq 0 ]; then text="$(cat)"; fi
    local dt="$(date -Iseconds)"
    printf '%s [%s] [Entrypoint]: %s\n' "$dt" "$type" "$text"
}
doris_note() {
    doris_log Note "$@"
}
doris_warn() {
    doris_log Warn "$@" >&2
}
doris_error() {
    doris_log ERROR "$@" >&2
    exit 1
}

# check to see if this file is being run or sourced from another script
_is_sourced() {
    [ "${#FUNCNAME[@]}" -ge 2 ] &&
        [ "${FUNCNAME[0]}" = '_is_sourced' ] &&
        [ "${FUNCNAME[1]}" = 'source' ]
}

docker_setup_env() {
    declare -g DATABASE_ALREADY_EXISTS BUILD_TYPE_K8S
    if [ -d "${DORIS_HOME}/fe/doris-meta/image" ]; then
        DATABASE_ALREADY_EXISTS='true'
    fi
}

# Check the variables required for   startup
docker_required_variables_env() {
    declare -g RUN_TYPE
    if [ -n "$BUILD_TYPE" ]; then
        RUN_TYPE="K8S"
        if [[ $BUILD_TYPE =~ ^([kK]8[sS])$ ]]; then
            doris_warn "BUILD_TYPE" $BUILD_TYPE
        else
            doris_error "BUILD_TYPE rule error！example: [k8s], Default Value: docker."
        fi
        return
    fi

    if [[ -n "$FE_SERVERS" && -n "$FE_ID" ]]; then
        RUN_TYPE="ELECTION"
        if [[ $FE_SERVERS =~ ^.+:[1-2]{0,1}[0-9]{0,1}[0-9]{1}(\.[1-2]{0,1}[0-9]{0,1}[0-9]{1}){3}:[1-6]{0,1}[0-9]{1,4}(,.+:[1-2]{0,1}[0-9]{0,1}[0-9]{1}(\.[1-2]{0,1}[0-9]{0,1}[0-9]{1}){3}:[1-6]{0,1}[0-9]{1,4})*$ || $FE_SERVERS =~ ^([0-9a-fA-F]{1,4}:){7,7}([0-9a-fA-F]{1,4}|:)|([0-9a-fA-F]{1,4}:){1,6}(:[0-9a-fA-F]{1,4}|:)|([0-9a-fA-F]{1,4}:){1,5}((:[0-9a-fA-F]{1,4}){1,2}|:)|([0-9a-fA-F]{1,4}:){1,4}((:[0-9a-fA-F]{1,4}){1,3}|:)|([0-9a-fA-F]{1,4}:){1,3}((:[0-9a-fA-F]{1,4}){1,4}|:)|([0-9a-fA-F]{1,4}:){1,2}((:[0-9a-fA-F]{1,4}){1,5}|:)|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6}|:)|:((:[0-9a-fA-F]{1,4}){1,7}|:)$ ]]; then
            doris_warn "FE_SERVERS" $FE_SERVERS
        else
            doris_error "FE_SERVERS rule error！example: \$FE_NAME:\$FE_HOST_IP:\$FE_EDIT_LOG_PORT[,\$FE_NAME:\$FE_HOST_IP:\$FE_EDIT_LOG_PORT]..."
        fi
        if [[ $FE_ID =~ ^[1-9]{1}$ ]]; then
            doris_warn "FE_ID" $FE_ID
        else
            doris_error "FE_ID rule error！If FE is the role of Master, please set FE_ID=1, and ensure that all IDs correspond to the IP of the current node."
        fi
        return
    fi
    doris_note $FE_MASTER_IP "  " $FE_MASTER_PORT "  " $FE_CURRENT_IP "  " $FE_CURRENT_PORT
    if [[ -n "$FE_MASTER_IP" && -n "$FE_MASTER_PORT" && -n "$FE_CURRENT_IP" && -n "$FE_CURRENT_PORT" ]]; then
        RUN_TYPE="ASSIGN"
        if [[ $FE_MASTER_IP =~ ^[1-2]{0,1}[0-9]{0,1}[0-9]{1}(\.[1-2]{0,1}[0-9]{0,1}[0-9]{1}){3}$ || $FE_MASTER_IP =~ ^([0-9a-fA-F]{1,4}:){7,7}([0-9a-fA-F]{1,4}|:)|([0-9a-fA-F]{1,4}:){1,6}(:[0-9a-fA-F]{1,4}|:)|([0-9a-fA-F]{1,4}:){1,5}((:[0-9a-fA-F]{1,4}){1,2}|:)|([0-9a-fA-F]{1,4}:){1,4}((:[0-9a-fA-F]{1,4}){1,3}|:)|([0-9a-fA-F]{1,4}:){1,3}((:[0-9a-fA-F]{1,4}){1,4}|:)|([0-9a-fA-F]{1,4}:){1,2}((:[0-9a-fA-F]{1,4}){1,5}|:)|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6}|:)|:((:[0-9a-fA-F]{1,4}){1,7}|:)$ ]]; then
            doris_warn "FE_MASTER_IP" $FE_MASTER_IP
        else
            doris_error "FE_MASTER_IP rule error！example: \$FE_MASTER_IP"
        fi
        if [[ $FE_MASTER_PORT =~ ^[1-6]{0,1}[0-9]{1,4}$ ]]; then
            doris_warn "FE_MASTER_PORT" $FE_MASTER_PORT
        else
            doris_error "FE_MASTER_PORT rule error！example: \$FE_MASTER_EDIT_LOG_HOST."
        fi
        if [[ $FE_CURRENT_IP =~ ^[1-2]{0,1}[0-9]{0,1}[0-9]{1}(\.[1-2]{0,1}[0-9]{0,1}[0-9]{1}){3}$ || $FE_CURRENT_IP =~ ^([0-9a-fA-F]{1,4}:){7,7}([0-9a-fA-F]{1,4}|:)|([0-9a-fA-F]{1,4}:){1,6}(:[0-9a-fA-F]{1,4}|:)|([0-9a-fA-F]{1,4}:){1,5}((:[0-9a-fA-F]{1,4}){1,2}|:)|([0-9a-fA-F]{1,4}:){1,4}((:[0-9a-fA-F]{1,4}){1,3}|:)|([0-9a-fA-F]{1,4}:){1,3}((:[0-9a-fA-F]{1,4}){1,4}|:)|([0-9a-fA-F]{1,4}:){1,2}((:[0-9a-fA-F]{1,4}){1,5}|:)|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6}|:)|:((:[0-9a-fA-F]{1,4}){1,7}|:)$ ]]; then
            doris_warn "FE_CURRENT_IP" $FE_CURRENT_IP
        else
            doris_error "FE_CURRENT_IP rule error！example: \$FE_CURRENT_IP"
        fi
        if [[ $FE_CURRENT_PORT =~ ^[1-6]{0,1}[0-9]{1,4}$ ]]; then
            doris_warn "FE_CURRENT_PORT" $FE_CURRENT_PORT
        else
            doris_error "FE_CURRENT_PORT rule error！example: \$FE_CURRENT_EDIT_LOG_HOST."
        fi
        return
    fi

    doris_error EOF "
                 Note that you did not configure the required parameters!
                 plan 1:
                 BUILD_TYPE
                 plan 2:
                 FE_SERVERS & FE_ID
                 plan 3:
                 FE_MASTER_IP & FE_MASTER_PORT & FE_CURRENT_IP & FE_CURRENT_PORT"
                EOF

}

get_doris_fe_args() {
    declare -g MASTER_FE_IP CURRENT_FE_IP MASTER_FE_EDIT_PORT CURRENT_FE_EDIT_PORT PRIORITY_NETWORKS CURRENT_FE_IS_MASTER
    if [ $RUN_TYPE == "ELECTION" ]; then
        local feServerArray=($(echo "${FE_SERVERS}" | awk '{gsub (/,/," "); print $0}'))
        for i in "${feServerArray[@]}"; do
            val=${i}
            val=${val// /}
            tmpFeName=$(echo "${val}" | awk -F ':' '{ sub(/fe/, ""); sub(/ /, ""); print$1}')
            tmpFeIp=$(echo "${val}" | awk -F ':' '{ sub(/ /, ""); print$2}')
            tmpFeEditLogPort=$(echo "${val}" | awk -F ':' '{ sub(/ /, ""); print$3}')
            check_arg "TMP_FE_NAME" $tmpFeName
            feIpArray[$tmpFeName]=${tmpFeIp}
            check_arg "TMP_FE_EDIT_LOG_PORT" $tmpFeEditLogPort
            feEditLogPortArray[$tmpFeName]=${tmpFeEditLogPort}
        done

        MASTER_FE_IP=${feIpArray[1]}
        check_arg "MASTER_FE_IP" $MASTER_FE_IP
        MASTER_FE_EDIT_PORT=${feEditLogPortArray[1]}
        check_arg "MASTER_FE_EDIT_PORT" $MASTER_FE_EDIT_PORT
        CURRENT_FE_IP=${feIpArray[FE_ID]}
        check_arg "CURRENT_FE_IP" $CURRENT_FE_IP
        CURRENT_FE_EDIT_PORT=${feEditLogPortArray[FE_ID]}
        check_arg "CURRENT_FE_EDIT_PORT" $CURRENT_FE_EDIT_PORT

        if [ ${MASTER_FE_IP} == ${CURRENT_FE_IP} ]; then
            CURRENT_FE_IS_MASTER=true
        else
            CURRENT_FE_IS_MASTER=false
        fi

        PRIORITY_NETWORKS=$(echo "${CURRENT_FE_IP}" | awk -F '.' '{print$1"."$2"."$3".0/24"}')
        check_arg "PRIORITY_NETWORKS" $PRIORITY_NETWORKS

        doris_note "FE_IP_ARRAY = ${feIpArray[*]}"
        doris_note "FE_EDIT_LOG_PORT_ARRAY = ${feEditLogPortArray[*]}"
        doris_note "MASTER_FE = ${feIpArray[1]}:${feEditLogPortArray[1]}"
        doris_note "CURRENT_FE = ${CURRENT_FE_IP}:${CURRENT_FE_EDIT_PORT}"
        doris_note "PRIORITY_NETWORKS = ${PRIORITY_NETWORKS}"

    elif [ $RUN_TYPE == "ASSIGN" ]; then
        MASTER_FE_IP=${FE_MASTER_IP}
        check_arg "MASTER_FE_IP" $MASTER_FE_IP
        MASTER_FE_EDIT_PORT=${FE_MASTER_PORT}
        check_arg "MASTER_FE_EDIT_PORT" $MASTER_FE_EDIT_PORT
        CURRENT_FE_IP=${FE_CURRENT_IP}
        check_arg "CURRENT_FE_IP" $CURRENT_FE_IP
        CURRENT_FE_EDIT_PORT=${FE_CURRENT_PORT}
        check_arg "CURRENT_FE_EDIT_PORT" $CURRENT_FE_EDIT_PORT

        if [ ${MASTER_FE_IP} == ${CURRENT_FE_IP} ]; then
            CURRENT_FE_IS_MASTER=true
        else
            CURRENT_FE_IS_MASTER=false
        fi

        PRIORITY_NETWORKS=$(echo "${CURRENT_FE_IP}" | awk -F '.' '{print$1"."$2"."$3".0/24"}')
        check_arg "PRIORITY_NETWORKS" $PRIORITY_NETWORKS
    fi

    # check fe start
    check_fe_status true

}

add_priority_networks() {
    doris_note "add priority_networks ${1} to ${DORIS_HOME}/fe/conf/fe.conf"
    echo "priority_networks = ${1}" >>${DORIS_HOME}/fe/conf/fe.conf
}

# Execute sql script, passed via stdin
# usage: docker_process_sql sql_script
docker_process_sql() {
    set +e
    mysql -uroot -P9030 -h${MASTER_FE_IP} --comments "$@" 2>/dev/null
}

docker_setup_db() {
    set +e
    # check fe status
    local is_fe_start=false
    if [ ${CURRENT_FE_IS_MASTER} == true ]; then
        doris_note "Current FE is Master FE!  No need to register again！"
        return
    fi
    for i in {1..300}; do
        docker_process_sql <<<"alter system add FOLLOWER '${CURRENT_FE_IP}:${CURRENT_FE_EDIT_PORT}'"
        register_fe_status=$?
        if [[ $register_fe_status == 0 ]]; then
            doris_note "FE successfully registered！"
            is_fe_start=true
            break
        else
            check_fe_status
            if [ -n "$CURRENT_FE_ALREADY_EXISTS" ]; then
                doris_warn "Same frontend already exists! No need to register again！"
                break
            fi
            if [[ $(($i % 20)) == 1 ]]; then
                doris_warn "register_fe_status: ${register_fe_status}"
                doris_warn "FE failed registered!"
            fi
        fi
        if [[ $(($i % 20)) == 1 ]]; then
            doris_note "ADD FOLLOWER failed, retry."
        fi
        sleep 1
    done
    if ! [[ $is_fe_start ]]; then
        doris_error "Failed to register CURRENT_FE to FE！Tried 30 times！Maybe FE Start Failed！"
    fi
}

# Check whether the passed parameters are empty to avoid subsequent task execution failures. At the same time,
# enumeration checks can fe added, such as checking whether a certain parameter appears repeatedly, etc.
check_arg() {
    if [ -z $2 ]; then
        doris_error "$1 is null!"
    fi
}


check_fe_status() {
    set +e
    declare -g CURRENT_FE_ALREADY_EXISTS
    if [[ ${CURRENT_FE_IS_MASTER} == true ]]; then
        doris_note "Current FE is Master FE!  No need check fe status！"
        return
    fi
    for i in {1..300}; do
        if [[ $1 == true ]]; then
            docker_process_sql <<<"show frontends" | grep "[[:space:]]${MASTER_FE_IP}[[:space:]]" | grep "[[:space:]]${MASTER_FE_EDIT_PORT}[[:space:]]"
        else
            docker_process_sql <<<"show frontends" | grep "[[:space:]]${CURRENT_FE_IP}[[:space:]]" | grep "[[:space:]]${CURRENT_FE_EDIT_PORT}[[:space:]]"
        fi
        fe_join_status=$?
        if [[ "${fe_join_status}" == 0 ]]; then
            if [[ $1 == true ]]; then
                doris_note "Master FE is started!"
            else
                doris_note "Verify that CURRENT_FE is registered to FE successfully"
            fi
            CURRENT_FE_ALREADY_EXISTS=true
            break
        else
            if [[ $(($i % 20)) == 1 ]]; then
                if [[ $1 == true ]]; then
                    doris_note "Master FE is not started, retry."
                else
                    doris_warn "Verify that CURRENT_FE is registered to FE failed, retry."
                fi
            fi
        fi
        if [[ $(($i % 20)) == 1 ]]; then
            doris_note "try session Master FE."
        fi
        sleep 1
    done
}

cleanup() {
    doris_note "Container stopped, running stop_fe script"
    ${DORIS_HOME}/fe/bin/stop_fe.sh
}

_main() {
    docker_required_variables_env
    trap 'cleanup' SIGTERM SIGINT
    if [[ $RUN_TYPE == "K8S" ]]; then
        start_fe.sh &
        child_pid=$!
    else
        docker_setup_env
        get_doris_fe_args

        if [ -z "$DATABASE_ALREADY_EXISTS" ]; then
            add_priority_networks $PRIORITY_NETWORKS
        fi

        docker_setup_db
        check_fe_status
        doris_note "Ready to start CURRENT_FE！"

        if [ $CURRENT_FE_IS_MASTER == true ]; then
            start_fe.sh &
            child_pid=$!
        else
            start_fe.sh --helper ${MASTER_FE_IP}:${MASTER_FE_EDIT_PORT} &
            child_pid=$!
        fi
    fi
    wait $child_pid
    exec "$@"
}

if ! _is_sourced; then
    _main "$@"
fi

init_be.sh

cd ~/docker-build/be/resource
# 输入以下内容
vi init_be.sh

init_be.sh内容，不需要任何改动

#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

set -eo pipefail
shopt -s nullglob

DORIS_HOME="/opt/apache-doris"

# Obtain necessary and basic information to complete initialization

# logging functions
# usage: doris_[note|warn|error] $log_meg
#    ie: doris_warn "task may be risky!"
#   out: 2023-01-08T19:08:16+08:00 [Warn] [Entrypoint]: task may be risky!
doris_log() {
  local type="$1"
  shift
  # accept argument string or stdin
  local text="$*"
  if [ "$#" -eq 0 ]; then text="$(cat)"; fi
  local dt="$(date -Iseconds)"
  printf '%s [%s] [Entrypoint]: %s\n' "$dt" "$type" "$text"
}
doris_note() {
  doris_log Note "$@"
}
doris_warn() {
  doris_log Warn "$@" >&2
}
doris_error() {
  doris_log ERROR "$@" >&2
  exit 1
}

# check to see if this file is being run or sourced from another script
_is_sourced() {
  [ "${#FUNCNAME[@]}" -ge 2 ] &&
    [ "${FUNCNAME[0]}" = '_is_sourced' ] &&
    [ "${FUNCNAME[1]}" = 'source' ]
}

docker_setup_env() {
  declare -g DATABASE_ALREADY_EXISTS
  if [ -d "${DORIS_HOME}/be/storage/data" ]; then
    DATABASE_ALREADY_EXISTS='true'
  fi
}

add_priority_networks() {
  doris_note "add priority_networks ${1} to ${DORIS_HOME}/be/conf/be.conf"
  echo "priority_networks = ${1}" >>${DORIS_HOME}/be/conf/be.conf
}

show_be_args(){
  doris_note "============= init args ================"
  doris_note "MASTER_FE_IP " ${MASTER_FE_IP}
  doris_note "CURRENT_BE_IP " ${CURRENT_BE_IP}
  doris_note "CURRENT_BE_PORT " ${CURRENT_BE_PORT}
  doris_note "RUN_TYPE " ${RUN_TYPE}
  doris_note "PRIORITY_NETWORKS " ${PRIORITY_NETWORKS}
}

# Execute sql script, passed via stdin
# usage: docker_process_sql sql_script
docker_process_sql() {
  set +e
  mysql -uroot -P9030 -h${MASTER_FE_IP} --comments "$@" 2>/dev/null
}

node_role_conf(){
  if [[ ${NODE_ROLE} == 'computation' ]]; then
    doris_note "this node role is computation"
    echo "be_node_role=computation" >>${DORIS_HOME}/be/conf/be.conf
  else
    doris_note "this node role is mix"
  fi
}

register_be_to_fe() {
  set +e
  # check fe status
  local is_fe_start=false
  for i in {1..300}; do
    docker_process_sql <<<"alter system add backend '${CURRENT_BE_IP}:${CURRENT_BE_PORT}'"
    register_be_status=$?
    if [[ $register_be_status == 0 ]]; then
      doris_note "BE successfully registered to FE！"
      is_fe_start=true
      break
    else
      check_be_status
      if [ -n "$BE_ALREADY_EXISTS" ]; then
        doris_warn "Same backend already exists! No need to register again！"
        break
      fi
      if [[ $(( $i % 20 )) == 1 ]]; then
          doris_warn "register_be_status: ${register_be_status}"
          doris_warn "BE failed registered to FE!"
      fi
    fi
    if [[ $(( $i % 20 )) == 1 ]]; then
      doris_note "Register BE to FE is failed. retry."
    fi
    sleep 1
  done
  if ! [[ $is_fe_start ]]; then
    doris_error "Failed to register BE to FE！Tried 30 times！Maybe FE Start Failed！"
  fi
}

check_be_status() {
    set +e
    local is_fe_start=false
    for i in {1..300}; do
        if [[ $(($i % 20)) == 1 ]]; then
            doris_warn "start check be register status~"
        fi
        docker_process_sql <<<"show backends;" | grep "[[:space:]]${CURRENT_BE_IP}[[:space:]]" | grep "[[:space:]]${CURRENT_BE_PORT}[[:space:]]"
        be_join_status=$?
        if [[ "${be_join_status}" == 0 ]]; then
            doris_note "Verify that BE is registered to FE successfully"
            is_fe_start=true
            return
        else
            if [[ $(($i % 20)) == 1 ]]; then
                doris_note "register is failed, wait next~"
            fi
        fi
        sleep 1
    done
    if [[ ! $is_fe_start ]]; then
        doris_error "Failed to register BE to FE！Tried 30 times！Maybe FE Start Failed！"
    fi
}

cleanup() {
    doris_note "Container stopped, running stop_be script"
    ${DORIS_HOME}/be/bin/stop_be.sh
}

_main() {
    trap 'cleanup' SIGTERM SIGINT
    if [[ $RUN_TYPE == "K8S" ]]; then
        start_be.sh &
        child_pid=$!
    else
        docker_setup_env
        if [ -z "$DATABASE_ALREADY_EXISTS" ]; then
          add_priority_networks $PRIORITY_NETWORKS
          node_role_conf
          show_be_args
          register_be_to_fe
        fi
        check_be_status
        doris_note "Ready to start BE！"
        start_be.sh &
        child_pid=$!
    fi
    wait $child_pid
    exec "$@"
}

if ! _is_sourced; then
  _main "$@"
fi

编辑Dockerfile

fe-Dockerfile

cd ~/docker-build/fe/
# 输入以下内容
vi Dockerfile

Dockerfile(fe)内容，不需要任何改动

FROM openjdk:8u342-jdk

# 设置环境变量
ENV JAVA_HOME="/usr/local/openjdk-8/" \
    PATH="/opt/apache-doris/fe/bin:$PATH"

# 下载软件至镜像内，可根据需要替换
ADD ./resource/apache-doris-fe-1.2.4.1-bin-x86_64.tar.gz /opt/

RUN apt-get update && \
    apt-get install -y default-mysql-client && \
    apt-get clean && \
    mkdir /opt/apache-doris && \
    cd /opt && \
    mv apache-doris-fe-1.2.4.1-bin-x86_64 /opt/apache-doris/fe

ADD ./resource/init_fe.sh /opt/apache-doris/fe/bin
RUN chmod 755 /opt/apache-doris/fe/bin/init_fe.sh

ENTRYPOINT ["/opt/apache-doris/fe/bin/init_fe.sh"]

be-Dockerfile

cd ~/docker-build/be/
# 输入以下内容
vi Dockerfile

 Dockerfile(be)内容，不需要任何改动

# 选择基础镜像
FROM openjdk:8u342-jdk

# 设置环境变量
ENV JAVA_HOME="/usr/local/openjdk-8/" \
    PATH="/opt/apache-doris/be/bin:$PATH"

# 下载软件至镜像内，可根据需要替换
ADD ./resource/apache-doris-be-1.2.4.1-bin-x86_64.tar.gz /opt/

RUN apt-get update && \
    apt-get install -y default-mysql-client && \
    apt-get clean && \
    mkdir /opt/apache-doris && \
    cd /opt && \
    mv apache-doris-be-1.2.4.1-bin-x86_64 /opt/apache-doris/be

ADD ./resource/init_be.sh /opt/apache-doris/be/bin
RUN chmod 755 /opt/apache-doris/be/bin/init_be.sh

ENTRYPOINT ["/opt/apache-doris/be/bin/init_be.sh"]

构建镜像

doris-fe

cd ~/docker-builder/fe
docker build . -t apache-doris:1.2.4-fe

doris-be

cd ~/docker-builder/be
docker build . -t apache-doris:1.2.4-be

构建结果

[root@hdp04 be]# docker images
REPOSITORY                            TAG        IMAGE ID       CREATED         SIZE
apache-doris                          1.2.4-be   0d8aadf01bf0   6 hours ago     7.76GB
apache-doris                          1.2.4-fe   2f5961488ce7   6 hours ago     1.5GB

镜像运行

doris-fe

docker run -itd \
--name=doris-fe \
--env FE_SERVERS="fe1:${服务器ip}:9010" \
--env FE_ID=1 \
-p 8030:8030 \
-p 9030:9030 \
-v /data/docker/doris/fe/log:/opt/apache-doris/fe/log \
--net=host \
apache-doris:1.2.4-fe

doris-be

注意：如果你要重新run请先删除 /data/docker/doris/be/storage

docker run -itd \
--name=doris-be \
--env FE_SERVERS="fe1:${服务器ip}:9010" \
--env BE_ADDR="${服务器ip}:9050" \
-p 8040:8040 \
-v /data/docker/doris/be/storage:/opt/apache-doris/be/storage \
-v /data/docker/doris/be/log:/opt/apache-doris/be/log \
--net=host \
apache-doris:1.2.4-be

修改Be配置

[root@hdp04 be]# docker ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED             STATUS             PORTS                                                                          NAMES
f198db1d315c   apache-doris:1.2.4-be   "/data/soft/apache-d…"   51 minutes ago      Up 51 minutes                                                                                     doris-be
2b7970de5fff   apache-doris:1.2.4-fe   "/data/soft/apache-d…"   About an hour ago   Up About an hour                                                                                  doris-fe

# 进入fe容器
docker exec -it doris-fe /bin/bash  
# 查看fe的网段配置priority_networks，并复制出来 (priority_networks = 172.16.10.0/24)
cat /opt/apache-doris/fe/conf/fe.conf
# 退出fe容器
exit;

# 进入be容器
docker exec -it doris-fe /bin/bash  
# 下载vim
apt-get install vim
vi /opt/apache-doris/be/conf/be.conf
# 修改fe的网段配置（与fe一致）
priority_networks = 172.16.10.0/24

添加udf依赖

# 进入be容器
docker exec -it doris-fe /bin/bash  
# 进入lib目录
cd /opt/apache-doris/be/lib
# 下载udf依赖
wget  https://archive.apache.org/dist/doris/1.2/1.2.4.1-rc01/apache-doris-dependencies-1.2.4.1-bin-x86_64.tar.xz
# 解压
tar -xvf apache-doris-dependencies-1.2.4.1-bin-x86_64.tar.xz
# 替换文件
mv apache-doris-dependencies-1.2.4.1-bin-x86_64/* ./

启动be

be是需要启动完容器后，手动运行的，fe是随容器启动自启的

# 进入be容器
docker exec -it doris-fe /bin/bash  
# 进入lib目录  --daemon 后台运行，如果不放心先去掉，前台运行看日志
/opt/apache-doris/be/bin/start_be.sh --daemon

注册be

# 进入一台有mysql的服务器，端口9093，账号root，默认密码是空
mysql -h ${fe的ip} -P9030 -uroot 
# 注册be
ALTER SYSTEM ADD BACKEND "${be的ip}:9050";
# 查看be Alive属性如果是true就ok了
show PROC '/backends';

MySQL [(none)]> show PROC '/backends';
+-----------+-----------------+---------------+----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+-----------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+
| BackendId | Cluster         | IP            | HostName | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | RemoteUsedCapacity | Tag                      | ErrMsg | Version               | Status                                                                                                                        | HeartbeatFailureCounter | NodeRole |
+-----------+-----------------+---------------+----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+-----------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+
| 10003     | default_cluster | 172.16.10.142 | hdp04    | 9050          | 9060   | 8041     | 8060     | 2023-06-15 08:42:32 | 2023-06-15 08:42:45 | true  | false                | false                 | 0         | 0.000            | 713.640 GB    | 899.561 GB    | 20.67 % | 20.67 %        | 0.000              | {"location" : "default"} |        | doris-1.2.4-1-Unknown | {"lastSuccessReportTabletsTime":"2023-06-15 08:42:39","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
+-----------+-----------------+---------------+----------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+-----------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+

doris连接测试

账号root，默认密码为空

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.coloradmin.cn/o/670726.html

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈，一经查实，立即删除！

相关文章

软件测试日常工作和前景是怎么样的？

软件测试日常工作和前景是怎么样的？

笔者从测试的工作情况&#xff0c;职业发展&#xff0c;还有测试的工作日常等等来给大家讲解一下软件测试到底是什么样的工作&#xff1f; 通俗来说软件测试工程师就相当于一个质检员&#xff0c;专门处理软件测试质量的工作&#xff0c;不管是功能测试也好&#xff0c;性能测…
阅读更多...
BK7231N开发平台原厂烧录工具使用说明

BK7231N开发平台原厂烧录工具使用说明

BK7231N开发平台原厂烧录工具使用说明 烧录流程介绍 1.打开原厂烧录工具 以管理员身份打开名为 bk_writer_gui_V1.6.3.exe 的可执行文件。 2. 烧录对象 烧录对象选择 BK7231n 3.烧录地址 当我们烧录UA文件的时候&#xff0c;需要把起始地址设置为&#xff1a; 0X00011000。…
阅读更多...
Windows提示“找不到rgss202j.dll”怎么办？

Windows提示“找不到rgss202j.dll”怎么办？

Rgss202j.dll文件是Windows操作系统最重要的系统文件之一&#xff0c;它包含了一组程序和驱动函数。如果此文件丢失或损坏&#xff0c;驱动程序将无法正常工作&#xff0c;并且相应的应用程序也将无法正常启动且运行。通常情况下&#xff0c;造成Rgss202j.dll文件无法找到的原因…
阅读更多...
爬虫 - ProtoBuf 协议

爬虫 - ProtoBuf 协议

一、抓取请求 以下是请求的大致内容&#xff1a; 是乱码&#xff0c;需要解析。 二、解析 通过分析 request 和 response 的 Content-Type: application/x-protobuf 得知&#xff1a;使用了谷歌的 protobuf 协议来传输数据&#xff0c;需要破解。 大致破解过程&#xff…
阅读更多...
随时随地保持连接：数字游民适用的远程桌面

随时随地保持连接：数字游民适用的远程桌面

随着世界迅速适应数字革命&#xff0c;一种全新的职业——数字游民应运而生。数字游民指利用技术远程办公的专业人群&#xff0c;这是一种允许人们在旅行中办公、不受地点限制的工作生活方式。游牧式工作生活趋势并非一时的风尚&#xff0c;而是我们工作观念的彻底转变&#xf…
阅读更多...
MUR8060PT-ASEMI快恢复二极管MUR8060PT

MUR8060PT-ASEMI快恢复二极管MUR8060PT

编辑-Z MUR8060PT在TO-247封装里采用的2个芯片&#xff0c;其尺寸都是140MIL&#xff0c;是一款高耐压大电流快恢复二极管。MUR8060PT的浪涌电流Ifsm为600A&#xff0c;漏电流(Ir)为10uA&#xff0c;其工作时耐温度范围为-55~150摄氏度。MUR8060PT采用抗冲击硅芯片材质&#x…
阅读更多...
实在智能率先拥抱大模型技术，旗下“AI+RPA”系列产品迎来全面智能升级

实在智能率先拥抱大模型技术，旗下“AI+RPA”系列产品迎来全面智能升级

实在智能RPA ​ AI时代&#xff0c;所有产品都将迎来用大模型进行全面智能升级。 随着以ChatGPT等为代表的生成式AI持续火热&#xff0c;大型语言模型&#xff08;Large Language Model, LLM&#xff09;领域的研发和布局在国内外有目共睹&#xff0c;微软、谷歌、百度系等生…
阅读更多...
回收小程序是什么？有什么特点？

回收小程序是什么？有什么特点？

回收小程序旨在为用户提供便捷、环保的废品回收服务。以下是关于上门回收小程序的介绍&#xff1a;回收小程序旨在解决废品回收的难题&#xff0c;为用户提供一种方便、可持续的回收方式。通过小程序&#xff0c;可以轻松预约回收服务&#xff0c;将废品交由专业回收人员处理&a…
阅读更多...
基于QFT的量子加法器的原理与实现-mindspore quantum

基于QFT的量子加法器的原理与实现-mindspore quantum

1 量子Fourier变换 离散Fourier变换以一一个复向量 x 0 , . . . , x N − 1 {x_0},...,{x_{N - 1}} x0​,...,xN−1​为输入&#xff0c;输出的数据是如下复向量 y 0 , . . . , y N − 1 {y_0},...,{y_{N - 1}} y0​,...,yN−1​&#xff1a; y k ≡ 1 N ∑ j 0 N − 1 x j …
阅读更多...
【初识C语言】变量和常量

【初识C语言】变量和常量

文章目录 1. 局部变量和全局变量2. 变量的作用域和生命周期3. 常量 生活中的有些值是不变的&#xff08;比如&#xff1a;圆周率&#xff0c;性别&#xff0c;身份证号码&#xff0c;血型等等&#xff09;有些值是可变的&#xff08;比如&#xff1a;年龄&#xff0c;体重&…
阅读更多...
18款奔驰S450 4MATIC升级发光出风口，提升车内氛围感

18款奔驰S450 4MATIC升级发光出风口，提升车内氛围感

完美匹配&#xff0c;全部都是原装位&#xff0c;安装很快&#xff0c;瞬间发光。随着氛围灯颜色的变化而变化。美丽靓丽与质感同存&#xff0c;大大提升了车的颜值档次。让您车生活更加的富有乐趣与满足&#xff01;
阅读更多...
决策树笔记

决策树笔记

决策树模型学习过程 1.根节点选择什么特征 2.只关注决策树的左侧分支 3.再关注右侧分支 纯度 熵就是样本的混乱程度&#xff0c;熵越大&#xff0c;越混乱&#xff0c;纯度就越低 减小熵信息增益 选择拆分信息增益 选择信息增益大的&#xff0c;以增加纯度 p1left 定义为…
阅读更多...
2023年功能测试会被淘汰？自动化测试到测试开发？我的测试之路...

2023年功能测试会被淘汰？自动化测试到测试开发？我的测试之路...

目录&#xff1a;导读 前言一、Python编程入门到精通二、接口自动化项目实战三、Web自动化项目实战四、App自动化项目实战五、一线大厂简历六、测试开发DevOps体系七、常用自动化测试工具八、JMeter性能测试九、总结&#xff08;尾部小惊喜&#xff09; 前言 现在2023年&#…
阅读更多...
数据技术在金融行业有哪些应用_光点科技

数据技术在金融行业有哪些应用_光点科技

随着信息技术的迅猛发展&#xff0c;大数据技术逐渐成为金融行业的重要工具。大数据技术的应用&#xff0c;不仅可以提高金融机构的运营效率&#xff0c;还能够提供更准确的风险评估和预测&#xff0c;从而为投资者和决策者提供更好的决策依据。 那么&#xff0c;大数据技术在…
阅读更多...
ES应用风暴

ES应用风暴

文章目录 使用案例启发使用ElasticSearch作为主要的后端&#xff0c;作为搜索引擎在现有系统增加ElasticSearch使用ElasticSearch和现有的工具 常用于日志收集&#xff08;分析&#xff09;系统、实时警报 使用案例 维基百科&#xff1a;全文检索、高亮、搜索推荐。The Guar…
阅读更多...
【无标题】Instant NGP（使用哈希编码的多分辨率的即时神经图形原语）

【无标题】Instant NGP（使用哈希编码的多分辨率的即时神经图形原语）

论文基本信息 作者&#xff1a;THOMAS MLLER&#xff0c;NVIDIA&#xff0c;瑞士ALEX EV ANS&#xff0c;NVIDIA&#xff0c;英国CHRISTOPH SCHIED&#xff0c;美国NVIDIA ALEXANDER KELLER&#xff0c;德国 关键词&#xff1a; Image Synthesis, Neural Networks, En- codings…
阅读更多...
[230606] 听力TPO66汇总·最终篇｜L3-WATM Wether Animals have a Theory Mind｜19:30-21:30

[230606] 听力TPO66汇总·最终篇｜L3-WATM Wether Animals have a Theory Mind｜19:30-21:30

[230605] 听力TPO66汇总下篇&#xff5c;L2-Fish Movement&#xff5c;13:00~14:0015:20~17:00 http://t.csdn.cn/jxqyr [230604] 听力TPO66汇总上篇&#xff5c; C1 L1 C2&#xff5c;10:20~12:00 http://t.csdn.cn/zyEGw 极度缺少相关背景知识
阅读更多...
蓝精灵协会 | 23 年 6 月通讯

蓝精灵协会 | 23 年 6 月通讯

您好&#xff0c;Smurf 朋友们&#xff0c;欢迎阅读我们的新版时事通讯。它旨在分享来自社区及其项目的最新更新。我们希望您会喜欢它&#xff01; 水晶特卖&#xff1a;折扣从 6 月 19 日开始 根据社区的投票&#xff0c;我们延长了水晶薄荷的持续时间&#xff0c;并减少了折扣…
阅读更多...
这所天津的热门院校，保护一志愿，曾连续两年专业课题目完全相同，平均140分！

这所天津的热门院校，保护一志愿，曾连续两年专业课题目完全相同，平均140分！

一、学校及专业介绍 天津工业大学&#xff08;Tiangong University&#xff09;&#xff0c;简称“天工大”&#xff0c;位于天津市&#xff0c;是教育部与天津市共建高校、国家国防科技工业局和天津市共建的天津市重点建设高校、国家“双一流”建设高校、天津市高水平特色大学…
阅读更多...
Python 常用内置函数与匿名函数的应用

Python 常用内置函数与匿名函数的应用

目录 一、前言二、常用内置函数2.1 max()2.2 enumerate()2.3 map()3.4 reduce()2.5 filter()2.6 sorted() 一、前言 我们知道&#xff0c;Python 函数总体可分为两类&#xff0c;一类是标准函数&#xff0c;一类是匿名函数。其中标准函数中又可细分为内置标准函数、自定义标准…
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023