dolphinscheduler海豚调度（五）seatunnel案例

news2024/10/7 17:29:30

seatunnel作为新一代流行的数据集成工具，其功能非常强大且简单易用，今天演示一下如何通过dolphinscheduler创建并运行seatunnel任务

本次dolphinscheduler和seatunnel均部署在同一机器上的单机版本

1、环境配置

打开dolphinscheduler安装目录，编辑dolphinscheduler_env.sh，添加seatunnel的环境

vim bin/env/dolphinscheduler_env.sh

export SEATUNNEL_HOME=${SEATUNNEL_HOME:-/opt/seatunnel/apache-seatunnel-2.3.3}

2、创建seatunnel任务

在数据集成下选择seatunnel组件

3、编辑任务

本次采用seatunnel默认引擎，所以选择seatunnel.sh

需要选择部署模式，否则任务报错

编辑脚本

env {
  execution.parallelism = 2
  job.mode = "BATCH"
}
source {
    Jdbc {
        url = "jdbc:mysql://ip:3306/database"
        driver = "com.mysql.cj.jdbc.Driver"
        connection_check_timeout_sec = 100
        user = "user "
        password = "password"
        query = "select * from table1 limit 4"
    }
}

transform {
    # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
    # please go to https://seatunnel.apache.org/docs/transform/sql
}

sink {
  jdbc {
    url = "jdbc:mysql://ip:3306/database"
    driver = "com.mysql.cj.jdbc.Driver"
    user = "user"
    password = "password "
    query = "insert into table1 (id,name) values(?,?)"
  }
}