安装

Iceberg 是一种适用于大型分析表的高性能工具，通过spark启动并运行iceberg，文章是通过docker来进行安装并测试的

新建一个docker-compose.yml文件

文件内容

version: "3"

services:

  spark-iceberg:

    image: tabulario/spark-iceberg

    container_name: spark-iceberg

    build: spark/

    networks:

      iceberg_net:

    depends_on:

      - rest

      - minio

    volumes:

      - ./warehouse:/home/iceberg/warehouse

      - ./notebooks:/home/iceberg/notebooks/notebooks

    environment:

      - AWS_ACCESS_KEY_ID=admin

      - AWS_SECRET_ACCESS_KEY=password

      - AWS_REGION=us-east-1

    ports:

      - 8888:8888

      - 8080:8080

      - 10000:10000

      - 10001:10001

  rest:

    image: tabulario/iceberg-rest

    container_name: iceberg-rest

    networks:

      iceberg_net:

    ports:

      - 8181:8181

    environment:

      - AWS_ACCESS_KEY_ID=admin

      - AWS_SECRET_ACCESS_KEY=password

      - AWS_REGION=us-east-1

      - CATALOG_WAREHOUSE=s3://warehouse/

      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO

      - CATALOG_S3_ENDPOINT=http://minio:9000

  minio:

    image: minio/minio

    container_name: minio

    environment:

      - MINIO_ROOT_USER=admin

      - MINIO_ROOT_PASSWORD=password

      - MINIO_DOMAIN=minio

    networks:

      iceberg_net:

        aliases:

          - warehouse.minio

    ports:

      - 9001:9001

      - 9000:9000

    command: ["server", "/data", "--console-address", ":9001"]

  mc:

    depends_on:

      - minio

    image: minio/mc

    container_name: mc

    networks:

      iceberg_net:

    environment:

      - AWS_ACCESS_KEY_ID=admin

      - AWS_SECRET_ACCESS_KEY=password

      - AWS_REGION=us-east-1

    entrypoint: >

      /bin/sh -c "

      until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;

      /usr/bin/mc rm -r --force minio/warehouse;

      /usr/bin/mc mb minio/warehouse;

      /usr/bin/mc policy set public minio/warehouse;

      tail -f /dev/null

      "

networks:

  iceberg_net:

通过命令启动 docker 容器

docker-compose up

可以在docker中查到到具体容器

启动spark会话

docker exec -it spark-iceberg spark-sql

还可以通过命令来启动notebook 服务器。 Notebook 服务器将于 http://localhost:8888

docker exec -it spark-iceberg notebook

创建表

CREATE TABLE demo.nyc.taxis

(

  vendor_id bigint,

  trip_id bigint,

  trip_distance float,

  fare_amount double,

  store_and_fwd_flag string

)

PARTITIONED BY (vendor_id);

Iceberg 目录支持所有 SQL DDL 命令，包括：

CREATE TABLE ... PARTITIONED BY

CREATE TABLE ... AS SELECT

ALTER TABLE

DROP TABLE

向表中写入数据

INSERT INTO demo.nyc.taxis

VALUES (1, 1000371, 1.8, 15.32, 'N'), (2, 1000372, 2.5, 22.15, 'N'), (2, 1000373, 0.9, 9.01, 'N'), (1, 1000374, 8.4, 42.13, 'Y');

从表中读取数据

SELECT * FROM demo.nyc.taxis;

这就是iceberg-spark的基本入门说明了，后续会更新一些高级的操作，欢迎大家关注，评论沟通，大家一起学习交流~