byzer plugin install log

news2024/11/14 6:45:11

离线插件参考地址：

Byzer Documentation

离线安装方式（错误过程记录）：

参考文档：https://docs.byzer.org/#/byzer-lang/zh-cn/extension/README

Byzer-lang 支持插件安装，删除，获取列表等。安装和删除插件的语法如下

!plugin <pluginType> <operation> [-] "<pluginName>";

pluginType 表示插件类型，目前有 app ds et 三种。
operation 表示操作类型，add remove。
pluginName 表示插件名称。
安装插件时，请在插件名称前加横杠，并以空格分割。删除时，无需空格。

安装插件前，请配置 Byzer-lang 数据湖目录 streaming.datalake.path，支持本地文件系统，HDFS，对象存储。 Byzer-lang 下载插件至数据湖目录，并热加载之。获取列表语法如下：

!plugin list [pluginType];

若不指定插件类型，Byzer-lang 返回所有插件。

!plugin list 会报错，提示目录不存在。

具体解决方式

通过手工方式，将一台已经安装好openmldb插件的目录，copy到新服务器相应的目录：

插件目录是engine根下面的data 和store，这两个目录配置为delta数据库路径。

重启./bin/bizer.sh restart

在note上执行plugin list，不报错;

执行创建openml的数据表代码，报错如下：

查看log文件，错误信息如下：从日志看，是以为delta没有元数据。估计这样直接复制文件方式不行。

24/05/20 04:20:34 INFO DeltaLog: Returning initial snapshot Snapshot(path=file:/home/byzerllm/softwares/byzer-lang-all-in-one-linux-amd64-3.3.0-2.3.9/data/__instances__/Byzer-lang-engine/__mlsql__/plugins/_delta_log, version=0, metadata=Metadata(03706428-a82a-4dfe-9793-d08aa2539134,null,null,Format(parquet,Map()),{"type":"struct","fields":[{"name":"pluginName","type":"string","nullable":true,"metadata":{}},{"name":"path","type":"string","nullable":true,"metadata":{}},{"name":"pluginType","type":"string","nullable":true,"metadata":{}},{"name":"version","type":"string","nullable":true,"metadata":{}}]},List(),Map(path -> ./data/__instances__/Byzer-lang-desktop/__mlsql__/plugins),Some(1715907739552)), logSegment=LogSegment(file:/home/byzerllm/softwares/byzer-lang-all-in-one-linux-amd64-3.3.0-2.3.9/data/__instances__/Byzer-lang-engine/__mlsql__/plugins/_delta_log,0,ArrayBuffer(SerializableFileStatus(file:/home/byzerllm/softwares/byzer-lang-all-in-one-linux-amd64-3.3.0-2.3.9/data/__instances__/Byzer-lang-engine/__mlsql__/plugins/_delta_log/00000000000000000000.json,1598,false,1715907739000)),List(),None,1715907739000), checksumOpt=None)
24/05/20 04:20:34 INFO DelegatingLogStore: LogStore `LogStoreAdapter(io.delta.storage.HDFSLogStore)` is used for scheme `file`
24/05/20 04:20:34 INFO DeltaLog: Creating initial snapshot without metadata, because the directory is empty

----

----

----

24/05/20 04:20:36 INFO SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1513
24/05/20 04:20:36 INFO FileScanRDD: Reading File path: file:/home/byzerllm/softwares/byzer-lang-all-in-one-linux-amd64-3.3.0-2.3.9/data/__instances__/Byzer-lang-engine/__mlsql__/plugins/part-00000-7e61365c-1417-464c-8d36-23e8fafcb529-c000.snappy.parquet, range: 0-2041, partition values: [empty row]
24/05/20 04:20:36 INFO CodecPool: Got brand-new decompressor [.snappy]
24/05/20 04:20:36 INFO CodeGenerator: Code generated in 10.923573 ms
24/05/20 04:20:36 INFO PluginHook: Plugin byzer-openmldb-3.0 in ./data/__instances__/Byzer-lang-desktop/__mlsql__/files/store/plugins/byzer-openmldb-3.0_2.12-0.1.0-SNAPSHOT.jar
24/05/20 04:20:36 INFO PluginUtils: [owner] [admin] [groupId] [0] __MMMMMM__ Download [local]/home/byzerllm/softwares/byzer-lang-all-in-one-linux-amd64-3.3.0-2.3.9/store/plugins/byzer-openmldb-3.0_2.12-0.1.0-SNAPSHOT.jar from [remote]./data/__instances__/Byzer-lang-desktop/__mlsql__/files/store/plugins/byzer-openmldb-3.0_2.12-0.1.0-SNAPSHOT.jar
24/05/20 04:20:36 ERROR TryTool: Uncaught exception in thread main
java.lang.NullPointerException: null
at java.util.Objects.requireNonNull(Objects.java:203) ~[?:1.8.0-332]
at java.nio.file.Files.copy(Files.java:2984) ~[?:1.8.0-332]
at tech.mlsql.runtime.PluginUtils$.downloadFromHDFSToLocal(PluginUtils.scala:215) ~[byzer-lang-3.3.0-2.12-2.3.9.jar:?]
at tech.mlsql.runtime.PluginHook.$anonfun$beforeDispatcher$14(PluginHook.scala:55) ~[byzer-lang-3.3.0-2.12-2.3.9.jar:?]
at tech.mlsql.runtime.PluginHook.$anonfun$beforeDispatcher$14$adapted(PluginHook.scala:53) ~[byzer-lang-3.3.0-2.12-2.3.9.jar:?]
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) ~[scala-library-2.12.15.jar:?]
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) ~[scala-library-2.12.15.jar:?]
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) ~[scala-library-2.12.15.jar:?]
at tech.mlsql.runtime.PluginHook.beforeDispatcher(PluginHook.scala:53) ~[byzer-lang-3.3.0-2.12-2.3.9.jar:?]
at streaming.core.strategy.platform.PlatformManager.$anonfun$run$9(PlatformManager.scala:161) ~[byzer-lang-3.3.0-2.12-2.3.9.jar:?]
at streaming.core.strategy.platform.PlatformManager.$anonfun$run$9$adapted(PlatformManager.scala:161) ~[byzer-lang-3.3.0-2.12-2.3.9.jar:?]
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) ~[scala-library-2.12.15.jar:?]
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) ~[scala-library-2.12.15.jar:?]
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?]
at streaming.core.strategy.platform.PlatformManager.$anonfun$run$8(PlatformManager.scala:161) ~[byzer-lang-3.3.0-2.12-2.3.9.jar:?]
at tech.mlsql.common.utils.base.TryTool$.tryLogNonFatalError(TryTool.scala:94) ~[byzer-lang-3.3.0-2.12-2.3.9.jar:?]
at streaming.core.strategy.platform.PlatformManager.run(PlatformManager.scala:161) ~[byzer-lang-3.3.0-2.12-2.3.9.jar:?]
at streaming.core.StreamingApp$.main(StreamingApp.scala:45) ~[byzer-lang-3.3.0-2.12-2.3.9.jar:?]
at tech.mlsql.example.app.LocalSparkServiceApp$.main(LoalSparkServiceApp.scala:21) ~[byzer-lang-3.3.0-2.12-2.3.9.jar:?]
at tech.mlsql.example.app.LocalSparkServiceApp.main(LoalSparkServiceApp.scala) ~[byzer-lang-3.3.0-2.12-2.3.9.jar:?]
24/05/20 04:20:44 INFO impl: Completed 200 in 1ms GET /health/liveness

离线安装方式2（成功过程记录）：

1.下载地址：

http://store.mlsql.tech/run?action=downloadPlugin&pluginType=MLSQL_PLUGIN&pluginName=byzer-openmldb-3.0&version=0.1.0-SNAPSHOT

2.将jar包放到byzerlang下的plugin目录

3.修改byzer-lang下面conf目录的byzer.properties.override的streaming.plugin.clzznames

增加入口：tech.mlsql.plugins.openmldb.ByzerApp

byzer.server.mode=all-in-one
byzer.server.dryrun=false

byzer.server.runtime.driver-memory=24g

streaming.name=Byzer-lang-engine
streaming.rest=true
streaming.thrift=false
streaming.platform=spark
streaming.spark.service=true
streaming.job.cancel=true
streaming.datalake.path=./data/
streaming.driver.port=9003
streaming.enableHiveSupport=false
streaming.plugin.clzznames=tech.mlsql.plugins.ds.MLSQLExcelApp,tech.mlsql.plugins.assert.app.MLSQLAssert,tech.mlsql.plugins.shell.app.MLSQLShell,tech.mlsql.plugins.mllib.app.MLSQLMllib,tech.mlsql.plugins.llm.LLMApp,tech.mlsql.plugins.execsql.ExecSQLApp,tech.mlsql.plugins.openmldb.ByzerApp

spark.mlsql.log.driver.enablePrint=true
spark.mlsql.path.schemas=oss,s3a,s3,abfs,file
spark.mlsql.session.expireTime=10d
spark.local.dir=/home/byzerllm/byzerllm_stroage