【Rust日报】2023-05-10 llm - 使用Rust在CPU上运行大模型

news2024/10/6 10:42:42

llm - 使用Rust在CPU上运行大模型

这是一套工具。目前支持这些模型：

GPT-2
GPT-J
LLaMA: LLaMA, Alpaca, Vicuna, Koala, GPT4All v1, GPT4-X, Wizard
GPT-NeoX: GPT-NeoX, StableLM, Dolly v2 (partial, not the same tensor names?)
BLOOM: BLOOMZ

https://github.com/rustformers/llm

从Rust 1.71开始，musl编译目标依赖的musl版本升级了。

Beginning with Rust 1.71 (slated for stable release on 2023-07-13),

从 musl 1.1.24 升级到了musl 1.2.3.

https://blog.rust-lang.org/2023/05/09/Updating-musl-targets.html

Salvo 0.41.0 发布，初步支持 openapi

use salvo::oapi::extract::*;
use salvo::oapi::swagger_ui::SwaggerUi;
use salvo::oapi::{ToSchema, Info, OpenApi};
use salvo::prelude::*;
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize, ToSchema, Debug)]
struct MyObject<T: ToSchema + std::fmt::Debug> {
    value: T,
}

#[endpoint]
async fn use_string(body: JsonBody<MyObject<String>>, res: &mut Response) {
    res.render(format!("{:?}", body))
}
#[endpoint]
async fn use_i32(body: JsonBody<MyObject<i32>>, res: &mut Response) {
    res.render(format!("{:?}", body))
}
#[endpoint]
async fn use_u64(body: JsonBody<MyObject<u64>>, res: &mut Response) {
    res.render(format!("{:?}", body))
}

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt().init();

    let router = Router::new()
        .push(Router::with_path("i32").post(use_i32))
        .push(Router::with_path("u64").post(use_u64))
        .push(Router::with_path("string").post(use_string));

    let doc = OpenApi::new(Info::new("test api", "0.0.1")).merge_router(&router);

    let router = router
        .push(doc.into_router("/api-doc/openapi.json"))
        .push(SwaggerUi::new("/api-doc/openapi.json").into_router("swagger-ui"));

    let acceptor = TcpListener::new("127.0.0.1:5800").bind().await;
    Server::new(acceptor).serve(router).await;
}

https://github.com/salvo-rs/salvo/blob/main/examples/oapi-generics/src/main.rs