试运行llama-7B、vicuna-7b-delta-v1.1和vicuna-7b-v1.3

Chatgpt的出现给NLP领域带来了让人振奋的消息，可以很逼真的模拟人的对话，回答人们提出的问题，不过Chatgpt参数量，规模，训练代价都很昂贵。

幸运的是，出现了开源的一些相对小的模型，可以在本地或者云端部署体验，

动手体验了下Vicuna-7b，翻译过来是小羊驼」（骆马），拥有70亿参数，据作者实验能达到GPT-4的90%性能。在作者官网发布了三个版本，其中3个月前发布了v1.1，17天前发布了v1.3。

官网：lmsys (Large Model Systems Organization)

环境：

ubuntu18.04

九天毕昇8核32G内存，仅有cpu

时间：2023年7月5号

llama-7B

python3 -m cli --model-path /root/transformer_model_7b  --device cpu

文件：cli是fastchat/serve目录下的cli.py文件

cli.py内容

"""
Chat with a model with command line interface.

Usage:
python3 -m fastchat.serve.cli --model lmsys/vicuna-7b-v1.3
python3 -m fastchat.serve.cli --model lmsys/fastchat-t5-3b-v1.0

Other commands:
- Type "!!exit" or an empty line to exit.
- Type "!!reset" to start a new conversation.
"""
import argparse
import os
import re
import sys

from prompt_toolkit import PromptSession
from prompt_toolkit.auto_suggest import AutoSuggestFromHistory
from prompt_toolkit.completion import WordCompleter
from prompt_toolkit.history import InMemoryHistory
from prompt_toolkit.key_binding import KeyBindings
from rich.console import Console
from rich.live import Live
from rich.markdown import Markdown

from fastchat.model.model_adapter import add_model_args
from fastchat.modules.gptq import GptqConfig
from fastchat.serve.inference import ChatIO, chat_loop


class SimpleChatIO(ChatIO):
    def prompt_for_input(self, role) -> str:
        return input(f"{role}: ")

参数： --model-path 是指模型权重存放路径，这个目录下包含tokenizer.model文件，请仔细核对

参数： --device cpu 指定在cpu上运行

开始推理：输入 who is Musk? what is baidu? china

要等待几分钟，看到几个词几个词的蹦出来

看回答，字数还不少，内容只能说是相关

看下cpu使用率

top

可以看到python3进程，推理时cpu使用率在81.1%，一直在波动。

换用v100有时候会到98%，甚至更高，速度会快很多，一次输出一小段文字

vicuna-7b-delta-v1.1

python3 -m cli --model-path /root/vicuna-7b/vicuna-7b-delta-v11 --device cpu

加载模型，中间过程要几分钟

python3 -m cli --model-path /root/vicuna-7b/vicuna-7b-delta-v11 --device cpu
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████| 2/2 [02:58<00:00, 89.16s/it]

开始推理：实现一个python斐波那契函数，可以看到能够写出函数，还简单介绍了怎么用

USER: hello
ASSISTANT: Hello! How can I help you today? Is there something you would like to know or talk about? I'm here to help answer any questions you may have.
USER: implement a python function to compute fibonacci numbers
ASSISTANT: Sure! Here's an implementation of a Python function that computes the Fibonacci sequence:
```python
def fibonacci(n):
if n <= 1:
return n
else:
return fibonacci(n-1) + fibonacci(n-2)
```
This function takes an integer `n` as input and returns the `n`-th Fibonacci number. The first two numbers in the sequence are 0 and 1, so the function handles these cases separately: if `n` is less than or equal to 1, it returns `n`, and otherwise it computes the sum of the previous two Fibonacci numbers and returns that.

To use the function, you can call it with an integer `n` and it will return the `n`-th Fibonacci number. For example:
```python
>>> fibonacci(5)
13
```
This would compute the 5-th Fibonacci number, which is 13.

USER: china
ASSISTANT: China is a country located in East Asia, bordered by several other countries, including Russia, Mongolia, North Korea, Vietnam, Laos, Myanmar (Burma), Bhutan, India, Nepal, and Pakistan. It is the world's most populous country, with over 1.4 billion people.

China has a long and complex history, dating back over 5,000 years. It is known for its rich cultural heritage, including its cuisine, literature, and arts. The country has a diverse range of landscapes, from the high plateaus in the west to the fertile river valleys in the east, and from the tropical beaches in the south to the frigid deserts in the north.

vicuna-7b-v1.3

 python3 -m cli --model-path /root//vicuna-7b-v13 --device cpu

参数：-m 后面指定cli.py文件，注意【.py】省略。python3 -m cli是python3执行cli.py文件，具体内容，要看cli.py写了什么

加载模型，等待几分钟

python3 -m cli --model-path /root//vicuna-7b-v13 --device cpu
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████| 2/2 [03:17<00:00, 98.73s/it]

推理：

USER: Introduce China

ASSISTANT: China, officially known as the People's Republic of China (PRC), is a sovereign state located in East Asia. It is the world's most populous country, with a population of over 1.4 billion people, and the third-largest country by land area, covering approximately 9.6 million square kilometers.

China has a long and complex history, dating back more than 5,000 years. It has been ruled by various dynasties, including the Zhou, Qin, Han, Tang, Song, Yuan, Ming, and Qing dynasties, among others. In 1949, the Communist Party of China (CPC) established the PRC on the Chinese mainland, while the Republic of China (ROC) retained control of Taiwan and other offshore islands.

The People's Republic of China is a single-party socialist state with a planned economy, guided by the CPC. The country is governed according to the Constitution of the People's Republic of China, which was adopted in 1982. The government is divided into various branches, including the National People's Congress (NPC), the State Council, the Supreme People's Court, and the Central Military Commission.

China has a diverse landscape and culture, encompassing various ethnic groups, including the Han, Tibetans, Uighurs, Mongols, and Manchus, among others. The official language is Standard Chinese (Mandarin), although there are numerous regional languages and dialects spoken throughout the country.

The economy of China has experienced rapid growth in recent decades, making it the world's second-largest economy by nominal GDP and the largest by purchasing power parity (PPP). The country is a leading global exporter of goods and services, with industries such as electronics, textiles, automobiles, and telecommunications playing a significant role in its economic development.

China has been actively involved in international affairs, participating as a permanent member of the United Nations Security Council and a member of various multilateral organizations, including the World Trade Organization (WTO), the Asia-Pacific Economic Cooperation (APEC), and the Group of 20 (G20) major economies.

USER: china
ASSISTANT: China is a country located in East Asia, covering approximately 9.6 million square kilometers. With a population of over 1.4 billion people, it is the most populous country in the world. The official language is Mandarin Chinese, and the capital city is Beijing. China has a rich history and a diverse cultural heritage, with many different ethnic groups and over 56 recognized ethnic minorities.