使用Hey对vllm进行模型并发压测
docker run --rm --network=knowledge_network \
registry.cn-shanghai.aliyuncs.com/zhph-server/hey:latest \
-n 200 -c 200 -m POST -H "Content-Type: application/json" \
-H "Authorization: xxx" \
-d '{
"model": "codechat",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello!"}
],
"stream": false,
"max_tokens": 100,
"temperature": 0.0
}' http://vllm-openai:80/v1/chat/completions
docker run --rm --network=knowledge_network \
registry.cn-shanghai.aliyuncs.com/zhph-server/hey:latest \
-n 200 -c 200 -m POST -H "Content-Type: application/json" \
-H "Authorization: xxx" \
-d '{
"model": "codebase",
"prompt": "# write a python code to print hello world",
"stream": false,
"max_tokens": 100,
"temperature": 0.5
}' http://vllm-openai:80/v1/completions
结果
Summary:
Total: 2.2220 secs
Slowest: 1.3603 secs
Fastest: 0.7641 secs
Average: 1.0815 secs
Requests/sec: 43.2034
Total data: 28992 bytes
Size/request: 302 bytes
Response time histogram:
0.764 [1] |■
0.824 [5] |■■■■■■■
0.883 [4] |■■■■■■
0.943 [7] |■■■■■■■■■■
1.003 [11] |■■■■■■■■■■■■■■■■
1.062 [7] |■■■■■■■■■■
1.122 [28] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
1.181 [7] |■■■■■■■■■■
1.241 [9] |■■■■■■■■■■■■■
1.301 [9] |■■■■■■■■■■■■■
1.360 [8] |■■■■■■■■■■■
Latency distribution:
10% in 0.9175 secs
25% in 0.9570 secs
50% in 1.0721 secs
75% in 1.2131 secs
90% in 1.2790 secs
95% in 1.3599 secs
0% in 0.0000 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0036 secs, 0.7641 secs, 1.3603 secs
DNS-lookup: 0.0013 secs, 0.0000 secs, 0.0075 secs
req write: 0.0003 secs, 0.0000 secs, 0.0051 secs
resp wait: 1.0774 secs, 0.7640 secs, 1.3533 secs
resp read: 0.0001 secs, 0.0000 secs, 0.0002 secs
Status code distribution:
[200] 96 responses
