当Hadoop集群搭建完成后，为了弄清楚HDFS的读写性能，我们可以对集群做一次基准测试。Hadoop安装包中自带了对HDFS进行基准测试的工具类。
一. 测试写入速度
-
查看命令帮助
[root@hadoop1 www]# yarn jar hadoop-3.2.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.4-tests.jar TestDFSIO 2024-12-28 03:59:58,695 INFO fs.TestDFSIO: TestDFSIO.1.8 Missing arguments. Usage: TestDFSIO [genericOptions] -read [-random | -backward | -skip [-skipSize Size]] | -write | -append | -truncate | -clean [-compression codecClassName] [-nrFiles N] [-size Size[B|KB|MB|GB|TB]] [-resFile resultFileName] [-bufferSize Bytes] [-storagePolicy storagePolicyName] [-erasureCodePolicy erasureCodePolicyName] 关键参数: -read: 读文件 -write: 写文件 -nrFiles: 文件数量 -size: 文件大小 -
提交测试写入速度的命令
# 写入10个文件,每个文件的大小为1G yarn jar hadoop-3.2.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.4-tests.jar TestDFSIO -write -nrFiles 10 -size 1GB -
运行结果
2024-12-28 04:18:57,771 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write 2024-12-28 04:18:57,771 INFO fs.TestDFSIO: Date & time: Sat Dec 28 04:18:57 UTC 2024 2024-12-28 04:18:57,771 INFO fs.TestDFSIO: Number of files: 10 2024-12-28 04:18:57,771 INFO fs.TestDFSIO: Total MBytes processed: 10240 2024-12-28 04:18:57,771 INFO fs.TestDFSIO: Throughput mb/sec: 22.33 2024-12-28 04:18:57,771 INFO fs.TestDFSIO: Average IO rate mb/sec: 22.73 2024-12-28 04:18:57,771 INFO fs.TestDFSIO: IO rate std deviation: 3.21 2024-12-28 04:18:57,772 INFO fs.TestDFSIO: Test exec time sec: 77.77 2024-12-28 04:18:57,772 INFO fs.TestDFSIO:
二. 测试读取速度
-
提交测试读取速度的命令
# 读取10个文件,每个文件的大小为1G yarn jar hadoop-3.2.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.4-tests.jar TestDFSIO -read -nrFiles 10 -size 1GB -
运行结果
2024-12-28 04:21:15,709 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read 2024-12-28 04:21:15,710 INFO fs.TestDFSIO: Date & time: Sat Dec 28 04:21:15 UTC 2024 2024-12-28 04:21:15,710 INFO fs.TestDFSIO: Number of files: 10 2024-12-28 04:21:15,710 INFO fs.TestDFSIO: Total MBytes processed: 10240 2024-12-28 04:21:15,710 INFO fs.TestDFSIO: Throughput mb/sec: 81.52 2024-12-28 04:21:15,710 INFO fs.TestDFSIO: Average IO rate mb/sec: 99.66 2024-12-28 04:21:15,710 INFO fs.TestDFSIO: IO rate std deviation: 47.22 2024-12-28 04:21:15,710 INFO fs.TestDFSIO: Test exec time sec: 43 2024-12-28 04:21:15,710 INFO fs.TestDFSIO: -
清理测试生成的文件
yarn jar hadoop-3.2.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.4-tests.jar TestDFSIO -clean
