测试用例
针对 RBD 设备的性能测试，可以设置具体的性能指标。通过这些指标可以综合衡量 Ceph 集群中 RBD 的极限性能；
rbd bench-write
# rbd bench-write rbd: bench-write is deprecated, use rbd bench --io-type write ... rbd: image name was not specified
rbd bench-write，该命令已废弃，建议使用 rbd bench 命令；
rbd bench
# rbd help bench
usage: rbd bench [--pool <pool>] [--namespace <namespace>] [--image <image>]
[--io-size <io-size>] [--io-threads <io-threads>]
[--io-total <io-total>] [--io-pattern <io-pattern>]
[--rw-mix-read <rw-mix-read>] --io-type <io-type>
<image-spec>
创建用于性能测试的磁盘：
rbd -p benchmark-rados-bench create benchmark-rbd-bench --size 20G rbd -p benchmark-rados-bench info --image benchmark-rbd-bench rbd -p benchmark-rados-bench map benchmark-rbd-bench rbd showmapped
Bandwidth
Write
rbd bench --pool benchmark-rados-bench --image benchmark-rbd-bench \
--io-size 4M --io-threads 32 --io-total 10G --io-pattern seq --io-type write
...
308 2496 6.16315 25 MiB/s
312 2528 6.18411 25 MiB/s
elapsed: 316 ops: 2560 ops/sec: 8.08599 bytes/sec: 32 MiB/s
...
313 2496 7.94257 32 MiB/s
316 2528 7.80007 31 MiB/s
elapsed: 330 ops: 2560 ops/sec: 7.73511 bytes/sec: 31 MiB/s
...
303 2496 7.81532 31 MiB/s
306 2528 7.87688 32 MiB/s
elapsed: 311 ops: 2560 ops/sec: 8.23063 bytes/sec: 33 MiB/s
IO Size = 1M
...
227 10144 36.8895 37 MiB/s
228 10208 36.7443 37 MiB/s
elapsed: 229 ops: 10240 ops/sec: 44.5707 bytes/sec: 45 MiB/s
... IO Size = 10M
267 960 3.61849 36 MiB/s
281 992 3.26707 33 MiB/s
elapsed: 290 ops: 1024 ops/sec: 3.52986 bytes/sec: 35 MiB/s
Read
rbd bench --pool benchmark-rados-bench --image benchmark-rbd-bench \
--io-size 128K --io-threads 32 --io-total 10G --io-pattern seq --io-type read
... IO Size = 128K
30 76992 2446.68 306 MiB/s
31 79616 2478.71 310 MiB/s
elapsed: 31 ops: 81920 ops/sec: 2566.98 bytes/sec: 321 MiB/s
... IO Size = 1M
28 9760 336.831 337 MiB/s
29 10144 346.698 347 MiB/s
elapsed: 29 ops: 10240 ops/sec: 348.382 bytes/sec: 348 MiB/s
... IO Size = 2M
27 4768 161.938 324 MiB/s
28 4960 170.207 340 MiB/s
elapsed: 28 ops: 5120 ops/sec: 176.814 bytes/sec: 354 MiB/s
... IO Size = 4M
27 2400 82.4062 330 MiB/s
28 2496 87.4289 350 MiB/s
elapsed: 29 ops: 2560 ops/sec: 87.7285 bytes/sec: 351 MiB/s
IOPS
Write
# rbd bench --pool benchmark-rados-bench --image benchmark-rbd-bench \
--io-size 4K --io-threads 32 --io-total 10M --io-pattern rand --io-type write
... IO Size = 4K
55 2432 11.0212 44 KiB/s
59 2464 10.5439 42 KiB/s
60 2528 12.8048 51 KiB/s
...
// Write IOPS 表现非常差，可能存在错误，需要额外留意；
... IO Size = 4K
elapsed: 2916 ops: 2560000 ops/sec: 877.707 bytes/sec: 3.4 MiB/s
elapsed: 296 ops: 256000 ops/sec: 863.636 bytes/sec: 3.4 MiB/s
elapsed: 1486 ops: 1280000 ops/sec: 860.928 bytes/sec: 3.4 MiB/s
Read
# rbd bench --pool benchmark-rados-bench --image benchmark-rbd-bench \
--io-size 4K --io-threads 32 --io-total 10G --io-pattern rand --io-type read
... IO Size = 4K
elapsed: 153 ops: 2621440 ops/sec: 17040 bytes/sec: 67 MiB/s
elapsed: 154 ops: 2621440 ops/sec: 16949.6 bytes/sec: 66 MiB/s
elapsed: 156 ops: 2621440 ops/sec: 16734.4 bytes/sec: 65 MiB/s
elapsed: 164 ops: 2621440 ops/sec: 15952.5 bytes/sec: 62 MiB/s
fio
// –ioengine=rbd –clientname=admin –pool=benchmark-rados-bench –rbdname=benchmark-rbd-bench
在实际场景中，我们将 rbd 被映射为块设备，然后再使用。所以，在性能测试时，我们也将针对映射后的块设备进行测试。而 fio –ioengine=rbd 则是使用 librbd 库，将跳过内核模块，所以我们未采用 fio –ioengine=rbd 进行测试；
Bandwidth
Write:
fio --name=write_bandwidth_test \ --filename=/dev/rbd0 --filesize=16Gi --offset_increment=1Gi \ --time_based --ramp_time=2s --runtime=1m \ --ioengine=libaio --direct=1 --verify=0 --randrepeat=0 \ --bs=1M --iodepth=64 --rw=write --numjobs=16 --group_reporting Run status group 0 (all jobs): WRITE: bw=80.2MiB/s (84.1MB/s), 80.2MiB/s-80.2MiB/s (84.1MB/s-84.1MB/s), io=6694MiB (7019MB), run=83482-83482msec Run status group 0 (all jobs): WRITE: bw=66.2MiB/s (69.4MB/s), 66.2MiB/s-66.2MiB/s (69.4MB/s-69.4MB/s), io=6735MiB (7062MB), run=101696-101696msec Run status group 0 (all jobs): WRITE: bw=39.5MiB/s (41.4MB/s), 39.5MiB/s-39.5MiB/s (41.4MB/s-41.4MB/s), io=5581MiB (5852MB), run=141467-141467msec Run status group 0 (all jobs): WRITE: bw=33.5MiB/s (35.2MB/s), 33.5MiB/s-33.5MiB/s (35.2MB/s-35.2MB/s), io=4637MiB (4862MB), run=138249-138249msec Run status group 0 (all jobs): WRITE: bw=24.8MiB/s (25.0MB/s), 24.8MiB/s-24.8MiB/s (25.0MB/s-25.0MB/s), io=5896MiB (6182MB), run=238118-238118msec Run status group 0 (all jobs): WRITE: bw=65.7MiB/s (68.9MB/s), 65.7MiB/s-65.7MiB/s (68.9MB/s-68.9MB/s), io=6310MiB (6617MB), run=96017-96017msec Run status group 0 (all jobs): WRITE: bw=94.6MiB/s (99.2MB/s), 94.6MiB/s-94.6MiB/s (99.2MB/s-99.2MB/s), io=6484MiB (6799MB), run=68556-68556msec Run status group 0 (all jobs): WRITE: bw=85.0MiB/s (90.2MB/s), 85.0MiB/s-85.0MiB/s (90.2MB/s-90.2MB/s), io=6372MiB (6682MB), run=74095-74095msec
Read:
fio --name=read_bandwidth_test \ --filename=/dev/rbd0 --filesize=16Gi --offset_increment=1Gi \ --time_based --ramp_time=2s --runtime=1m \ --ioengine=libaio --direct=1 --verify=0 --randrepeat=0 \ --bs=1M --iodepth=64 --rw=read --numjobs=16 --group_reporting Run status group 0 (all jobs): READ: bw=358MiB/s (375MB/s), 358MiB/s-358MiB/s (375MB/s-375MB/s), io=22.5GiB (24.2GB), run=64448-64448msec Run status group 0 (all jobs): READ: bw=358MiB/s (375MB/s), 358MiB/s-358MiB/s (375MB/s-375MB/s), io=22.5GiB (24.2GB), run=64531-64531msec Run status group 0 (all jobs): READ: bw=357MiB/s (374MB/s), 357MiB/s-357MiB/s (374MB/s-374MB/s), io=22.4GiB (24.1GB), run=64437-64437msec
4MB Bandwidth:
IOPS
Rnd. W 4KB IOPS:
fio --name=write_iops_test \ --filename=/dev/rbd0 --filesize=16Gi \ --time_based --ramp_time=2s --runtime=1m \ --ioengine=libaio --direct=1 --verify=0 --randrepeat=0 \ --bs=4K --iodepth=256 --rw=randwrite write: IOPS=424, BW=1713KiB/s (1755kB/s)(101MiB/60484msec); 0 zone resets write: IOPS=398, BW=1612KiB/s (1651kB/s)(95.5MiB/60683msec); 0 zone resets write: IOPS=411, BW=1662KiB/s (1702kB/s)(98.4MiB/60638msec); 0 zone resets write: IOPS=414, BW=1677KiB/s (1717kB/s)(98.8MiB/60331msec); 0 zone resets write: IOPS=410, BW=1659KiB/s (1699kB/s)(97.9MiB/60431msec); 0 zone resets write: IOPS=399, BW=1616KiB/s (1655kB/s)(95.1MiB/60230msec); 0 zone resets
Rnd. R 4KB IOPS:
fio --name=read_iops_test \ --filename=/dev/rbd0 --filesize=16Gi \ --time_based --ramp_time=2s --runtime=1m \ --ioengine=libaio --direct=1 --verify=0 --randrepeat=0 \ --bs=4K --iodepth=256 --rw=randread read: IOPS=14.4k, BW=56.4MiB/s (59.1MB/s)(3385MiB/60010msec) read: IOPS=14.6k, BW=56.0MiB/s (59.8MB/s)(3420MiB/60014msec) read: IOPS=14.5k, BW=56.5MiB/s (59.2MB/s)(3391MiB/60010msec) read: IOPS=14.8k, BW=57.9MiB/s (60.8MB/s)(3477MiB/60010msec) read: IOPS=14.5k, BW=56.6MiB/s (59.3MB/s)(3394MiB/60012msec)
Seq. W 4KB IOPS:
Seq. R 4KB IOPS:
Latency
Rnd. W 4KB Latency:
fio --name=write_latency_test \ --filename=/dev/rbd0 --filesize=16Gi \ --time_based --ramp_time=2s --runtime=1m \ --ioengine=libaio --direct=1 --verify=0 --randrepeat=0 \ --bs=4K --iodepth=4 --rw=randwrite lat (msec) : 10=13.72%, 20=2.84%, 50=27.53%, 100=0.38%, 250=1.42% lat (msec) : 500=53.83%, 750=0.57% lat (msec) : 10=16.17%, 20=0.56%, 50=27.60%, 100=0.65%, 250=1.12% lat (msec) : 500=53.81%, 750=0.37% lat (msec) : 10=15.85%, 20=1.50%, 50=26.83%, 100=0.66%, 250=0.84% lat (msec) : 500=54.22%, 750=0.38% lat (msec) : 10=14.70%, 20=1.23%, 50=28.18%, 100=0.47%, 250=1.23% lat (msec) : 500=54.29%, 750=0.19% lat (msec) : 10=16.65%, 20=1.20%, 50=27.51%, 100=0.09%, 250=1.10% lat (msec) : 500=53.54%, 750=0.18% lat (msec) : 10=14.12%, 20=2.09%, 50=27.39%, 100=0.28%, 250=1.61% lat (msec) : 500=54.31%, 750=0.47%
Rnd. R 4KB Latency:
fio --name=read_latency_test \ --filename=/dev/rbd0 --filesize=16Gi \ --time_based --ramp_time=2s --runtime=1m \ --ioengine=libaio --direct=1 --verify=0 --randrepeat=0 \ --bs=4K --iodepth=4 --rw=randread lat (usec) : 250=3.68%, 500=19.14%, 750=46.75%, 1000=18.72% lat (msec) : 2=10.79%, 4=0.60%, 10=0.31%, 20=0.01% lat (usec) : 250=3.60%, 500=18.92%, 750=47.20%, 1000=18.67% lat (msec) : 2=10.68%, 4=0.62%, 10=0.30%, 20=0.01%, 50=0.01% lat (usec) : 250=3.75%, 500=19.30%, 750=46.83%, 1000=18.65% lat (msec) : 2=10.67%, 4=0.55%, 10=0.24%, 20=0.01% lat (usec) : 250=3.76%, 500=19.22%, 750=46.88%, 1000=18.52% lat (msec) : 2=10.76%, 4=0.60%, 10=0.27%, 20=0.01% lat (usec) : 250=3.54%, 500=18.68%, 750=47.05%, 1000=18.92% lat (msec) : 2=10.93%, 4=0.62%, 10=0.26%, 20=0.01%, 50=0.01%
Seq. W 4KB Latency:
Seq. R 4KB Latency: