● 随机写会导致磁头不停地换道，造成效率的极大降低；顺序写磁头几乎不用换道，或者换道的时间很短 ● 本文来讨论一下两者具体的差别以及相应的内核调用

组件

版本

OS Ubuntu 16.04.4 LTS fio 2.2.10

通过fio测试，能够反映在读写中的状态，我们需要重点关注fio的输出报告中的几个关键指标： slat ：是指从 I/O 提交到实际执行 I/O 的时长（Submission latency） clat ：是指从 I/O 提交到 I/O 完成的时长（Completion latency） lat ：指的是从 fio 创建 I/O 到 I/O 完成的总时长 bw ：吞吐量 iops ：每秒 I/O 的次数

（1）同步随机写

主要采用fio作为测试工具，为了能够看到系统调用，使用strace工具，命令看起来是这样：

先来测试一个随机写

strace-f-tt-o/tmp/randwrite.log-Dfio-name=randwrite-rw=randwrite\-direct=1-bs=4k-size=1G-numjobs=1-group_reporting-filename=/tmp/test.db

提取关键信息

root@wilson-ubuntu:~#strace-f-tt-o/tmp/randwrite.log-Dfio-name=randwrite-rw=randwrite\>-direct=1-bs=4k-size=1G-numjobs=1-group_reporting-filename=/tmp/test.dbrandwrite:(g=0):rw=randwrite,bs=4K-4K/4K-4K/4K-4K,ioengine=sync,iodepth=1fio-2.2.10Starting1process...randwrite:(groupid=0,jobs=1):err=0:pid=26882:WedAug1410:39:022019write:io=1024.0MB,bw=52526KB/s,iops=13131,runt=19963msecclat(usec):min=42,max=18620,avg=56.15,stdev=164.79lat(usec):min=42,max=18620,avg=56.39,stdev=164.79...bw(KB/s):min=50648,max=55208,per=99.96%,avg=52506.03,stdev=1055.83...Runstatusgroup0(alljobs):WRITE:io=1024.0MB,aggrb=52525KB/s,minb=52525KB/s,maxb=52525KB/s,mint=19963msec,maxt=19963msecDiskstats(read/write):...sda:ios=0/262177,merge=0/25,ticks=0/7500,in_queue=7476,util=36.05%

列出了我们需要重点关注的信息： （1）clat ，平均时长56ms左右 （2）lat ，平均时长56ms左右 （3）bw ，吞吐量，大概在52M左右

再来看内核调用信息：

root@wilson-ubuntu:~#more/tmp/randwrite.log...2688210:38:41.919904lseek(3,665198592,SEEK_SET)=6651985922688210:38:41.919920write(3,"\220\240@\6\371\341\277>\0\200\36\31\0\0\0\0\202\2\7\320\343\6H\26P\340\277\370\330\30e\30"...,4096)=40962688210:38:41.919969lseek(3,4313088,SEEK_SET)=43130882688210:38:41.919985write(3,"\220\240@\6\371\341\277>\0\200\36\31\0\0\0\0\202\2\7\320\343\6H\26P\340\277\370\330\30e\30"...,4096)=40962688210:38:41.920032lseek(3,455880704,SEEK_SET)=4558807042688210:38:41.920048write(3,"\220\240@\6\371\341\277>\0\200\36\31\0\0\0\0\202\2\7\320\343\6H\26P\340\277\370\330\30e\30"...,4096)=40962688210:38:41.920096lseek(3,338862080,SEEK_SET)=3388620802688210:38:41.920112write(3,"\220\240@\6\371\341\277>\0\2402\24\0\0\0\0\202\2\7\320\343\6H\26P\340\277\370\330\30e\30"...,4096)=40962688210:38:41.920161lseek(3,739086336,SEEK_SET)=7390863362688210:38:41.920177write(3,"\220\240@\6\371\341\277>\0\2402\24\0\0\0\0\202\2\7\320\343\6H\26P\340\277\370\330\30e\30"...,4096)=40962688210:38:41.920229lseek(3,848175104,SEEK_SET)=8481751042688210:38:41.920245write(3,"\220\240@\6\371\341\277>\0\2402\24\0\0\0\0\202\2\7\320\343\6H\26P\340\277\370\330\30e\30"...,4096)=40962688210:38:41.920296lseek(3,1060147200,SEEK_SET)=10601472002688210:38:41.920312write(3,"\220\240@\6\371\341\277>\0\2402\24\0\0\0\0\202\2\7\320\343\6H\26P\340\277\370\330\30e\30"...,4096)=40962688210:38:41.920362lseek(3,863690752,SEEK_SET)=8636907522688210:38:41.920377write(3,"\220\240@\6\371\341\277>\0\2402\24\0\0\0\0\202\2\7\320\343\6H\26P\340\277\370\330\30e\30"...,4096)=40962688210:38:41.920428lseek(3,279457792,SEEK_SET)=2794577922688210:38:41.920444write(3,"\220\240@\6\371\341\277>\0\2402\24\0\0\0\0\202\2\7\320\343\6H\26P\340\277\370\330\30e\30"...,4096)=40962688210:38:41.920492lseek(3,271794176,SEEK_SET)=2717941762688210:38:41.920508write(3,"\220\240@\6\371\341\277>\0\2402\24\0\0\0\0\202\2\7\320\343\6H\26P\340\277\370\330\30e\30"...,4096)=40962688210:38:41.920558lseek(3,1067864064,SEEK_SET)=10678640642688210:38:41.920573write(3,"\220\240@\6\371\341\277>\0\2402\24\0\0\0\0\202\2\7\320\343\6H\26P\340\277\370\330\30e\30"...,4096)=4096...

随机读每一次写入之前都要通过lseek去定位当前的文件偏移量

同步顺序写

用刚才的方法来测试顺序写

root@wilson-ubuntu:~#strace-f-tt-o/tmp/write.log-Dfio-name=write-rw=write\-direct=1-bs=4k-size=1G-numjobs=1-group_reporting-filename=/tmp/test.dbwrite:(g=0):rw=write,bs=4K-4K/4K-4K/4K-4K,ioengine=sync,iodepth=1fio-2.2.10Starting1processJobs:1(f=1):[W(1)][100.0%done][0KB/70432KB/0KB/s][0/17.7K/0iops][eta00m:00s]write:(groupid=0,jobs=1):err=0:pid=27005:WedAug1410:53:022019write:io=1024.0MB,bw=70238KB/s,iops=17559,runt=14929msecclat(usec):min=43,max=7464,avg=55.95,stdev=56.24lat(usec):min=43,max=7465,avg=56.15,stdev=56.25...bw(KB/s):min=67304,max=72008,per=99.98%,avg=70225.38,stdev=1266.88...Runstatusgroup0(alljobs):WRITE:io=1024.0MB,aggrb=70237KB/s,minb=70237KB/s,maxb=70237KB/s,mint=14929msec,maxt=14929msecDiskstats(read/write):...sda:ios=0/262162,merge=0/10,ticks=0/6948,in_queue=6932,util=46.49%

可以看到： 吞吐量提升至70M左右

再来看一下内核调用：

root@wilson-ubuntu:~#more/tmp/write.log...2704610:54:28.194508write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\360\t\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.194568write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.194627write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.194687write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.194747write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.194807write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.194868write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.194928write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.194988write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195049write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195110write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195197write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195262write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195330write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195426write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195497write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195567write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195637write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195704write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195757write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195807write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195859write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195910write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.195961write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.196012write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.196062write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\220\24\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.196112write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\26\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.196162write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\26\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.196213write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\26\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.196265write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\26\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.196314write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\26\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.196363write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\26\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.196414write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\26\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.196472write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\26\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.196524write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\26\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=40962704610:54:28.196573write(3,"\0\0\23\0\0\0\0\0\0\300\16\0\0\0\0\0\0\26\0\0\0\0\0\0\320\17\0\0\0\0\0"...,4096)=4096...

由于顺序读，不需要反复定位文件偏移量，所以能够专注于写操作

从上面的测试，我们在fio的测试报告中，并没有发现slat的身影，那是由于上述都是同步操作，对同步 I/O 来说，由于 I/O 提交和 I/O 完成是一个动作，所以 slat 实际上就是 I/O 完成的时间

异步顺序写，将同步顺序写的命令添加-ioengine=libaio：

root@wilson-ubuntu:~#fio-name=write-rw=write-ioengine=libaio-direct=1-bs=4k-size=1G-numjobs=1-group_reporting-filename=/tmp/test.dbwrite:(g=0):rw=write,bs=4K-4K/4K-4K/4K-4K,ioengine=libaio,iodepth=1fio-2.2.10Starting1processJobs:1(f=1):[W(1)][100.0%done][0KB/119.3MB/0KB/s][0/30.6K/0iops][eta00m:00s]write:(groupid=0,jobs=1):err=0:pid=27258:WedAug1411:14:362019write:io=1024.0MB,bw=120443KB/s,iops=30110,runt=8706msecslat(usec):min=3,max=70,avg=4.31,stdev=1.56clat(usec):min=0,max=8967,avg=28.13,stdev=55.68lat(usec):min=22,max=8976,avg=32.53,stdev=55.72...bw(KB/s):min=118480,max=122880,per=100.00%,avg=120467.29,stdev=1525.68...Runstatusgroup0(alljobs):WRITE:io=1024.0MB,aggrb=120442KB/s,minb=120442KB/s,maxb=120442KB/s,mint=8706msec,maxt=8706msecDiskstats(read/write):...sda:ios=0/262147,merge=0/1,ticks=0/6576,in_queue=6568,util=74.32%

可以看到，slat指标出现，lat 近似等于 slat + clat 之和（avg平均值）；并且换成异步io之后，吞吐量得到了极大的提升，120M左右