Tagged queueing = lower write throughput?

[P.C. Uiterlinden]

While playing with RAID0, I found out that enabling tagged queueing
deteriorates write throughput considerably. The MD driver seems not
the one to blame: with a single disk it also happens. I did not
expect this behavior nor can I explain it. What I would expect was
a slight improvement (especially with multiple processes), but
certainly not a rather big deterioration! I hope someone can explain
this. Any help would be highly appreciated.


Below, I have included three reports. The first one is from bonnie,
on a single disk. The second one is generated with bonnie on a MD
device (two disks). The third one is on one disk, generated with
dd.

My systems looks like:

CPU  : Intel Pentium II, 350 MHz, 512 kB cache
RAM  : 128 MB
HD   : 2* SEAGATE ST34520W
SCSI : Adaptec AIC-7880 Ultra Wide (on mobo)


===============================================================================
Bonnie output of /dev/sdb11 for different tagged queue depths.
Note the drop in write speed (both per char and per block) when
queue depth >= 3. Read speed is unaffected.

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
qdepth_1 1500 11009 74.8 11501 16.2  4167 12.0 11293 56.4 11671 10.8  50.1  0.8
qdepth_2 1500 10749 73.0 11257 16.1  3445  9.9 11139 56.7 11336 10.8  52.8  1.0
qdepth_3 1500  6596 44.3  7532 10.8  3252  9.3 10861 54.1 11502 11.4  56.0  1.1
qdepth_4 1500  6745 45.2  7380 10.6  3300  9.6 10899 54.5 11434 10.8  56.9  1.1
qdepth_8 1500  6909 46.4  7426 10.5  3237  9.5 10888 54.0 11368 11.4  57.2  1.2

===============================================================================
Bonnie output of /dev/md4 (/dev/sda11 + /dev/sdb11, chunk size 32k)
for different tagged queue depths. The per block read and write speed
has doubled compared to the single disk situation. However, again when
queue depth >= 3, things get worse: the per block write speed drops 40%.
Strange: this is the only thing that seems to be affected.

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
qdepth_1 1500 13305 90.9 22619 34.3  7107 21.7 12821 65.0 23834 26.4  83.9  1.3
qdepth_2 1500 13232 89.8 21828 33.8  5599 16.9 13003 66.1 22517 22.1  86.8  1.5
qdepth_3 1500 12491 84.8 14262 19.8  5541 16.8 12625 64.1 22963 23.8  87.4  1.9
qdepth_4 1500 12381 83.8 14271 19.8  5529 16.9 12542 64.2 22134 23.2  89.3  2.0
qdepth_8 1500 10460 70.8 13573 19.7  5249 15.8 12877 65.6 22440 23.7  85.6  1.7
qdepth16 1500 12097 81.7 14025 20.1  5580 16.6 12822 65.8 22308 24.4  90.6  2.0
qdepth32 1500 11880 79.6 12818 18.3  5515 16.3 12642 64.6 22137 22.9  88.9  1.9

===============================================================================
Transfer rate numbers obtained with dd (on /dev/sdb11) do confirm
the numbers obtained with bonnie: only write throughput is affected
if the queue depth is larger than 2. Note the constantness of the
user+sys time, while the real time makes a jump at queue depth = 3.

Write  : time dd if=/dev/zero of=/mnt/t bs=1024k count=1500
Read   : time dd if=/mnt/t of=/dev/null bs=1024k
Remove : time rm /mnt/t


Queue depth    1         2         3         4         8        16

Write       133.808   137.218   218.119   217.751   207.189   216.199   real time (sec)
             23.190    23.180    23.590    23.550    23.520    23.370   user+sys time (sec)
             11.21     10.93      6.87      6.88      7.23      6.93    transfer rate (Mbyte/sec)

Read        135.385   137.304   139.691   140.149   132.901   140.170   real time (sec)
             22.360    23.120    22.310    22.430    23.090    23.240   user+sys time (sec)
             11.07     10.92     10.73     10.70     11.28     10.70    transfer rate (Mbyte/sec)

Remove       29.396    29.892    29.527    29.564    29.273    29.526   real time (sec)
              0.530     0.590     0.530     0.670     0.510     0.590   user+sys time (sec)

===============================================================================


To Unsubscribe: send mail to majordomo at FreeBSD.org
with "unsubscribe aic7xxx" in the body of the message