Slow on heavy I/O operations.

[Eric Anderson]

On 03/22/07 08:20, Roman Gorohov. wrote:
> Hi, Oliver.
> Thanks for explanations.
> 
>> Roman Gorohov. <roma.a.g at gmail.com> wrote:
>  >> Hello list.
>  >> There is a server with FreeBSD FreeBSD 4.11-STABLE(can't upgrade fornow) on HP ProLiant DL140.
>  >> Disk system: <SEAGATE ST373207LC 0005> at scbus0 target 0 lun0(pass0,da0) on a ASC-29320A.
>  >> If I do:
>  >> [idle at hst ~]#dd if=/dev/zero of=bigfile bs=64k count=5000
>  >> 5000+0 records in
>  >> 5000+0 records out
>  >> 327680000 bytes transferred in 37.515699 secs (8734477 bytes/sec)
>  >> All is hung while dd working.
>  >> 
>  >> Its look like that:
>  >> [idle at hst ~]#systat -v
>  >> skip...
>  >> Disks   da0 pass0 pass1   md0         
>  >> KB/t  20.33  0.00  0.00  0.00        
>  >> tps     812     0     0     0        
>  >> MB/s  16.11  0.00  0.00  0.00                      
>  >> % busy  100     0     0     0                                                               
>  >> skip...
>  >> 
>  >> [idle at hst ~]#vmstat -w 5
>  >>  procs      memory      page                    disks     faults      cpu
>  >>  r b w     avm    fre  flt  re  pi  po  fr  sr da0 md0   in   sy  cs us sy id
>  >>  1 0 0 1996756  54208   42  23  25   7  39 336   0   0  531  638 404 21  5 74
>  >>  1 6 0 2011488  13272  571   0   0   1 2506   8 129   0 1356 2125 159  3  4 93
>  >>  4 27 0 2022852   4228  267   0   0   1 1657   4 272   0 1450  784  96  1  3 96
>  >>  1 10 0 2027492  40220  449   5   0  12 3257 9384 574   0 1773 4366 189  8  7 85
>  >>  1 10 0 2035064  25604  746   2   1   7 2965 4466 157   0 1380 4486 184  8  7 85
>  >>  0 11 0 2030860   4228  515   2   1   3 2098  31 214   0 1412 3077 149 16  5 79
>  >>  3 14 0 2042792  10004 1014   4   1   5 2787 4432 267   0 1440 1735 118  2  4 94
>  >>  2 16 0 2062664   4228  301   0   1   2 2843  27 564   0 1759 1250 107  5  4 91
>  >> 
>  >> [idle at hst ~]#iostat -w 5
>  >>       tty             da0            pass0            pass1             cpu
>  >>  tin tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
>  >>    0   25  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00  21  0  4  0 74
>  >>    1  109 62.56 242 14.81   0.00   0  0.00   0.00   0  0.00   7  0  9  0 83
>  >>    0  101 18.29 433  7.73   0.00   0  0.00   0.00   0  0.00   2  0  2  0 95
>  >>    0   76 26.45 281  7.25   0.00   0  0.00   0.00   0  0.00   3  0  4  0 93
>  >>    0   94 15.99 357  5.57   0.00   0  0.00   0.00   0  0.00   2  0  2  0 96
>  >>    0   15 16.04 667 10.45   0.00   0  0.00   0.00   0  0.00   0  0  3  1 96
>  >>    0   93 15.99 558  8.71   0.00   0  0.00   0.00   0  0.00   0  0  2  1 97
>  >>    0   80 16.16 252  3.98   0.00   0  0.00   0.00   0  0.00   0  0  1  0 98
>  >>    0  117 23.44 240  5.49   0.00   0  0.00   0.00   0  0.00   9  0  6  0 85
>  >> 
>  >> Why this happen?
> 
>> The numbers from your vmstat output indicate that your
>> machine is paging ("swapping"), i.e. it's running out
>> or RAM.
> 
> That seems like true, and I forget to mention that 2/3 of swap is
> located at physical disk(da0).
> [idle at hst ~]#swapinfo
> Device          1K-blocks     Used    Avail Capacity  Type
> /dev/da0s1b        524160   463300    60860    88%    Interleaved
> /dev/rvn0b        1048448   578724   469724    55%    Interleaved
> Total             1572608  1042024   530584    66%
> 
> But what I can't understand - why such non-critical(as it seems to me) disk activity,
> cause 100% busyness for disk? Its claim to be 40.000MB/s transfers, but
> sometimes its 100% busy when iostat show only 1 MB/s.


If the disks are truly busy, then the heads might be wiggling away all 
your 40MB/s due to seeks.  Run gstat and see what it shows also.

I don't think you can do any real tests while the machine is swapping, 
since that will most likely destroy any notion of performance for you.

The dd test you did above also isn't that accurate, because you were 
writing to a file system on the disk, and that can change your 
performance quite a bit depending on a lot of factors based on the file 
system.

Eric