sa and mpt in FreeBSD 11.0-RELEASE-p2

Tue Feb 21 17:15:56 UTC 2017

> On 20 Feb 2017, at 21:56, Kenneth D. Merry <ken at FreeBSD.ORG> wrote:
> 
> On Mon, Feb 20, 2017 at 21:04:11 +0100, Dirk-Willem van Gulik wrote:
>> A machine, recently upgraded from 8.4 to FreeBSD 11.0-RELEASE-p2 seems to have lost pretty much any and all performance on mpt(4) with its attached tape drives
>> 
>> Read performance is around 50Mbyte/second - and write a paltry 100-200kbyte/second (and occasionally hitting 800kbyte/second) (from/to memory disk&/dev/null; no risk of shoe shining, compression off).
>> 
>> SCSI bus looks happy; with no kernel messages. Output of the DLT script below.
>> 
>> Normal LTO drives in a tape robot, G9 server; pretty much all SAS *except* for a U320 to tape drive with the performance issue. Active terminator.
>> 
>> Does this ring a bell with anyone? Was anything changed in either the sa or mpt driver since 8.x ? 
>> 
>> One odd thing is that a 'dd(1)' -without- a block size (compression is off, data is a prepared urandom file on md disk) writes much faster -- while on 8.x one -had- to use a sane blockwise to get the 'normal' speed of around 120Mbyte/second. Could it be that one needs to fiddle with MAXPHYS (which AFAIK is a readonly sysctl).
>> 
> 
> What blocksize are you using?  What blocksize were you using before?  What
> application do you normally use to talk to the tape drive?

Amanda (or plain DD); we increased Amanda’s default of 32 to 64k (which stopped any and all shoe shining at that time. It was tested in 2013 (so likely in FreeBSD 8) up to 16 Mbyte in factors of 2 according to our notes - but with no further improvements in performance - and left at 64k).

The actual drive (mt status -v) reports (prior to increasing MAXPHYS) on 11.0-p2 with default kernel config:

> Drive: sa0: <QUANTUM ULTRIUM 4 W52T> Serial Number: HU1027B53Y
> ---------------------------------
> Mode      Density              Blocksize      bpi      Compression
> Current:  0x46:LTO-4           variable       323215   enabled (0x1)
> ---------------------------------
> Current Driver State: at rest.
> ---------------------------------
> Partition:   0      Calc File Number:   0     Calc Record Number: 0
> Residual:    0  Reported File Number:   0 Reported Record Number: 0
> Flags: BOP
> ---------------------------------
> Tape I/O parameters:
>   Maximum I/O size allowed by driver and controller (maxio): 131072 bytes
>   Maximum I/O size reported by controller (cpi_maxio): 131072 bytes
>   Maximum block size supported by tape drive and media (max_blk): 16777215 bytes
>   Minimum block size supported by tape drive and media (min_blk): 1 bytes
>   Block granularity supported by tape drive and media (blk_gran): 0 bytes
>   Maximum possible I/O size (max_effective_iosize): 131072 bytes

….
> If you want to re-enable I/O splitting (which I should have disabled by
> now), you can set the following loader tunable in /boot/loader.conf:
> 
> kern.cam.sa.allow_io_split=1

Ok - this changes behaviour - non-blocksized limited writes are now up; 2Mbyte/second; any block-sized write is 200kb/second (regardless of bs).

> If so, you probably want to look at the blocksize you're using, and bump up
> MAXPHYS in your kernel configuration to a larger value.

Post MAXPHYS & DFLTPHYS increase to 4MB (4x1024x1024) (as 16MB caused the kernel to hang during boot just post usb/knd - likely as it is there that it starts on SAS disk & tape discovery) I get
as ‘mt status -v’ the output:

> Tape I/O parameters:
>   Maximum I/O size allowed by driver and controller (maxio): 1372160 bytes
>   Maximum I/O size reported by controller (cpi_maxio): 1372160 bytes
>   Maximum block size supported by tape drive and media (max_blk): 16777215 bytes
>   Minimum block size supported by tape drive and media (min_blk): 1 bytes
>   Block granularity supported by tape drive and media (blk_gran): 0 bytes
>   Maximum possible I/O size (max_effective_iosize): 1372160 bytes

And see the same speeds; non blocksize limited (with dd) are 2.7Mbyte/second (so slightly higher); the blocksized writes are better (but still) low; 200kbyte/second for 16k, 600k for 32k, 800k for 64k, 500k for 128k, 1M at 256k, 1M at 1024k. The writes at 2048k and 4096k fail with a ‘hardware failure asc:44,0 — tape is now frozen” error).

This is with `kern.cam.sa.allow_io_split=1’ in /boot/loader.conf’. Doing so without that (after a reboot) give substantially similar results: (1.8Mbyte/second (0), 450k (16k), 560k (32k), 0.9M (64), 560k (128k), 730k (256k) and 600k (1M) — it fairly errors out with a si_iosize_max=1372160 for sizes above that.

Dw

#!/bin/sh
set -x

# create a ram disk of about 1 Gbyte with a noncompressible file on it.
#
mdconfig -a -s 1200M
newfs md0
mount /dev/md0 /mnt
dd if=/dev/urandom bs=1m count=1024 of=/mnt/test-1024m

for i in "" bs=16k bs=32k bs=64k bs=128k bs=256k bs=1024k bs=2048k bs=4196k
do
	mt rewind
	echo Run started with  ${i:-no blocksize set}
	dd if=/mnt/test-1024m $i of=/dev/nsa0
done