sa and mpt in FreeBSD 11.0-RELEASE-p2

Fri Mar 3 22:23:24 UTC 2017

On 21 Feb 2017, at 21:45, Kenneth D. Merry <ken at freebsd.org> wrote:
> 
> On Tue, Feb 21, 2017 at 18:15:06 +0100, Dirk-Willem van Gulik wrote:
>> 
>>> On 20 Feb 2017, at 21:56, Kenneth D. Merry <ken at FreeBSD.ORG> wrote:
>>> 
>>> On Mon, Feb 20, 2017 at 21:04:11 +0100, Dirk-Willem van Gulik wrote:
>>>> A machine, recently upgraded from 8.4 to FreeBSD 11.0-RELEASE-p2 seems to have lost pretty much any and all performance on mpt(4) with its attached tape drives
>>>> 
>>>> Read performance is around 50Mbyte/second - and write a paltry 100-200kbyte/second (and occasionally hitting 800kbyte/second) (from/to memory disk&/dev/null; no risk of shoe shining, compression off).
>>>> 
>>>> SCSI bus looks happy; with no kernel messages. Output of the DLT script below.
>>>> 
>>>> Normal LTO drives in a tape robot, G9 server; pretty much all SAS *except* for a U320 to tape drive with the performance issue. Active terminator.
>>>> 
>>>> Does this ring a bell with anyone? Was anything changed in either the sa or mpt driver since 8.x ? 
>>>> 
>>>> One odd thing is that a 'dd(1)' -without- a block size (compression is off, data is a prepared urandom file on md disk) writes much faster -- while on 8.x one -had- to use a sane blockwise to get the 'normal' speed of around 120Mbyte/second. Could it be that one needs to fiddle with MAXPHYS (which AFAIK is a readonly sysctl).
>>>> 
>>> 
>>> What blocksize are you using?  What blocksize were you using before?  What
>>> application do you normally use to talk to the tape drive?
>> 
>> Amanda (or plain DD); we increased Amanda???s default of 32 to 64k (which stopped any and all shoe shining at that time. It was tested in 2013 (so likely in FreeBSD 8) up to 16 Mbyte in factors of 2 according to our notes - but with no further improvements in performance - and left at 64k).
>> 
> This tells me that Amanda is keeping the tape driver well supplied with
> I/O.  Since 64K was probably the blocksize going down to the hardware
> previously, it makes sense that increasing it in Amanda didn't make a
> difference.

Ok. clear. Thanks. !
> 
>> The actual drive (mt status -v) reports (prior to increasing MAXPHYS) on 11.0-p2 with default kernel config:
>> 
>>> Drive: sa0: <QUANTUM ULTRIUM 4 W52T> Serial Number: HU1027B53Y
>>> ---------------------------------
>>> Mode      Density              Blocksize      bpi      Compression
>>> Current:  0x46:LTO-4           variable       323215   enabled (0x1)
>>> ---------------------------------
>>> Current Driver State: at rest.
>>> ---------------------------------
>>> Partition:   0      Calc File Number:   0     Calc Record Number: 0
>>> Residual:    0  Reported File Number:   0 Reported Record Number: 0
>>> Flags: BOP
>>> ---------------------------------
>>> Tape I/O parameters:
>>>  Maximum I/O size allowed by driver and controller (maxio): 131072 bytes
>>>  Maximum I/O size reported by controller (cpi_maxio): 131072 bytes
>>>  Maximum block size supported by tape drive and media (max_blk): 16777215 bytes
>>>  Minimum block size supported by tape drive and media (min_blk): 1 bytes
>>>  Block granularity supported by tape drive and media (blk_gran): 0 bytes
>>>  Maximum possible I/O size (max_effective_iosize): 131072 bytes
>> 
>> ???.
>>> If you want to re-enable I/O splitting (which I should have disabled by
>>> now), you can set the following loader tunable in /boot/loader.conf:
>>> 
>>> kern.cam.sa.allow_io_split=1
>> 
>> Ok - this changes behaviour - non-blocksized limited writes are now up; 2Mbyte/second; any block-sized write is 200kb/second (regardless of bs).
>> 
>>> If so, you probably want to look at the blocksize you're using, and bump up
>>> MAXPHYS in your kernel configuration to a larger value.
>> 
>> Post MAXPHYS & DFLTPHYS increase to 4MB (4x1024x1024) (as 16MB caused the kernel to hang during boot just post usb/knd - likely as it is there that it starts on SAS disk & tape discovery) I get
>> as ???mt status -v??? the output:
>> 
>>> Tape I/O parameters:
>>>  Maximum I/O size allowed by driver and controller (maxio): 1372160 bytes
>>>  Maximum I/O size reported by controller (cpi_maxio): 1372160 bytes
>>>  Maximum block size supported by tape drive and media (max_blk): 16777215 bytes
>>>  Minimum block size supported by tape drive and media (min_blk): 1 bytes
>>>  Block granularity supported by tape drive and media (blk_gran): 0 bytes
>>>  Maximum possible I/O size (max_effective_iosize): 1372160 bytes
>> 
>> 
>> And see the same speeds; non blocksize limited (with dd) are 2.7Mbyte/second (so slightly higher); the blocksized writes are better (but still) low; 200kbyte/second for 16k, 600k for 32k, 800k for 64k, 500k for 128k, 1M at 256k, 1M at 1024k. The writes at 2048k and 4096k fail with a ???hardware failure asc:44,0 ??? tape is now frozen??? error).
>> 
>> This is with `kern.cam.sa.allow_io_split=1??? in /boot/loader.conf???. Doing so without that (after a reboot) give substantially similar results: (1.8Mbyte/second (0), 450k (16k), 560k (32k), 0.9M (64), 560k (128k), 730k (256k) and 600k (1M) ??? it fairly errors out with a si_iosize_max=1372160 for sizes above that.
>> 
> 
> The fact that you're getting substantially higher performance with I/O
> splitting enabled points to probable delays in getting the data into the
> kernel.
> 
> Instead of doing a single threaded read/write dd test, I would suggest
> doing something like this:
> 
> dd if=inputfile bs=10m | dd of=/dev/nsa0 bs=1m

This does not change the numbers at all (with a PHYSMEM=4M,  kern.cam.sa.0.*maxio reporting 1372160 and with or without splitting).

We've also tried a different SCSI card (albeit also an LSI one - so also mpt(4)); which gives near idential numbers - around 600k. 

> Also, the fact that you're getting a hardware failure with larger writes
> may mean that the tape drive might not support what it claims to.  The
> hardware failure is probably coming from the tape drive.

Do we've downgraded the machine back to FreeBSD 8.4-- and tested the same setup against the same drive. That gave us normal LTO4 speeds. We tried two mpt cards and one adaptec card.

I guess that suggest it is not a drive, card or cable issue. 

However we are seeing a difference in the output of the pciconf command with respect to the busses and order of the cards detected; and see an adaptec card work fine in both 8.4 and 11.0-p7 (and the adaptec cards are reported identical ordered).

So this may well be sa(4) unrelated - but PCI or LSI card specific.

But thanks to your help - we do have a working solution (and able to rid the last machines off LTO3 and LTO4).

Thanks!

Dw