sa(4) driver changes available for test
Dan Langille
dan at langille.org
Mon Mar 2 16:10:00 UTC 2015
> On Mar 1, 2015, at 9:29 PM, Kenneth D. Merry <ken at FreeBSD.ORG> wrote:
>
> On Sun, Mar 01, 2015 at 19:41:07 -0500, Dan Langille wrote:
>>
>>> On Mar 1, 2015, at 7:31 PM, Kenneth D. Merry <ken at FreeBSD.ORG> wrote:
>>>
>>> On Sun, Mar 01, 2015 at 19:15:05 -0500, Dan Langille wrote:
>>>>
>>>>> On Feb 17, 2015, at 1:36 PM, Kenneth D. Merry <ken at freebsd.org> wrote:
>>>>>
>>>>> On Sat, Feb 14, 2015 at 18:22:43 -0500, Dan Langille wrote:
>>>>>>
>>>>>>> On Feb 13, 2015, at 7:32 PM, Kenneth D. Merry <ken at freebsd.org> wrote:
>>>>>>>
>>>>>>>
>>>>>>> I have a fairly large set of changes to the sa(4) driver and mt(1) driver
>>>>>>> that I'm planning to commit in the near future.
>>>>>>>
>>>>>>> A description of the changes is here and below in this message.
>>>>>>>
>>>>>>> If you have tape hardware and the inclination, I'd appreciate testing and
>>>>>>> feedback.
>>>>>>
>>>>>> I have a DLT 8000 and an SDLT 220.
>>>>>>
>>>>>> I don't have anything running current, but I have a spare machine which I could use for testing.
>>>>>>
>>>>>> Do you see any value is tests with that hardware? I'd be testing it via Bacula.
>>>>>>
>>>>>> disclosure: I'm the sysutils/bacula-* maintainer and a Bacula committer.
>>>>>>
>>>>>
>>>>> Actually, yes. Bacula is a bit tricky to configure, so your trying it out
>>>>> would be helpful if you have the time.
>>>>>
>>>>> In looking at the manuals for both the SDLT 220 and the DLT 8000, they both
>>>>> claim to support long position information for the SCSI READ POSITION
>>>>> command.
>>>>>
>>>>> You can see what I'm talking about by doing:
>>>>>
>>>>> mt eod
>>>>> mt status
>>>>>
>>>>> On my DDS-4 tape drive, this shows:
>>>>>
>>>>> # mt -f /dev/nsa3 status
>>>>> Drive: sa3: <SEAGATE DAT 06240-XXX 8071> Serial Number: HJ00YWY
>>>>> ---------------------------------
>>>>> Mode Density Blocksize bpi Compression
>>>>> Current: 0x26:DDS-4 1024 bytes 97000 enabled (DCLZ)
>>>>> ---------------------------------
>>>>> Current Driver State: at rest.
>>>>> ---------------------------------
>>>>> Partition: 0 Calc File Number: -1 Calc Record Number: -1
>>>>> Residual: 0 Reported File Number: -1 Reported Record Number: -1
>>>>> Flags: None
>>>>>
>>>>> But on an LTO-5, which will give long position information, I get:
>>>>>
>>>>> [root at doc ~]# mt status
>>>>> Drive: sa0: <IBM ULTRIUM-HH5 E4J1>
>>>>> ---------------------------------
>>>>> Mode Density Blocksize bpi Compression
>>>>> Current: 0x58:LTO-5 variable 384607 enabled (0x1)
>>>>> ---------------------------------
>>>>> Current Driver State: at rest.
>>>>> ---------------------------------
>>>>> Partition: 0 Calc File Number: 2 Calc Record Number: -1
>>>>> Residual: 0 Reported File Number: 2 Reported Record Number: 32373
>>>>> Flags: None
>>>>>
>>>>> That, in combination with the changes I made to the position information
>>>>> code in the driver, mean that even the old MTIOCGET ioctl should return an
>>>>> accurate file number at end of data. e.g., on the LTO-5:
>>>>>
>>>>> [root at doc ~]# mt ostatus
>>>>> Mode Density Blocksize bpi Compression
>>>>> Current: 0x58:LTO-5 variable 384607 0x1
>>>>> ---------available modes---------
>>>>> 0: 0x58:LTO-5 variable 384607 0x1
>>>>> 1: 0x58:LTO-5 variable 384607 0x1
>>>>> 2: 0x58:LTO-5 variable 384607 0x1
>>>>> 3: 0x58:LTO-5 variable 384607 0x1
>>>>> ---------------------------------
>>>>> Current Driver State: at rest.
>>>>> ---------------------------------
>>>>> File Number: 2 Record Number: -1 Residual Count -1
>>>>>
>>>>> So the thing to try, in addition to just making sure that Bacula continues
>>>>> to work properly, is to try setting this for the tape drive in
>>>>> bacula-sd.conf:
>>>>>
>>>>> Hardware End of Medium = yes
>>>>>
>>>>> It looks like the Bacula tape program (btape) has a test mode, and it would
>>>>> be good to run through the tests on one of the tape drives and see whether
>>>>> they work, and whether the results are different before and after the
>>>>> changes. I'm not sure how to enable the test mode.
>>>>
>>>> I have this in /usr/local/etc/bacula/bacula-sd.conf
>>>>
>>>> Device {
>>>> Name = DLT
>>>> Description = "QUANTUM DLT7000 1624"
>>>> Media Type = DLT
>>>> Archive Device = /dev/nsa1
>>>>
>>>> Autochanger = YES
>>>> Drive Index = 0
>>>>
>>>> Offline On Unmount = no
>>>> Hardware End of Medium = yes
>>>> BSF at EOM = yes
>>>> Backward Space Record = no
>>>> Fast Forward Space File = no
>>>> TWO EOF = yes
>>>> }
>>>>
>>>> FYI, http://www.freebsddiary.org/digital-tl891.php (from 2006) has a btape test on this same model.
>>>>
>>>> Here's the test I ran tonight:
>>>>
>>>> [root at cuppy:/usr/home/dan] # btape -c /usr/local/etc/bacula/bacula-sd.conf /dev/nsa1
>>>> Tape block granularity is 1024 bytes.
>>>> btape: butil.c:287-0 Using device: "/dev/nsa1" for writing.
>>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK
>>>> *test
>>>>
>>>> === Write, rewind, and re-read test ===
>>>>
>>>> I'm going to write 10000 records and an EOF
>>>> then write 10000 records and an EOF, then rewind,
>>>> and re-read the data to verify that it is correct.
>>>>
>>>> This is an *essential* feature ...
>>>>
>>>> btape: btape.c:1152-0 Wrote 10000 blocks of 64412 bytes.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:1168-0 Wrote 10000 blocks of 64412 bytes.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:1210-0 Rewind OK.
>>>> 10000 blocks re-read correctly.
>>>> Got EOF on tape.
>>>> 10000 blocks re-read correctly.
>>>> === Test Succeeded. End Write, rewind, and re-read test ===
>>>>
>>>> btape: btape.c:1277-0 Block position test
>>>> btape: btape.c:1289-0 Rewind OK.
>>>> Reposition to file:block 0:4
>>>> Block 5 re-read correctly.
>>>> Reposition to file:block 0:200
>>>> Block 201 re-read correctly.
>>>> Reposition to file:block 0:9999
>>>> Block 10000 re-read correctly.
>>>> Reposition to file:block 1:0
>>>> Block 10001 re-read correctly.
>>>> Reposition to file:block 1:600
>>>> Block 10601 re-read correctly.
>>>> Reposition to file:block 1:9999
>>>> Block 20000 re-read correctly.
>>>> === Test Succeeded. End Write, rewind, and re-read test ===
>>>>
>>>>
>>>>
>>>> === Append files test ===
>>>>
>>>> This test is essential to Bacula.
>>>>
>>>> I'm going to write one record in file 0,
>>>> two records in file 1,
>>>> and three records in file 2
>>>>
>>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1)
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK
>>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1)
>>>> btape: btape.c:1420-0 Now moving to end of medium.
>>>
>>> This is the critical piece. The test moves the tape to the end of the
>>> medium. With hardware position information, you can tell what filemark
>>> you're on. Without it, you can't.
>>>
>>>> btape: btape.c:622-0 tape_dev.c:345 ioctl MTIOCGET error on "DLT" (/dev/nsa1). ERR=No error: 0.
>>>> We should be in file 3. I am at file 0. This is NOT correct!!!!
>>>>
>>>> Append test failed. Attempting again.
>>>> Setting "Hardware End of Medium = no
>>>> and "Fast Forward Space File = no
>>>> and retrying append test.
>>>
>>> This is not surprsing, given that the drive doesn't support long read
>>> position data. (It's a SCSI-2 device.) So that means that Bacula will
>>> need to do it manually.
>>
>> Yes, I have nothing newer than SCSI-2. Even my SDLT is SCSI-2 but that
>> tape library is hooked up to a different computer and was doing backups today.
>
> So, here is one thing that we can try to see whether these drives support
> long position information, even though they only claim to be SCSI-2. If
> they do, we can potentially add a quirk (or autodetection) to enable it.
> The code currently doesn't bother asking drives that claim to be SCSI-2
> for long position information. (Because that feature was added in the
> SSC spec, which came after SCSI-2.)
>
> Issue a READ POSITION with the short form specified:
>
> camcontrol cmd sa1 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd
>
> Issue a READ POSITION with the vendor-specific block numbers:
>
> camcontrol cmd sa1 -v -c "34 1 0 0 0 0 0 0 0 0" -i 20 - |hd
>
> Issue a READ POSITION with the long form data:
>
> camcontrol cmd sa1 -v -c "34 6 0 0 0 0 0 0 0 0" -i 32 - |hd
>
> If it supports the last one, then I can put a quirk (or autodetection) in
> the driver and Bacula will get the hardware filemarks. You should try this
> on your SDLT as well. It may well support it.
Sadly, no:
[root at cuppy:~] # camcontrol cmd sa1 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd
00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 |....|
00000014
[root at cuppy:~] # camcontrol cmd sa1 -v -c "34 1 0 0 0 0 0 0 0 0" -i 20 - |hd
00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 |....|
00000014
[root at cuppy:~] # camcontrol cmd sa1 -v -c "34 6 0 0 0 0 0 0 0 0" -i 32 - |hd
camcontrol: error sending command
(pass2:ahc0:0:2:0): READ POSITION. CDB: 34 06 00 00 00 00 00 00 00 00
(pass2:ahc0:0:2:0): CAM status: SCSI Status Error
(pass2:ahc0:0:2:0): SCSI status: Check Condition
(pass2:ahc0:0:2:0): SCSI sense: ILLEGAL REQUEST asc:24,0 (Invalid field in CDB)
(pass2:ahc0:0:2:0): Command byte 1 bit 2 is invalid
[root at cuppy:~] #
The SDLT server is on 9.3 though:
[root at knew:/usr/home/dan] # camcontrol cmd sa1 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd
camcontrol: cam_lookup_pass: CAMGETPASSTHRU ioctl failed
cam_lookup_pass: No such file or directory
cam_lookup_pass: either the pass driver isn't in your kernel
cam_lookup_pass: or sa1 doesn't exist
[root at knew:/usr/home/dan] # uname -a
FreeBSD knew.unixathome.org 9.3-RELEASE-p10 FreeBSD 9.3-RELEASE-p10 #0: Tue Feb 24 21:28:03 UTC 2015 root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
[root at knew:/usr/home/dan] #
It took me a while to figure that out... there is no sa1 on *this* system.
But, my SDLT:
[root at knew:/usr/home/dan] # camcontrol cmd sa0 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd
00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 |....|
00000014
[root at knew:/usr/home/dan] # camcontrol cmd sa0 -v -c "34 1 0 0 0 0 0 0 0 0" -i 20 - |hd
00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 |....|
00000014
[root at knew:/usr/home/dan] # camcontrol cmd sa0 -v -c "34 6 0 0 0 0 0 0 0 0" -i 32 - |hd
00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020
[root at knew:/usr/home/dan] #
>
> Google didn't quickly produce a SCSI manual for the DEC drive, but the
> Quantum SDLT manual indicates that it supports long position data, despite
> identifying itself as a SCSI-2 drive.
>
>>> === Append files test ===
>>>>
>>>> This test is essential to Bacula.
>>>>
>>>> I'm going to write one record in file 0,
>>>> two records in file 1,
>>>> and three records in file 2
>>>>
>>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1)
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK
>>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1)
>>>> btape: btape.c:1420-0 Now moving to end of medium.
>>>> btape: btape.c:625-0 Moved to end of medium.
>>>> We should be in file 3. I am at file 3. This is correct!
>>>>
>>>> Now the important part, I am going to attempt to append to the tape.
>>>>
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1)
>>>> Done appending, there should be no I/O errors
>>>>
>>>> Doing Bacula scan of blocks:
>>>> 1 block of 64448 bytes in file 1
>>>> End of File mark.
>>>> 2 blocks of 64448 bytes in file 2
>>>> End of File mark.
>>>> 3 blocks of 64448 bytes in file 3
>>>> End of File mark.
>>>> 1 block of 64448 bytes in file 4
>>>> End of File mark.
>>>> Total files=4, blocks=7, bytes = 451,136
>>>> End scanning the tape.
>>>> We should be in file 4. I am at file 4. This is correct!
>>>>
>>>>
>>>> It looks like the test worked this time, please add:
>>>>
>>>> Hardware End of Medium = No
>>>>
>>>> Fast Forward Space File = No
>>>> to your Device resource in the Storage conf file.
>>>>
>>>> The above Bacula scan should have output identical to what follows.
>>>> Please double check it ...
>>>> === Sample correct output ===
>>>> 1 block of 64448 bytes in file 1
>>>> End of File mark.
>>>> 2 blocks of 64448 bytes in file 2
>>>> End of File mark.
>>>> 3 blocks of 64448 bytes in file 3
>>>> End of File mark.
>>>> 1 block of 64448 bytes in file 4
>>>> End of File mark.
>>>> Total files=4, blocks=7, bytes = 451,136
>>>> === End sample correct output ===
>>>>
>>>> If the above scan output is not identical to the
>>>> sample output, you MUST correct the problem
>>>> or Bacula will not be able to write multiple Jobs to
>>>> the tape.
>>>>
>>>> Skipping read backwards test because BSR turned off.
>>>>
>>>>
>>>> === Forward space files test ===
>>>>
>>>> This test is essential to Bacula.
>>>>
>>>> I'm going to write five files then test forward spacing
>>>>
>>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1)
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes.
>>>> btape: btape.c:1909-0 Wrote block to device.
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1)
>>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1)
>>>> btape: btape.c:1634-0 Now forward spacing 1 file.
>>>> We should be in file 1. I am at file 1. This is correct!
>>>> btape: btape.c:1646-0 Now forward spacing 2 files.
>>>> We should be in file 3. I am at file 3. This is correct!
>>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1)
>>>> btape: btape.c:1659-0 Now forward spacing 4 files.
>>>> We should be in file 4. I am at file 4. This is correct!
>>>>
>>>> btape: btape.c:1677-0 Now forward spacing 1 more file.
>>>> We should be in file 5. I am at file 5. This is correct!
>>>>
>>>> === End Forward space files test ===
>>>>
>>>>
>>>> Ah, I see you have an autochanger configured.
>>>> To test the autochanger you must have a blank tape
>>>> that I can write on in Slot 1.
>>>>
>>>> Do you wish to continue with the Autochanger test? (y/n): y
>>>>
>>>>
>>>> === Autochanger test ===
>>>>
>>>> 3301 Issuing autochanger "loaded" command.
>>>> Nothing loaded in the drive. OK.
>>>> 3303 Issuing autochanger "load 1 0" command.
>>>> 3303 Autochanger "load 1 0" status is OK.
>>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK
>>>> btape: btape.c:1564-0 Rewound "DLT" (/dev/nsa1)
>>>> btape: btape.c:1571-0 Wrote EOF to "DLT" (/dev/nsa1)
>>>>
>>>> The test autochanger worked!!
>>>
>>> Great, thanks for running the test! Looks like things are working as well
>>> as they were before.
>>>
>>>>>
>>>>>> I'll let the other Bacula devs know about this. They deal with the hardware. I work on PostgreSQL.
>>>>>>
>>>>>
>>>>> Thanks! If there are additional features they would like out of the tape
>>>>> driver, I'm happy to talk about it. (Or help if they'd like to use the new
>>>>> status reporting ioctl, MTIOCEXTGET or any of the other new ioctls.)
>>>>
>>>> Errors are interesting to me. Especially corrected errors. They are a good indicator of tape quality.
>>>>
>>>
>>> Yes. At least on modern drives, there is a good bit available in the log
>>> pages. You can try seeing what logs your drive supports by installing the
>>> sg3_utils package and running 'sg_logs sa1 --all'.
>>>
>>> Given the large amount of data available on some drives, and the difficulty
>>> distilling it down to a clear good/bad, I probably won't stick that in the
>>> 'mt status' output.
>>>
>>> But you can certainly take a look at it and have an idea of whether your
>>> particular tape/drive are experiencing issues.
>>
>> That's a lot of output: https://gist.github.com/dlangille/0e15a7fbf7acab56fd32
>
> That is a lot.
>
>> I will run some Bacula jobs soon. I'm still setting up config files.
>
> Thanks for all the testing, I really appreciate it!
>
> Ken
> --
> Kenneth Merry
> ken at FreeBSD.ORG
—
Dan Langille
http://langille.org/
More information about the freebsd-scsi
mailing list