RFC: Suggesting ZFS "best practices" in FreeBSD

[Jason Keltz]

On 22/01/2013 9:16 PM, Warren Block wrote:
> On Tue, 22 Jan 2013, Michael DeMan wrote:
>
>> On Jan 22, 2013, at 7:04 AM, Warren Block <wblock at wonkity.com> wrote:
>>>
>>> I'm a proponent of using various types of labels, but my impression 
>>> after a recent experience was that ZFS metadata was enough to 
>>> identify the drives even if they were moved around. That is, ZFS 
>>> bare metadata on a drive with no other partitioning or labels.
>>>
>>> Is that incorrect?
>>
>> I don't know if it is correct or not, but the best I could figure out 
>> was to both label the drives and also force the mapping so the 
>> physical and logical drives always show up associated correctly. I 
>> also ended up deciding I wanted the hostname as a prefix for the 
>> labels - so if they get moved around to say another machine I can 
>> look and know what is going on - 'oh yeah, those disks are from the 
>> ones we moved over to this machine'...
>
> It helps to avoid duplicate labels, a good idea.
>
>> #1.  Map the physical drive slots to how they show up in FBSD so if a 
>> disk is removed and the machine is rebooted all the disks after that 
>> removed one do not have an 'off by one error'.  i.e. if you have 
>> ada0-ada14 and remove ada8 then reboot - normally FBSD skips that 
>> missing ada8 drive and the next drive (that used to be ada9) is now 
>> called ada8 and so on...
>
> How do you do that?  If I'm in that situation, I think I could find 
> the bad drive, or at least the good ones, with diskinfo and the drive 
> serial number.  One suggestion I saw somewhere was to use disk serial 
> numbers for label values.
I think that was using /boot/device.hints.  Unfortunately it only works 
for some systems, and not for all..  and someone shared an experience 
with me where a kernel update caused the card probe order to change, the 
devices to change, and then it all broke...  It worked for one card, not 
for the other...  I gave up because I wanted consistency across 
different systems..

In my own opinion, the whole process of partitioning drives, labelling 
them, all kinds of tricks for dealing with 4k drives, manually 
configuring /boot/device.hints, etc. is something that we have to do, 
but honestly, I really believe there *has* to be a better way....  Years 
back when I was using a 3ware/AMCC RAID card (actually, I AM still using 
a few), none of this was an issue... every disk just appeared in order.. 
I didn't have to configure anything specially ..  ordering never changed 
when I removed a drive, I didn't need to partition or do anything with 
the disks - just give it the raw disks, and it knew what to do...  If 
anything, I took my labeller and labelled the disk bays with a numeric 
label so when I got an error, I knew which disk to pull, but order never 
changed, and I always pulled the right drive... Now, I look at my pricey 
"new" system, see disks ordered by default in what seems like an almost 
"random" order... I dded each drive to figure out the exact ordering, 
and labelled the disks, but it just gets really annoying....

>
>> #2.  Use gpart+gnop to deal with 4K disk sizes in a standardized way 
>> and also to leave a little extra room so if when doing a replacement 
>> disk and that disk is a few MB smaller than the original - it all 
>> 'just works'.  (All disks are partitioned to a slightly smaller size 
>> than their physical capacity).
>
> I've been told (but have not personally verified) that newer versions 
> of ZFS actually leaves some unused space at the end of a drive to 
> allow for variations in nominally-sized drives.  Don't know how much.

You see... this point has been mentioned on the list a whole bunch of 
times, and is exactly the type of information that needs to make it into 
a "best practices".  Does anyone know if this applies to ZFS in 
FreeBSD?  From what version?  Who knows, maybe a whole bunch of people 
are partitioning devices that don't need to be!  :)

Jason.


-- 
Jason Keltz
Manager of Development
Department of Computer Science and Engineering
York University, Toronto, Canada
Tel: 416-736-2100 x. 33570
Fax: 416-736-5872