Using ccd with zfs

[John Nielsen]

On Tuesday 22 July 2008 12:18:31 am Steven Schlansker wrote:
> Hello -questions,
> I have a FreeBSD ZFS storage system working wonderfully with 7.0.
> It's set up as three 3-disk RAIDZs -triplets of 500, 400, and 300GB
> drives.
>
> I recently purchased three 750GB drives and would like to convert to
> using a RAIDZ2.  As ZFS has no restriping capabilities yet, I will
> have to nuke the zpool from orbit and make a new one.  I would like to
> verify my methodology against your experience to see if what I wish to
> do is reasonable:
>
> I plan to first take 2 of the 750GB drives and make an unreplicated
> 1.5TB zpool as a temporary storage.  Since ZFS doesn't seem to have
> the ability to create zpools in degraded mode (with missing drives) I
> plan to use iSCSI to create two additional drives (backed by /dev/
> zero) to fake having two extra drives, relying on ZFS's RAIDZ2
> protection to keep everything running despite the fact that two of the
> drives are horribly broken ;)
>
> To make these 500, 400, and 300GB drives useful, I would like to
> stitch them together using ccd.  I would use it as 500+300 = 800GB and
> 400+400=800GB
>
> That way, in the end I would have
> 750 x 3
> 500 + 300 x3
> 400 + 400 x 1
> 400 + 200 + 200 x 1
> as the members in my RAIDZ2 group.  I understand that this is slightly
> less reliable than having "real" drives for all the members, but I am
> not interested in purchasing 5 more 750GB drives.  I'll replace the
> drives as they fail.
>
> I am wondering if there are any logistical problems.  The three parts
> I am worried about are:
>
> 1) Are there any problems with using an iSCSI /dev/zero drive to fake
> drives for creation of a new zpool, with the intent to replace them
> later with proper drives?

I don't know about the iSCSI approach but I have successfully created a 
degraded zpool using md and a sparse file in place of the missing disk. 
Worked like a charm and I was able to transfer everything to the zpool 
before nuking the real device (which I had been using for temporary 
storage) and replacing the md file with it.

You can create a sparse file using dd:
	dd if=/dev/zero of=sparsefile bs=512 seek=(size of the fake device in 
512-byte blocks) count=0

Turn it into a device node using mdconfig:
	mdconfig -a -t vnode -f sparsefile

Then create your zpool using the /dev/md0 device (unless the mdconfig 
operation returns a different node number).

The size of the sparse file should not be bigger than the size of the real 
device you plan to replace it with. If using GEOM (which I think you 
should, see below), be sure to remember to subtract 512 bytes for each 
level of each provider (GEOM modules store their metadata in the last 
sector of each provider so that space is unavailable for use). To be on the 
safe side you can whack a few KB off.

You can't remove the fake device from a running zpool but the first time you 
reboot it will be absent and the zpool will come up degraded.

> 2) Are there any problems with using CCD under zpool?  Should I stripe
> or concatenate?  Will the startup scripts (either by design or less
> likely intelligently) decide to start CCD before zfs?  The zpool
> should start without me interfering, correct?

I would suggest using gconcat rather than CCD. Since it's a GEOM module (and 
you will have remembered to load it via /boot/loader.conf) it will 
initialize its devices before ZFS starts. It's also much easier to set up 
than CCD. If you are concatenating two devices of the same size you could 
consider using gstripe instead, but think about the topology of your drives 
and controllers and the likely usage patterns your final setup will create 
to decide if that's a good idea.

> 3) I hear a lot about how you should use whole disks so ZFS can enable
> write caching for improved performance.  Do I need to do anything
> special to let the system know that it's OK to enable the write
> cache?  And persist across reboots?

Not that I know of. As I understand it ZFS _assumes_ it's working with whole 
disks so since it uses its own i/o scheduler performance can be degraded 
for anything sharing a physical device with a ZFS slice.

> Any other potential pitfalls?  Also, I'd like to confirm that there's
> no way to do this pure ZFS-like - I read the documentation but it
> doesn't seem to have support for nesting vdevs (which would let me do
> this without ccd)

You're right, you can't do this with ZFS alone. Good thing FreeBSD is so 
versatile. :)

JN

> Thanks for any information that you might be able to provide,
> Steven Schlansker
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to
> "freebsd-questions-unsubscribe at freebsd.org"