comments on newfs raw disk ? Safe ? (7 terabyte array)

[Eric Anderson]

On 02/09/07 07:12, Ivan Voras wrote:
> Arone Silimantia wrote:
> 
>> dd if=/dev/zero of=/dev/da1 bs=1k count=1
>> newfs -m 0 /dev/da1
>> mount /dev/da1 /mnt
>>
>> And that's that.  But it seems too good to be true!  Can someone please 
>> comment on this scheme and if there are some hidden dangers or lack of 
>> functionality that I will regret in the future ?
> 
> No dangers at the system level - you can create your file system on any
> storage-like device, use it and mount it any way you want. Raw disks are
> a perfectly valid target.


As a side benefit, one might also get a performance *increase* because 
not having a slice/partition in the way might make the file system 
blocks line up better with the stripe size.  Scott Long has posted about 
this in the past, either here, or on -performance, or somewhere.  I 
can't find the thread right off, but it's out there (he wrote a very 
good description of what happens, why, etc).

FWIW, I almost always do it this way.


>> Will it fsck just like any other UFS2 partition I run ?  Can I run 
>> quotas and snapshots and everything else on it, just like normal ?
> 
> Yes.
> 
>> Other than the fact that I can't boot this, is there _any downside 
>> whatsoever_ to newfs'ing raw disk like this ?
> 
> Only "collateral" problems because of the partition size: a regular
> (non-softupdates) fsck will take a LONG time to finish and eat a LOT of
> memory while it's doing its stuff. You'll need a lot of swap space (1GB
> per TB? someone had empirical numbers on this, I'm sure) if you think
> you'll need to fsck it entirely. Creating snapshots will also take a
> long time on it, and you probably want to search the lists for
> recommendations about creating snapshots in a second level directory in
> order not to block the root directory. Related to this is
> background-fsck which works by creating snapshots, so you'll probably
> want to disable it.

I have 5 10Tb file systems (and some 2Tb ones, but who cares about those 
tiny things? :)), and I can tell you that an empty huge file system is 
pretty easily fsck-able, but a full one will kill you.  It greatly 
depends on how many files (inodes) you have used on the file system.  If 
you have a massive amount of small files, you'll be eating up a ton of 
memory.  My 'rule of thumb' for my data (which averages to about 
16k/file) is 1G of memory for each 1Tb  of disk space used.  So, on a 
10Tb file system, if I ever want the fsck to complete, I need an AMD64 
box with *at least* 10G of memory, plus a lot of time.  A *lot* of time. 
  By 'a lot', I mean anywhere from a day, to several days.


> In any case, try every feature you think you'll need before deploying it.
> 
> Also, write about your experience on this list :)

I second that.  It's important to share anything we can, so we see what 
others are doing, what the needs are, etc.

I also recommend looking at gjournal, now in -CURRENT.  I'm not sure if 
it's still considered beta, but care should be taken when using it of 
course.  However, I use it for many tens of TB of data, and I'm a happy 
gjournal fan (thanks Pawel!).

Eric