can not boot from RAIDZ with 8-STABLE

[Miroslav Lachman]

Artem Belevich wrote:
> On Wed, Aug 17, 2011 at 12:40 PM, Miroslav Lachman<000.fbsd at quip.cz>  wrote:
>> Thank you guys, you are right. The BIOS provides only 1 disk to the loader!
>> I checked it from loader prompt by lsdev (booted from USB external HDD).
>>
>> So I will try to make a small zpool mirror for root and boot (if ZFS mirror
>> can be made of 4 providers instead of two) and the rest will be in RAIDZ.
>>
>> If that fails, I will go my old way with internal USB flash disk with UFS
>> for booting and RAIDZ of 4 disks for storage as I did it few years ago with
>> 7.0 or 7.1.
>
> You seem to be booting from disks attached to some sort of add-on
> card. Sometimes those have per-disk 'bootable' option in their own
> extension ROM. You may investigate yours. Perhaps all you need to do
> is just tweak controller settings.

Advanced controller settings allows me to choose which disk will be 
bootable - but I can mark just one of them, not all.

So my working setup is made from 2 pools. First is 4 way ZFS mirror for 
/ (root), second is RAIDZ for the rest.
(plus swap made on the top of gmirrored partitions)

Each disk has following partitions:

# gpart show da0
=>       34  976773101  da0  GPT  (465G)
          34        128    1  freebsd-boot  (64k)
         162    8388608    2  freebsd-swap  (4.0G)
     8388770   20971520    3  freebsd-zfs  (10G)
    29360290  943718400    4  freebsd-zfs  (450G)
   973078690    3694445       - free -  (1.8G)


# zpool list
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
sys    9.94G   781M  9.17G     7%  1.00x  ONLINE  -
tank   1.75T  4.77G  1.75T     0%  1.00x  ONLINE  -


Filesystem                 Size    Mounted on
sys/root                   9.8G    /
devfs                      1.0k    /dev
tank/tmp                   1.3T    /tmp
tank/usr/home              1.3T    /usr/home
tank/usr/home/quip         1.3T    /usr/home/quip
tank/usr/local             1.3T    /usr/local
tank/usr/obj               1.3T    /usr/obj
tank/usr/ports             1.3T    /usr/ports
tank/usr/ports/distfiles   1.3T    /usr/ports/distfiles
tank/usr/ports/packages    1.3T    /usr/ports/packages
tank/usr/src               1.3T    /usr/src
tank/var/amavis            1.3T    /var/amavis
tank/var/audit             1.3T    /var/audit
tank/var/crash             1.3T    /var/crash
tank/var/db                1.3T    /var/db
tank/var/db/mysql          1.3T    /var/db/mysql
tank/var/log               1.3T    /var/log
tank/var/mail              1.3T    /var/mail
tank/var/tmp               1.3T    /var/tmp
tank/var/virusmails        1.3T    /var/virusmails
tank/vol0                  1.3T    /vol0


I hope that it helps to somebody with similar problem.

Miroslav Lachman