GEOM_MIRROR: Component concat/leg2 (device data) broken, skipping

[Ender]

I have a gmirror provider named "data" with 2 consumers "leg1" and 
"leg2". Each gconcat provider has 4 consumers. The gconcat consumers are 
actually ggatec devices.

Geom name: leg1
State: UP
Status: Total=4, Online=4
Type: AUTOMATIC
ID: 1899079681
Providers:
1. Name: concat/leg1
    Mediasize: 675976717312 (630G)
    Sectorsize: 512
    Mode: r1w1e1
Consumers:
1. Name: ggate14
    Mediasize: 95865781248 (89G)
    Sectorsize: 512
    Mode: r1w1e2
    Start: 580110936576
    End: 675976717312
2. Name: ggate11
    Mediasize: 120000000000 (112G)
    Sectorsize: 512
    Mode: r1w1e2
    Start: 0
    End: 119999999488
3. Name: ggate13
    Mediasize: 300069052416 (279G)
    Sectorsize: 512
    Mode: r1w1e2
    Start: 280041884672
    End: 580110936576
4. Name: ggate12
    Mediasize: 160041885696 (149G)
    Sectorsize: 512
    Mode: r1w1e2
    Start: 119999999488
    End: 280041884672

Geom name: leg2
State: UP
Status: Total=4, Online=4
Type: AUTOMATIC
ID: 260049114
Providers:
1. Name: concat/leg2
    Mediasize: 676009618432 (630G)
    Sectorsize: 512
    Mode: r0w0e0
Consumers:
1. Name: ggate21
    Mediasize: 120000000000 (112G)
    Sectorsize: 512
    Mode: r0w0e0
    Start: 0
    End: 119999999488
2. Name: ggate22
    Mediasize: 160041885696 (149G)
    Sectorsize: 512
    Mode: r0w0e0
    Start: 119999999488
    End: 280041884672
3. Name: ggate23
    Mediasize: 300069052416 (279G)
    Sectorsize: 512
    Mode: r0w0e0
    Start: 280041884672
    End: 580110936576
4. Name: ggate24
    Mediasize: 95898682368 (89G)
    Sectorsize: 512
    Mode: r0w0e0
    Start: 580110936576
    End: 676009618432

# ggatec list
ggate14
ggate11
ggate13
ggate12
ggate21
ggate22
ggate23
ggate24

The problem is my gmiror degraded.
# gmirror list
Geom name: data
State: DEGRADED
Components: 2
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 1
SyncID: 1
ID: 3615665854
Providers:
1. Name: mirror/data
    Mediasize: 675976716800 (630G)
    Sectorsize: 512
    Mode: r1w1e1
Consumers:
1. Name: concat/leg1
    Mediasize: 675976717312 (630G)
    Sectorsize: 512
    Mode: r1w1e1
    State: ACTIVE
    Priority: 0
    Flags: NONE
    GenID: 1
    SyncID: 1
    ID: 1250101353


And when i try andn insert a new disk i get this>
GEOM_MIRROR: Component concat/leg2 (device data) broken, skipping.
GEOM_MIRROR: Cannot add disk concat/leg2 to data (error=22).

I set kern.geom.mirror.debug: 1, and i tryed to dd if=/dev/zero 
of=/dev/ad* for each disk in leg2
as shown here: 
http://unix.derkeiler.com/Mailing-Lists/FreeBSD/questions/2005-01/2574.html

no luck, any suggestions?


Here is what i did to start this setup:
#ON 10.10.1.10
ggatec create -o rw -q 2048 -S 4194304 -R 4194304 -t 600 -u 11 
10.10.1.101 /dev/ad1
ggatec create -o rw -q 2048 -S 4194304 -R 4194304 -t 600 -u 12 
10.10.1.101 /dev/ad2
ggatec create -o rw -q 2048 -S 4194304 -R 4194304 -t 600 -u 13 
10.10.1.101 /dev/ad3
ggatec create -o rw -q 2048 -S 4194304 -R 4194304 -t 600 -u 14 
10.10.1.101 /dev/ad0s1f
ggatec create -o rw -q 2048 -S 4194304 -R 4194304 -t 600 -u 21 
10.10.1.102 /dev/ad1
ggatec create -o rw -q 2048 -S 4194304 -R 4194304 -t 600 -u 22 
10.10.1.102 /dev/ad2
ggatec create -o rw -q 2048 -S 4194304 -R 4194304 -t 600 -u 23 
10.10.1.102 /dev/ad3
ggatec create -o rw -q 2048 -S 4194304 -R 4194304 -t 600 -u 24 
10.10.1.102 /dev/ad0s1f

#ON 10.10.1.10
gconcat label -v leg1 /dev/ggate11 /dev/ggate12 /dev/ggate13 /dev/ggate14
gconcat label -v leg2 /dev/ggate21 /dev/ggate22 /dev/ggate23 /dev/ggate24
gmirror label -v -b round-robin data /dev/concat/leg1 /dev/concat/leg2
newfs -U /dev/mirror/data
mount /dev/mirror/data /data


#ON 10.10.1.101 and 10.10.1.102
#!/bin/sh
#

# PROVIDE: ggated
# REQUIRE: bar_service_required_to_precede_foo

. /etc/rc.subr

name="ggated"
rcvar=`set_rcvar`
command="/sbin/ggated"
command_args="-R 4194304 -S 4194304"

load_rc_config $name
run_rc_command "$1"