RELENG_8 -- shutdown -p no longer powers off boxes

[Jeremy Chadwick]

On Fri, Apr 23, 2010 at 05:20:00AM -0700, Jeremy Chadwick wrote:
> I'll spend some time later today messing around with combinations of
> ifconfig em0 -wol_ucast and -wol_mcast to see if I can narrow down the
> condition.  I'll report back with those findings.

Here are those results.  First, minor details of the setup:

box1 = 192.168.1.51, Supermicro X7SBA, connected to switch via em0
box2 = 192.168.1.52, Supermicro X7SBL-LN2, connected to switch via em0

Netmask is 255.255.255.0; absolutely nothing fancy about the setup.

My ifconfig "tests", which show some strange behaviour:

box2# ifconfig em0 | head -2
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=399b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_UCAST,WOL_MCAST,WOL_MAGIC>

box2# ifconfig em0 -wol_ucast
box2# ifconfig em0 | head -2
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=399b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_UCAST,WOL_MCAST,WOL_MAGIC>

box2# ifconfig em0 -wol_mcast
box2# ifconfig em0 | head -2
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=299b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_UCAST,WOL_MAGIC>

box2# ifconfig em0 wol
box2# ifconfig em0 | head -2
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=399b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_UCAST,WOL_MCAST,WOL_MAGIC>

box2# ifconfig em0 -wol_magic
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=199b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_UCAST,WOL_MCAST>

box2# ifconfig em0 -wol
box2# ifconfig em0 | head -2
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=99b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_UCAST>

box2# ifconfig em0 -wol_ucast
box2# ifconfig em0 | head -2
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=99b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_UCAST>

box2# ifconfig em0 wol_mcast -wol_ucast
box2# ifconfig em0 | head -2
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=199b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_UCAST,WOL_MCAST>

box2# ifconfig em0 wol
box2# ifconfig em0 -wol_mcast
box2# ifconfig em0 | head -2
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=299b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_UCAST,WOL_MAGIC>

At this point on box2, I executed the following:

box2# shutdown -p now

Then I did the following tests from box1.  To rule out any chance of ARP
expiring, I added a static ARP entry for box2's MAC --> 192.168.1.52.

1. ping 192.168.1.255 -- box2 did not power on
2. ping 192.168.1.52 -- box2 did not power on
3. ping 255.255.255.255 -- box2 did not power on
4. sent magic packet to 255.255.255.255 UDP port 40000 -- box2 did not power on
5. sent magic packet to 192.168.1.255 UDP port 40000 -- box2 powered on
6. sent magic packet to 192.168.1.52 UDP port 40000 -- box2 powered on

Between test #5 and #6, I executed "ifconfig em0 -wol_mcast" on box2 prior
to doing the shutdown.

Based on all this, we can more or less reliably tell that:

1) ifconfig's behaviour is unintuitive in this case; this could be a
   driver (7.0.5) bug, or it could be some odd capabilities bug.  I
   don't know.  If this is intentional behaviour, it's unintuitive,
   and there's no mention of it (or WOL!) in the em(4) man page.
   Likewise, ifconfig(8)'s descriptions of what the WOL_xxx flags do
   is also a bit unintuitive.

2) When WOL_MCAST capability disabled on the interface, the system
   behaviour is identical to that of the older (pre-7.0.5) driver.

3) WOL_MCAST may be buggy in some way, or not implemented correctly
   inside of the em (7.0.5) driver.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |