How to force IB cards to come up in IB mode?

Sat Apr 16 15:22:28 UTC 2016

Manually setting the ports via sysctl after boot works, although
in a kind of strange fashion.

When using the sysctl command to set the port type, the first
attempt (regardless of which port) always fails:

  # sysctl sys.device.mlx4_core0.mlx4_port1="ib"
  sys.device.mlx4_core0.mlx4_port1: eth
  sysctl: sys.device.mlx4_core0.mlx4_port1=ib: Invalid argument

The 2nd attempt, to the other port, then succeeds:

  # sysctl sys.device.mlx4_core0.mlx4_port2="ib"
  sys.device.mlx4_core0.mlx4_port2: eth -> ib

It doesn't matter which order or which ports this is done in. 
The 1st attempt always fails, the second attempt to "the other"
port will then work.

After the 2nd port has been changed, and IB mode becomes active, the
IP addressing set in rc.conf seems to automatically take effect.

eg:

  ib0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520
          options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
          lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.2.c9.3.0.0.ca.cd
          inet 172.16.1.1 netmask 0xffffff00 broadcast 172.17.1.255
          nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  ib1: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520
          options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
          lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.0.2.c9.3.0.0.ca.ce
          inet 172.16.2.1 netmask 0xffffff00 broadcast 172.17.1.255 
          nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

While that's ok-ish for a once off thing, having some way for the
cards to know which mode to start in seems like a better approach.

Is the some way to get loader.conf (or anything really) to make that
happen? :)

Regards and best wishes,

Justin Clift

On 16 Apr 2016, at 15:55, Meny Yossefi <menyy at mellanox.com> wrote:
> Hi Justin,
> 
> It seems as the pre-boot configuration didn't work.
> 
> loader.conf doesn't work on all sysctl params AFAIK.
> 
> Is it a pre-boot only issue?
> 
> What is the current sysctl output for those ports?
> 
> Have you tried switching to IB after system was loaded?
> 
> If I remember correctly, the driver was set for Eth configuration by default.
> 
> Of course, this is reversible..
> 
> 
> 
> Regards,
> Meny Yossefi | SW Manager | FreeBSD Team
> Mellanox Technologies Ltd
> Work: +972-74-7129121, Cell: +972-52-8379557
> 
> 
> 
> ________________________________________
> 
> From: owner-freebsd-infiniband at freebsd.orgOn<mailto:owner-freebsd-infiniband at freebsd.orgOn> Behalf OfJustin Clift
> 
> Sent: Saturday, April 16, 2016 11:15:11 AM (UTC) Monrovia, Reykjavik
> 
> To: freebsd-infiniband at freebsd.org<mailto:freebsd-infiniband at freebsd.org>
> 
> Subject: How to force IB cards to come up in IB mode?
> 
> 
> 
> Hi all,
> 
> 
> 
> I'm trying to get a new FreeBSD 10.3-STABLE install to work well with my MHGH28-XTC (dual port ConnectX DDR) cards.
> 
> 
> 
> The problem is they're coming up in eth mode, not ib mode, and I can't seem to get that fixed. :(
> 
> 
> 
> From boot:
> 
> 
> 
>  # ifconfig
> 
>  mlxen0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 
>          options=6d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> 
>          ether 00:02:c9:00:ca:cd
> 
>          nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> 
>          media: Ethernet autoselect
> 
>          status: no carrier
> 
>  mlxen1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 
>          options=6d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> 
>          ether 00:02:c9:00:ca:ce
> 
>          nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> 
>          media: Ethernet autoselect
> 
>          status: no carrier
> 
> 
> 
> My loader.conf file seems like it should be correct:
> 
> 
> 
>  # cat /boot/loader.conf
> 
>  sys.device.mlx4_core0.mlx4_port1="ib"
> 
>  sys.device.mlx4_core0.mlx4_port2="ib"
> 
> 
> 
> As background, this is a new install (coming from CentOS 6.x, where it all works well), so I know there's no problem with the cards.  They're plugged into a 24 port DDR switch (Voltaire ISR 9024D-M) using active optical cables.
> 
> 
> 
> Any ideas?
> 
> 
> 
> Regards and best wishes,
> 
> 
> 
> Justin Clift
> 
> 
> 
> --
> 
> "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there."
> 
> - Indira Gandhi
> 
> 
> 
> _______________________________________________
> 
> freebsd-infiniband at freebsd.org<mailto:freebsd-infiniband at freebsd.org> mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
> 
> To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe at freebsd.org<mailto:freebsd-infiniband-unsubscribe at freebsd.org>"
> _______________________________________________
> freebsd-infiniband at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
> To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe at freebsd.org"

--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi