cxl nic not working after reboot

[Marius Halden]

On Mon, Aug 27, 2018, at 17:13, Navdeep Parhar wrote:
> On Mon, Aug 27, 2018 at 04:19:29PM +0200, Marius Halden wrote:
> > Hi,
> > 
> > We have some routers with Chelsio T540-CR NICs using 1Gbps SFPs (1000
> > Base-LX IIRC) to connect to our ISPs. After upgrading them to FreeBSD
> > 11.2-p2 (BSDRP v1.91) we have run into some issues. When the machines
> > are rebooted networking does not function at all through the ports
> > connected with SFPs, ports connected with DAC cables work properly.
> > According to ifconfig the SFPs are detected properly and they both
> 
> What exact ifconfig command do you use to bring the interfaces up?

It's just the normal rc.conf lines with `ifconfig_cxl0="inet …"` and `ifconfig_cxl0_ipv6="inet6 …"`, nothing special beyond that.

> Provide the output of these commands when the link isn't up:
> # ifconfig -mvvv <interface>
> # sysctl -n dev.t5nex.0.misc.devlog

Thanks, I'll try these when I'm back at work tomorrow and report back.

> > send and receive light as they should. If we remove the SFPs from the
> > port and then reinsert them in the same port everything will start
> > working as it should.
> > 
> > Has anyone else observed this problem? Any suggestions for what can be
> > done to solve/work around this issue without actually going to the
> > datacenter would be great.
> 
> Try ifconfig down/up instead of physically removing and then reinserting
> the SFP module.  Does that work?

ifconfig down/up did not help, I tried that first (I should have mentioned it in the original email). Figuring out it would work by physically removing and reinserting the SFP was a side effect of initially trying to figure out if this was a hardware issue.

-- 
Marius Halden