if_ixl: Admin Queue memory allocation issue hangs the server

From: Antranig Vartanian <antranigv_at_freebsd.am>
Date: Thu, 01 Feb 2024 16:07:00 UTC
Greetings y’all,

We have a massive server with two IXL NICs (Ethernet Controller X710 for 10GBASE-T). One of them is on the host and the other is passed to a guest running in a bhyve VM.

Every once in a while, the system hangs. We cannot SSH, we cannot ping nor we can login using console (IPMI remote screen).

However, on the console, I see the following messages:

"ixl0: ixl_process_adminq: Unable to allocate memory for Admin Queue event!”

And it keeps repeating.

The only way to get access to the host is to… reboot it.

We don’t use SR-IOV (although when we did, the same thing happened), we don’t use large MTUs (1500) nor we have a memory issue (the system has 2TB of RAM, and right before hanging around 250G is available).

Any tip would be appreciated.

P.S. we also tried the kmod from ports, still seeing the same issue.

Kind regards,

—
Antranig Vartanian
https://antranigv.am/
PGP Key ID: 0x2D59F21C