[Bug 269133] bnxt(4): BCM57416 - HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error
- In reply to: bugzilla-noreply_a_freebsd.org: "[Bug 269133] bnxt(4): BCM57414 / BCM57416"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Tue, 16 Apr 2024 17:52:09 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269133
Kenneth D. Merry <ken@FreeBSD.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |ken@FreeBSD.org
--- Comment #64 from Kenneth D. Merry <ken@FreeBSD.org> ---
I have a machine with a BCM57414 on the motherboard:
bnxt0: <Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet> mem
0xd1d10000-0xd1d1ffff,0xd1c00000-0xd1cfffff,0xd1d22000-0xd1d23fff irq 36 at
device 0.0 numa-domain 0 on pci3
bnxt0: Using 256 TX descriptors and 256 RX descriptors
bnxt0: Using 16 RX queues 16 TX queues
bnxt0: Using MSI-X interrupts with 17 vectors
bnxt0: Ethernet address: 9c:6b:00:46:a2:0c
bnxt1: <Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet> mem
0xd1d00000-0xd1d0ffff,0xd1b00000-0xd1bfffff,0xd1d20000-0xd1d21fff irq 37 at
device 0.1 numa-domain 0 on pci3
bnxt1: Using 256 TX descriptors and 256 RX descriptors
bnxt1: Using 16 RX queues 16 TX queues
bnxt1: Using MSI-X interrupts with 17 vectors
bnxt1: Ethernet address: 9c:6b:00:46:a2:0d
Here is the firmware information:
dev.bnxt.0.nvram.available_size: 4173824
dev.bnxt.0.nvram.reserved_size: 16384
dev.bnxt.0.nvram.size: 8388608
dev.bnxt.0.nvram.sector_size: 4096
dev.bnxt.0.nvram.device_id: 16407
dev.bnxt.0.nvram.mfg_id: 239
dev.bnxt.0.ver.hwrm_min_ver: 1.10.2
dev.bnxt.0.ver.package_ver: <unknown>
dev.bnxt.0.ver.chip_type: ASIC
dev.bnxt.0.ver.chip_bond_id: 0
dev.bnxt.0.ver.chip_metal: 1
dev.bnxt.0.ver.chip_rev: 1
dev.bnxt.0.ver.chip_num: 5847
dev.bnxt.0.ver.phy_partnumber: MCP7F00-A002
dev.bnxt.0.ver.phy_vendor: Mellanox
dev.bnxt.0.ver.roce_fw_name: BONO_FW
dev.bnxt.0.ver.netctrl_fw_name: KONG_FW
dev.bnxt.0.ver.mgmt_fw_name: AFW_226.0.145.0
dev.bnxt.0.ver.hwrm_fw_name: CHIMP_FW
dev.bnxt.0.ver.phy: 13.1.11
dev.bnxt.0.ver.fw_ver: 226.0.145.0/pkg N/A
dev.bnxt.0.ver.roce_fw: 226.0.145
dev.bnxt.0.ver.netctrl_fw: 226.0.145
dev.bnxt.0.ver.mgmt_fw: 226.0.145
dev.bnxt.0.ver.hwrm_fw: 226.0.145
dev.bnxt.0.ver.driver_hwrm_if: 1.10.2.34
dev.bnxt.0.ver.hwrm_if: 1.10.2
dev.bnxt.0.%domain: 0
dev.bnxt.0.%parent: pci3
dev.bnxt.0.%pnpinfo: vendor=0x14e4 device=0x16d7 subvendor=0x1849
subdevice=0x1402 class=0x020000
dev.bnxt.0.%location: slot=0 function=0 dbsf=pci0:195:0:0
dev.bnxt.0.%driver: bnxt
dev.bnxt.0.%desc: Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet
I have no VLANs configured. I'm running stable/13 from mid-2023, but I've
tried the driver from the latest FreeBSD/head and FreeBSD stable/13 with no
success. I still get:
bnxt0: HWRM_RING_ALLOC command returned RESOURCE_ALLOC_ERROR error.
bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error.
bnxt0: set_multi: rx_mask set failed
bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error.
bnxt0: set_multi: rx_mask set failed
bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error.
bnxt0: set_multi: rx_mask set failed
bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error.
bnxt0: set_multi: rx_mask set failed
bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error.
bnxt0: set_multi: rx_mask set failed
bnxt0: Link is UP full duplex, FC - none - 25000 Mbps
bnxt0: link state changed to UP
bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error.
bnxt0: set_multi: rx_mask set failed
bnxt1: Link is UP full duplex, FC - none - 25000 Mbps
bnxt1: link state changed to UP
bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error.
bnxt0: set_multi: rx_mask set failed
bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error.
bnxt0: set_multi: rx_mask set failed
bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error.
bnxt0: set_multi: rx_mask set failed
Strangely enough, though, the driver works fine if I first PXE boot the machine
off of the chip. If I do that, it works normally. But if I boot off disk, I
get the RESOURCE_ALLOC_ERROR messages above.
This suggests that there is some kind of initialization issue that the PXE boot
environment takes care of, but the driver does not.
Also, bnxtnvm and niccli don't work with the driver in my kernel. But it
isn't, apparently, because of the state of the driver, it is because of the
ioctl definition in the driver. The ioctl call doesn't even get to
bnxt_mgmt_ioctl. I've verified that with dtrace, but here is the ioctl call
from ktrace/kdump:
7818 bnxtnvm CALL openat(AT_FDCWD,0x46199d,0x2<O_RDWR>)
7818 bnxtnvm NAMI "/dev/bnxt_mgmt"
7818 bnxtnvm RET openat 4
7818 bnxtnvm CALL ioctl(0x4,0x80000000,0x821199a70)
7818 bnxtnvm RET ioctl -1 errno 25 Inappropriate ioctl for device
7818 bnxtnvm CALL close(0x4)
Using this Dtrace script:
#pragma D option cleanrate=5000hz
#pragma D option dynvarsize=8192000
fbt::bnxt_mgmt_ioctl:entry
{
printf("cmd = %#x\n", args[1]);
}
fbt::bnxt_mgmt_open:entry
{
printf("opened bnxt mgmt device\n");
}
fbt::sys_ioctl:entry
/args[1]->com == 0x80000000/
{
printf("got ioctl command 0x80000000\n");
}
I verified that it isn't getting down to bnxt_mgmt_ioctl:
# dtrace -s bnxt.d
dtrace: script 'bnxt.d' matched 3 probes
CPU ID FUNCTION:NAME
31 2108 bnxt_mgmt_open:entry opened bnxt mgmt device
31 22882 sys_ioctl:entry got ioctl command 0x80000000
31 2108 bnxt_mgmt_open:entry opened bnxt mgmt device
31 22882 sys_ioctl:entry got ioctl command 0x80000000
^C
When I boot the machine via PXE, though, bnxtnvm listdev shows the device:
# ./bnxtnvm listdev
N/A #1
Device Interface Name : bnxt0
MACAddress : 9c:6b:00:46:a2:0c
PCI Device Name : 0000:c3:00.0
And strangely the ioctl works, although from my reading of sys_ioctl(), it
shouldn't but I think I've discovered why it does:
1904 bnxtnvm CALL openat(AT_FDCWD,0x46199d,0x2<O_RDWR>)
1904 bnxtnvm NAMI "/dev/bnxt_mgmt"
1904 bnxtnvm RET openat 3
1904 bnxtnvm CALL ioctl(0x3,0x80000000,0x8212303d0)
1904 bnxtnvm RET ioctl 0
1904 bnxtnvm CALL close(0x3)
This code is from sys_ioctl():
/*
* Interpret high order word to find amount of data to be
* copied to/from the user's address space.
*/
size = IOCPARM_LEN(com);
if ((size > IOCPARM_MAX) ||
((com & (IOC_VOID | IOC_IN | IOC_OUT)) == 0) ||
#if defined(COMPAT_FREEBSD5) || defined(COMPAT_FREEBSD4) || defined(COMPAT_43)
((com & IOC_OUT) && size == 0) ||
#else
((com & (IOC_IN | IOC_OUT)) && size == 0) ||
#endif
((com & IOC_VOID) && size > 0 && size != sizeof(int)))
return (ENOTTY);
My regular kernel config file doesn't have COMPAT_FREEBSD4/5, but the PXE
kernel config file does.
Here are the bit definitions from sys/ioccom.h:
#ifndef _SYS_IOCCOM_H_
#define _SYS_IOCCOM_H_
/*
* Ioctl's have the command encoded in the lower word, and the size of
* any in or out parameters in the upper word. The high 3 bits of the
* upper word are used to encode the in/out status of the parameter.
*
* 31 29 28 16 15 8 7 0
* +---------------------------------------------------------------+
* | I/O | Parameter Length | Command Group | Command |
* +---------------------------------------------------------------+
*/
#define IOCPARM_SHIFT 13 /* number of bits for ioctl size */
#define IOCPARM_MASK ((1 << IOCPARM_SHIFT) - 1) /* parameter length mask */
#define IOCPARM_LEN(x) (((x) >> 16) & IOCPARM_MASK)
#define IOCBASECMD(x) ((x) & ~(IOCPARM_MASK << 16))
#define IOCGROUP(x) (((x) >> 8) & 0xff)
#define IOCPARM_MAX (1 << IOCPARM_SHIFT) /* max size of ioctl */
#define IOC_VOID 0x20000000UL /* no parameters */
#define IOC_OUT 0x40000000UL /* copy out parameters */
#define IOC_IN 0x80000000UL /* copy in parameters */
#define IOC_INOUT (IOC_IN|IOC_OUT)/* copy parameters in and out */
#define IOC_DIRMASK (IOC_VOID|IOC_OUT|IOC_IN)/* mask for IN/OUT/VOID */
Because the BNXT_MGMT_OPCODE_GET_DEV_INFO ioctl the same as the IOC_IN bit
definition, the ioctl breaks if the old compat stuff isn't built into the
kernel.
The ioctls for the bnxt(4) driver need to be changed to use the usual
_IOW/_IOWR macros from sys/ioccom.h. I realize that will break the management
tools. Perhaps they can have a version check and a fallback to the old ioctls
if need be.
--
You are receiving this mail because:
You are the assignee for the bug.