RE: Soliciting infiniband ifconfig output
- Reply: Mina_Galić : "Re: Soliciting infiniband ifconfig output"
- In reply to: Chuck Tuffli : "Re: Soliciting infiniband ifconfig output"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Thu, 09 Feb 2023 09:44:33 UTC
Hi!
Here’s an example of output using Intel E810 family Ethernet Controller
#ifconfig -a
ice0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether b4:96:91:GG:GG:GG
inet 100.0.0.84 netmask 0xffffff00 broadcast 100.0.0.255
inet6 1234::84 prefixlen 64
inet6 fe80::b696:91ff:feGG:GGd8%ice0 prefixlen 64 scopeid 0x4
media: Ethernet autoselect (25GBase-CR <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ice1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether b4:96:91:GG:GG:GG
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ice2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether b4:96:91:GG:GG:GG
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ice3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether b4:96:91:GG:GG:GG
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
# pciconf -lva pci0:177:0:0
ice0@pci0:177:0:0: class=0x020000 card=0x00058086 chip=0x15938086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Controller E810-C for SFP'
class = network
subclass = ethernet
# ibv_devinfo -d irdma-ice0 -v
hca_id: irdma-ice0
transport: InfiniBand (0)
fw_ver: 1.69
node_guid: b696:91ff:feGG:GGGG
sys_image_guid: b696:91ff:feGG:GGGG
vendor_id: 0x8086
vendor_part_id: 5523
hw_ver: 0x2
phys_port_cnt: 1
max_mr_size: 0x200000000000
page_size_cap: 0x40201000
max_qp: 16380
max_qp_wr: 4063
device_cap_flags: 0x01229000
RC_RNR_NAK_GEN
MEM_WINDOW
MEM_MGT_EXTENSIONS
MEM_WINDOW_TYPE_2B
Unknown flags: 0x8000
max_sge: 13
max_sge_rd: 13
max_cq: 32765
max_cqe: 1048574
max_mr: 2097150
max_pd: 262141
max_qp_rd_atom: 32
max_ee_rd_atom: 0
max_res_rd_atom: 0
max_qp_init_rd_atom: 32
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_NONE (0)
max_ee: 0
max_rdd: 0
max_mw: 2097150
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 16384
max_mcast_qp_attach: 8
max_total_mcast_qp_attach: 131072
max_ah: 131072
max_fmr: 0
max_srq: 0
max_pkeys: 1
local_ca_ack_delay: 0
general_odp_caps:
rc_odp_caps:
NO SUPPORT
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
NO SUPPORT
completion_timestamp_mask not supported
core clock not supported
device_cap_flags_ex: 0x0
tso_caps:
max_tso: 0
rss_caps:
max_rwq_indirection_tables: 0
max_rwq_indirection_table_size: 0
rx_hash_function: 0x0
rx_hash_fields_mask: 0x0
max_wq_type_rq: 0
packet_pacing_caps:
qp_rate_limit_min: 0kbps
qp_rate_limit_max: 0kbps
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 1
port_lmc: 0x00
link_layer: Ethernet
max_msg_sz: 0x10001
port_cap_flags: 0x04050000
max_vl_num: invalid value (0)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 1
gid_tbl_len: 32
subnet_timeout: 0
init_type_reply: 0
active_width: 4X (2)
active_speed: 25.0 Gbps (32)
phys_state: LINK_UP (5)
GID[ 0]: fe80:0000:0000:0000:b696:91ff:feGG:GGGG
GID[ 1]: 0000:0000:0000:0000:0000:ffff:6400:0054
GID[ 2]: 1234:0000:0000:0000:0000:0000:0000:0084
# ibv_devinfo -d irdma-ice1 -v
hca_id: irdma-ice1
transport: iWARP (1)
fw_ver: 1.69
node_guid: b696:91ff:feGG:GGGG
sys_image_guid: b696:91ff:feGG:GGGG
vendor_id: 0x8086
vendor_part_id: 5523
hw_ver: 0x2
phys_port_cnt: 1
max_mr_size: 0x200000000000
page_size_cap: 0x40201000
max_qp: 16380
max_qp_wr: 4063
device_cap_flags: 0x01228000
MEM_WINDOW
MEM_MGT_EXTENSIONS
MEM_WINDOW_TYPE_2B
Unknown flags: 0x8000
max_sge: 13
max_sge_rd: 13
max_cq: 32765
max_cqe: 1048574
max_mr: 2097150
max_pd: 262141
max_qp_rd_atom: 32
max_ee_rd_atom: 0
max_res_rd_atom: 0
max_qp_init_rd_atom: 32
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_NONE (0)
max_ee: 0
max_rdd: 0
max_mw: 2097150
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 0
max_mcast_qp_attach: 0
max_total_mcast_qp_attach: 0
max_ah: 0
max_fmr: 0
max_srq: 0
max_pkeys: 0
local_ca_ack_delay: 0
general_odp_caps:
rc_odp_caps:
NO SUPPORT
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
NO SUPPORT
completion_timestamp_mask not supported
core clock not supported
device_cap_flags_ex: 0x0
tso_caps:
max_tso: 0
rss_caps:
max_rwq_indirection_tables: 0
max_rwq_indirection_table_size: 0
rx_hash_function: 0x0
rx_hash_fields_mask: 0x0
max_wq_type_rq: 0
packet_pacing_caps:
qp_rate_limit_min: 0kbps
qp_rate_limit_max: 0kbps
port: 1
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 1
port_lmc: 0x00
link_layer: Ethernet
max_msg_sz: 0x10001
port_cap_flags: 0x00050000
max_vl_num: invalid value (0)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 0
gid_tbl_len: 1
subnet_timeout: 0
init_type_reply: 0
active_width: 1X (1)
active_speed: 2.5 Gbps (1)
Hope this helps.
Bartek
-----Original Message-----
From: owner-freebsd-infiniband@freebsd.org <owner-freebsd-infiniband@freebsd.org> On Behalf Of Chuck Tuffli
Sent: Wednesday, February 8, 2023 20:35
To: Mina Galić <freebsd@igalic.co>
Cc: freebsd-infiniband@freebsd.org
Subject: Re: Soliciting infiniband ifconfig output
On Thu, Feb 2, 2023 at 2:10 AM Mina Galić <freebsd@igalic.co> wrote:
>
> Hi folks,
>
> I'm currently working on a project of making FreeBSD a Tier 1 cloud-init platform[1].
>
> That involves a lot of refactoring of very Linux specific code[2], and in order to be able to do that, I need information about hardware that I don't have access to.
>
> I'm still operating under the assumption that i can get (almost) all information from a network device via ifconfig[3].
>
> Can you share an `ifconfig -a` output with me on your FreeBSD systems that have Infiniband? And maybe any other commands that are more useful, if my above assumption is wrong.
I don't use IB but have hardware that is capable. Below is what I think you wanted. If others can provide info on how to configure this, I'm happy to do so.
# freebsd-version
13.1-RELEASE
# pciconf -lva pci0:55:0:0
mlx5_core0@pci0:55:0:0: class=0x020700 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x101b subvendor=0x15b3 subdevice=0x0008
vendor = 'Mellanox Technologies'
device = 'MT28908 Family [ConnectX-6]'
class = network
subclass = InfiniBand
# pciconf -lva pci0:55:0:1
mlx5_core1@pci0:55:0:1: class=0x020700 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x101b subvendor=0x15b3 subdevice=0x0008
vendor = 'Mellanox Technologies'
device = 'MT28908 Family [ConnectX-6]'
class = network
subclass = InfiniBand
# ifconfig -a
bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 08:f1:ea:98:62:80
inet 169.254.238.43 netmask 0xfffff800 broadcast 169.254.239.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 08:f1:ea:98:62:81
media: Ethernet autoselect
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bge2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 08:f1:ea:98:62:82
media: Ethernet autoselect
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bge3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 08:f1:ea:98:62:83
media: Ethernet autoselect
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib0: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 4092
options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
lladdr 0.0.1.5.fe.80.0.0.0.0.0.0.b8.59.9f.3.0.ec.56.cc
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ib1: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 4092
options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
lladdr 0.0.9.5.fe.80.0.0.0.0.0.0.b8.59.9f.3.0.ec.56.cd
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
---------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Spolka oswiadcza, ze posiada status duzego przedsiebiorcy w rozumieniu ustawy z dnia 8 marca 2013 r. o przeciwdzialaniu nadmiernym opoznieniom w transakcjach handlowych.
Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.