[Bug 273263] isp(4): 32Gbit Qlogic cards do not work

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273263

--- Comment #5 from Joerg Pulz <Joerg.Pulz@frm2.tum.de> ---
I kind of solved this or at least found a solution how to work around this
issue.
Also I know now why this all happened at all.

Some background:
The 27xx and 28xx based controllers feature two firmware images in flash, a
primary and a secondary one.
While the 27xx based controllers still share some amount of information between
both firmware images (e.g. NVRAM, NVMe parameters, ...) the 28xx based
controllers even have separate areas for those.
In penguin land the driver knows and handles this correctly as firmware loading
there is somewhat different compared to ours.
We just send a MBOX command the get the controller do some magic stuff to load
the firmware from flash without ever thinking about primary/secondary firmware
images. This is working quite well if either
  - primary/secondary firmware versions in flash are equal
or 
  - the primary firmware image is set as active.
The penguin driver is very different in this area as it checks for valid
primary/secondary firmware images and detects which one is the active one.
Firmware loading is done by reading the active firmware directly from flash and
writing it into the controllers RAM. Depending on the active firmware the
correct NVRAM (and other) areas are used too.

With all this in mind, back to the original PR:

Arne Steinkamm kindly provided two controllers out of a random system to me.
The first one had RISC firmware 8.8.231 in the primary and no firmware at all
in the secondary flash area (verified with the vendors firmware flash utility).
This controller was detected and initialized without problems.
The second controller was shown with firmware 9.8.2 in UEFI but our driver
reported 8.8.231 and initialization failed - as reported here.
After looking at this controller with the vendors firmware flash utility (tried
the EFI and the Linux one) it turns out that there was firmware 8.8.231 in the
primary and 9.8.2 in the secondary area. The secondary area was marked active.
Both Lenovo branded controllers where sold in this state, so it seems one can't
trust that newly sold controllers are working out of the box.
I got the most recent firmware update package from Marvell/QLogic only to find
out that the Lenovo branded controllers can't be flashed with this one.
Luckily I was able to find a Lenovo firmware update package for Windows that
contained the firmware as a separate file usable with the vendors EFI and Linux
flash utility. This was exactly the RISC firmware version 9.8.2 that already
was in the second controllers secondary firmware area.
After flashing the card, it showed RISC firmware version 9.8.2 in the primary
area and the primary area activated.
Starting FreeBSD the driver reported firmware 9.8.2 and initialization was
successful.
Only for testing I flashed the firmware again what resulted in both firmware
regions containing the same version but the secondary area active.
In this state the driver again reported firmware 9.8.2 and initialization was
successful.
For verification I took the first controller with no secondary firmware at all
and flashed this one resulting in primary firmware 8.8.231 secondary firmware
9.8.2 and secondary firmware active.
In this state I instantly got the same initialization error like with the
second card before and reported here.
After flashing this controller a second time (primary and secondary at version
9.8.2 with primary active) everything was fine and working again.
As a side note: I found no way to change the active firmware image without
flashing (neither the EFI nor the Linux flash utility provides an option for
this). This puts the whole concept of two firmware images in question.

I provided all this with the required firmware and flash utility files with
some documentation to Arne Steinkamm. With those he was able to get another
machine with previously failing controllers to work correctly.

Final advise for isp(4) with 27xx and 28xx based controllers:
To work around the shortcomings of handling primary/secondary firmware images
correctly, one has to make sure that either:
1. the primary firmware is active or
2. the primary and secondary firmware contain the same version, regardless
which one is active.
This alone makes the controllers work but the second option is problematic with
28xx based ones where NVRAM and so on is specific to the activated firmware.

So to be on the really safe side, either use option one or make sure primary
and secondary firmware are the same and primary is active for the time being.

One can use either the Marvell/QLogic EFI (preferred) or Linux utility to
verify or achieve this. If firmware flashing is required, one could try to use
the vendors native firmware or has to use specific firmware from his hardware
vendor (e.g. Lenovo, Dell, ...).

This whole situation should be documented in the BUGS section of the isp(4)
manpage for all supported FreeBSD versions (at least CURRENT, 14-STABLE and
13-STABLE) that support at least 27xx based controllers.

-- 
You are receiving this mail because:
You are the assignee for the bug.