Re: Help me grok the ath(4) device attach code

> On Jun 1, 2023, at 11:28 PM, John Nielsen <lists@jnielsen.net> wrote:
> 
>> On May 31, 2023, at 1:52 PM, John Nielsen <lists@jnielsen.net> wrote:
>> 
>>> On May 30, 2023, at 11:17 PM, Adrian Chadd <adrian@freebsd.org> wrote:
>>> 
>>> On Tue, 30 May 2023 at 22:12, John Nielsen <lists@jnielsen.net> wrote:
>>>> On May 30, 2023, at 10:56 PM, Adrian Chadd <adrian@freebsd.org> wrote:
>>>> 
>>>> On Tue, 30 May 2023 at 20:56, John Nielsen <lists@jnielsen.net> wrote:
>>>> > On May 30, 2023, at 8:02 PM, Adrian Chadd <adrian@freebsd.org> wrote:
>>>> > 
>>>> > Err, if it's coming up w/ that MAC then it's not finding and attaching right to the OTP/EEPROM calibration information. That's the big red flag that it in general won't work correctly.
>>>> > 
>>>> > Can you provide the rest of the ath_hal messages? I'd like to see what it's saying during boot around it checking the EEPROM/OTP contents. It's possible there's some work around required for this NIC.
>>>> 
>>>> He speaks! Thanks for taking the time. I just realized that ath_hal_printf doesn’t prepend “ath%d” so I’ve been missing those messages when grep-ing. Here’s the whole snippet:
>>>> 
>>>> ath0: <Atheros AR946x/AR948x> mem 0xf7a00000-0xf7a7ffff at device 0.0 on pci4
>>>> ar9300_flash_map: unimplemented for now
>>>> Restoring Cal data from DRAM
>>>> Restoring Cal data from EEPROM
>>>> Restoring Cal data from Flash
>>>> Restoring Cal data from Flash
>>>> Restoring Cal data from OTP
>>>> ar9300_eeprom_restore_internal[4338] No vaid CAL, calling default template
>>>> ar9300_hw_attach: ar9300_eeprom_attach returned 0
>>>> 
>>>> Yeah, this bit right here is the problem. It's not finding a valid calibration.
>>> 
>>> 
>>>> oh err, is there a wifi enable/disable switch or something? maybe it's asserted and somehow it's mucking up the NIC?
>>> 
>>> There is a physical switch and it’s in the “enable” position.
>>> 
>>>>  I wonder what ath9k is doing here? Is there some weird pci based workaround/flag for the given NIC PCI id?
>>> 
>>> That was the first breadcrumb BZ threw me but I can’t find anything. There are some .driver_data hints for adjacent subdevice IDs but none for this one (Dell 0x020d) in either FreeBSD or Linux that I could find.
>>> 
>>> [snip]
>>> There is this commit in ath9k which mentions an alternative EEPROM address, but I’m not sure if that’s relevant. From what I can tell the probe should succeed at the normal base_address 0x3ff instead of needing to try the “4k” one 0xfff.
>>> https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/commit/drivers/net/wireless/ath/ath9k?id=528782ecf59f7bab2f1368628a479f49be59b512
>>> 
>>> Yeah i'd try that. It'd be nice if I knew that the NIC used OTP or EEPROM though.
>>> 
>>> There's known issues with all the Atheros chips (sigh) with how the EEPROM and PCIe bus reset .. interact.
>>> (If the bus reset is too short then the EEPROM state machine gets stuck and nothing gets read.) It makes debugging this hard because the NIC itself will work in another device fine, because it's the BIOS/ACPI code. :(
>> 
>> 
>> The 4k EEPROM read didn’t work. But I did notice that where it says it’s restoring Cal data from OTP it actually just does an EEPROM read again. Shouldn’t this line be a call to ar9300_otp_read() instead of ar9300_eeprom_restore_internal_address()?
>> 
>> https://cgit.freebsd.org/src/tree/sys/contrib/dev/ath/ath_hal/ar9300/ar9300_eeprom.c#n4292
>> 
>> Otherwise it doesn’t use the 0x14000 (0x30000 for some cards) OTP offset as a starting point.
> 
> 
> I’d love to know if I’m up in the night about my theory above about the OTP call not actually happening. In the meantime I’ll carry on either trying to find/build a Linux kernel with CONFIG_ATH_DEBUG enabled that I can boot on the laptop (to see which EEPROM retrieval/check actually succeeds) and/or try to reimplement bits from the ath9k ar9300_eeprom_restore_internal() to match its behavior on FreeBSD.
> ( https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/tree/drivers/net/wireless/ath/ath9k/ar9003_eeprom.c#n3266 )

After learning more than I planned to about archiso and building kernel packages for Arch I managed to boot the laptop on a Linux kernel with CONFIG_ATH_DEBUG enabled and set as ath9k.debug=0x4 to get the EEPROM debug output. And.. it looks a lot more normal than I would have guessed. The EEPROM is located, identified and loaded from the default 0x3ff address (first try!) and everything just goes smoothly. Full output below.

That at least tells me where to focus in the FreeBSD code but as always I’d love some input or advice from Adrian or anyone who’s looked at this more than I have.

Thanks!

-JN

=== dmesg|grep ath output ===

[    0.039398] Kernel command line: BOOT_IMAGE=/arch/boot/x86_64/vmlinuz-linux-jn archisobasedir=arch archisodevice=UUID=2023-06-08-21-40-20-00 ath9k.debug=0x4
[   17.692752] ath9k 0000:04:00.0: enabling device (0000 -> 0002)
[   17.696711] ath: phy0: Trying EEPROM access at Address 0x03ff
[   17.697238] ath: phy0: Found valid EEPROM data
[   17.697763] ath: phy0: Found block at 3ff: code=3 ref=4 length=794 major=4 minor=6
[   17.803208] ath: phy0: checksum a5cc a5cc
[   17.803214] ath: phy0: restore eeprom 0: block, reference 4, length 794
[   17.803217] ath: phy0: Restore at 0: spot=2 offset=2 length=22
[   17.803219] ath: phy0: Restore at 24: spot=28 offset=4 length=1
[   17.803222] ath: phy0: Restore at 27: spot=35 offset=6 length=1
[   17.803224] ath: phy0: Restore at 30: spot=43 offset=7 length=1
[   17.803226] ath: phy0: Restore at 33: spot=48 offset=4 length=31
[   17.803228] ath: phy0: Restore at 66: spot=106 offset=27 length=10
[   17.803230] ath: phy0: Restore at 78: spot=128 offset=12 length=8
[   17.803232] ath: phy0: Restore at 88: spot=141 offset=5 length=3
[   17.803234] ath: phy0: Restore at 93: spot=147 offset=3 length=3
[   17.803236] ath: phy0: Restore at 98: spot=153 offset=3 length=3
[   17.803238] ath: phy0: Restore at 103: spot=159 offset=3 length=3
[   17.803240] ath: phy0: Restore at 108: spot=165 offset=3 length=3
[   17.803241] ath: phy0: Restore at 113: spot=171 offset=3 length=3
[   17.803243] ath: phy0: Restore at 118: spot=205 offset=31 length=31
[   17.803245] ath: phy0: Restore at 151: spot=240 offset=4 length=10
[   17.803247] ath: phy0: Restore at 163: spot=254 offset=4 length=10
[   17.803249] ath: phy0: Restore at 175: spot=268 offset=4 length=10
[   17.803251] ath: phy0: Restore at 187: spot=282 offset=4 length=10
[   17.803253] ath: phy0: Restore at 199: spot=296 offset=4 length=10
[   17.803255] ath: phy0: Restore at 211: spot=328 offset=22 length=9
[   17.803257] ath: phy0: Restore at 222: spot=344 offset=7 length=9
[   17.803259] ath: phy0: Restore at 233: spot=356 offset=3 length=79
[   17.803261] ath: phy0: Restore at 314: spot=438 offset=3 length=5
[   17.803263] ath: phy0: Restore at 321: spot=479 offset=36 length=2
[   17.803265] ath: phy0: Restore at 325: spot=489 offset=8 length=25
[   17.803267] ath: phy0: Restore at 352: spot=517 offset=3 length=3
[   17.803269] ath: phy0: Restore at 357: spot=523 offset=3 length=3
[   17.803271] ath: phy0: Restore at 362: spot=529 offset=3 length=3
[   17.803273] ath: phy0: Restore at 367: spot=535 offset=3 length=3
[   17.803275] ath: phy0: Restore at 372: spot=541 offset=3 length=3
[   17.803277] ath: phy0: Restore at 377: spot=547 offset=3 length=3
[   17.803279] ath: phy0: Restore at 382: spot=553 offset=3 length=3
[   17.803280] ath: phy0: Restore at 387: spot=559 offset=3 length=3
[   17.803282] ath: phy0: Restore at 392: spot=565 offset=3 length=3
[   17.803284] ath: phy0: Restore at 397: spot=571 offset=3 length=3
[   17.803286] ath: phy0: Restore at 402: spot=577 offset=3 length=3
[   17.803288] ath: phy0: Restore at 407: spot=583 offset=3 length=3
[   17.803290] ath: phy0: Restore at 412: spot=589 offset=3 length=3
[   17.803292] ath: phy0: Restore at 417: spot=595 offset=3 length=3
[   17.803294] ath: phy0: Restore at 422: spot=601 offset=3 length=3
[   17.803296] ath: phy0: Restore at 427: spot=678 offset=74 length=43
[   17.803298] ath: phy0: Restore at 472: spot=725 offset=4 length=10
[   17.803300] ath: phy0: Restore at 484: spot=739 offset=4 length=10
[   17.803302] ath: phy0: Restore at 496: spot=753 offset=4 length=10
[   17.803304] ath: phy0: Restore at 508: spot=767 offset=4 length=10
[   17.803306] ath: phy0: Restore at 520: spot=781 offset=4 length=10
[   17.803308] ath: phy0: Restore at 532: spot=795 offset=4 length=10
[   17.803309] ath: phy0: Restore at 544: spot=809 offset=4 length=10
[   17.803311] ath: phy0: Restore at 556: spot=823 offset=4 length=10
[   17.803313] ath: phy0: Restore at 568: spot=837 offset=4 length=10
[   17.803315] ath: phy0: Restore at 580: spot=851 offset=4 length=10
[   17.803317] ath: phy0: Restore at 592: spot=865 offset=4 length=10
[   17.803319] ath: phy0: Restore at 604: spot=879 offset=4 length=10
[   17.803321] ath: phy0: Restore at 616: spot=893 offset=4 length=10
[   17.803323] ath: phy0: Restore at 628: spot=907 offset=4 length=10
[   17.803325] ath: phy0: Restore at 640: spot=921 offset=4 length=10
[   17.803327] ath: phy0: Restore at 652: spot=945 offset=14 length=15
[   17.803329] ath: phy0: Restore at 669: spot=963 offset=3 length=5
[   17.803331] ath: phy0: Restore at 676: spot=971 offset=3 length=37
[   17.803333] ath: phy0: Restore at 715: spot=1011 offset=3 length=77
[   17.805147] ath: EEPROM regdomain: 0x6c
[   17.805150] ath: EEPROM indicates we should expect a direct regpair map
[   17.805152] ath: Country alpha2 being used: 00
[   17.805154] ath: Regpair used: 0x6c
[several more pages of calibration correction data omitted]