interrupt storm arge0, tplink 1043nd
Harm Weites
harm at weites.com
Sat Jul 26 08:00:16 UTC 2014
Hi Adrian,
Thanks for your pointers, I've attached a patch to enable counting the
interrupts. Also attached is a log with the interrupt counters in there
pristine state, just after boot and another from right after the storm.
Though It looks like none of the counters got incremented, so I most
probably wrote bad code.
Another thing I noticed, every interrupt (/group of 32) got printed 4
times. Printf also only sent the result to my serial console, even when
I ran /sbin/sysctl using ssh so that still deserves a little attention.
About vmstat, yes, it regards arge0:
# vmstat -ia
interrupt total rate
sint0: 0 0
sint1: 0 0
int0 ath0 2938765 80
int1: 0 0
int2 arge0 335493 9
int3 arge1 0 0
int4 apb0 1936 0
int5 clock0 1129353 30
apb intr3: uart0 1936 0
apb intr2: gpio0 0 0
apb irq5: pmc 0 0
Total 4407483 120
Regards,
Harm
op 22-07-14 21:26, Adrian Chadd schreef:
> Hi!
>
> So, ignore the ath0 stuff for now. int2 should be arge0, right?
>
> what's vmstat -ia say?
>
> Assuming it's actually arge0, we need to add some debugging counters
> to the interrupt path to count how many of each interrupt are
> occuring. The stuff i stuck behind ARGEDEBUG() is useful for debugging
> some silly bugs but not at the rate that you're getting interrupts.
>
> So I'd add something like this to the arge softc struct:
>
> uint32_t intr_status[32];
>
> .. then in the interrupt routine, something like this:
>
> temp_status = status;
> for (i = 0; i < 32; i++) {
> if (temp_status & 1) {
> intr_status[i]++;
> }
> temp_status = temp_status >> 1;
> }
>
> That'll count the number of interrupts that are firing for each
> interrupt status bit.
>
> Then, you'll want to write a sysctl for it. Have a look at
> if_ath_sysctl.c for the SYSCTL_PROC() entries. Just write one that
> when called will just printf() the intr_status array:
>
> for (i = 0; i < 32; i++) {
> printf("interrupt %d count %u\n", i, intr_status[i]);
> }
>
> Just make sure you do a complete kernel recompile as changing the
> headers doesn't always force the source files to recompile.
>
>
> -a
>
>
> On 22 July 2014 12:08, Harm Weites <harm at weites.com> wrote:
>> Hi,
>>
>> My 1043nd is complaining about interrupt storms, presumably only when
>> wifi is beeing used. When this occurs, networking is gone.
>>
>> The exact message thats flooding me:
>> interrupt storm detected on "int2"; throttling interrupt source
>>
>> The device associated with int2 is arge0.
>>
>> Some possibly related logs, though these messages start at boot:
>>
>> # /sbin/dmesg | tail
>> ath0: stuck beacon; resetting (bmiss count 4)
>> ar5416StopDmaReceive: dma failed to stop in 10ms
>> AR_CR=0x00000024
>> AR_DIAG_SW=0x42000020
>> MBSSID Set bit 22 of AR_STA_ID 0xb8c16866
>> ath0: stuck beacon; resetting (bmiss count 4)
>> ar5416StopDmaReceive: dma failed to stop in 10ms
>> AR_CR=0x00000024
>> AR_DIAG_SW=0x42000020
>> MBSSID Set bit 22 of AR_STA_ID 0xb8c16866
>>
>> This unit is configured with (arge0) port0 bound to device vlan1, port4
>> to vlan2 and ports 1,2,3 make up vlan3. There is wlan0, bound to ath0
>> and a bridge device that connects wlan0 to vlan3. There is a dhcp server
>> running in vlan3 to answer to wifi clients, internet is routed through
>> vlan1. This initially works but after a little while the storm begins
>> and the wifi client is left to die.
>>
>> Adrian@ suggested to start with reading which interrupt(s) occur(s), but
>> that is perhaps a little to hard for me to code :) Looking at if_arge.c,
>> it seems there is some debug code already in place (ARGEDEBUG()) though
>> I'm not sure on how to use that. Reading from the AR71XX_DMA_INTR
>> register and mapping its content to AR71XX_DMA_INTR_STATUS would be
>> something I'd like to do with a separate program (instead of boldly
>> taking a deepdive in to if_arge.c and recompiling/flashing untill
>> something works).
>>
>> One of my other units is configured with just a vlan device per switch
>> port, no wifi and no bridge. A third unit is configured with a wlan0,
>> vlan1 (port0) and vlan2 (ports 1,2,3,4). Both not showing any issues in
>> the past months. The only difference would be this problem-unit has a
>> bridge.
>>
>> Any thoughts on how to approach or 'just' fix this?
>>
>> Regards,
>> Harm
>> _______________________________________________
>> freebsd-mips at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-mips
>> To unsubscribe, send any mail to "freebsd-mips-unsubscribe at freebsd.org"
-------------- next part --------------
# sysctl -a|grep interrupt
interrupt 0 count 2154400768
interrupt 1 count 2153943040
interrupt 2 count 0
interrupt 3 count 0
interrupt 4 count 2153893728
interrupt 5 count 2153893728
interrupt 6 count 2151296140
interrupt 7 count 2151296104
interrupt 8 count 16
interrupt 9 count 1048576
interrupt 10 count 0
interrupt 11 count 2432062003
interrupt 12 count 1366556672
interrupt 13 count 2154442496
interrupt 14 count 0
interrupt 15 count 2154442368
interrupt 16 count 2154441280
interrupt 17 count 0
interrupt 18 count 0
interrupt 19 count 0
interrupt 20 count 0
interrupt 21 count 0
interrupt 22 count 0
interrupt 23 count 0
interrupt 24 count 0
interrupt 25 count 2153896656
interrupt 26 count 16973824
interrupt 27 count 0
interrupt 28 count 0
interrupt 29 count 4
interrupt 30 count 0
interrupt 31 count 0
interrupt 0 count 2154400768
interrupt 1 count 2153943040
interrupt 2 count 0
interrupt 3 count 0
interrupt 4 count 2153893728
interrupt 5 count 2153893728
interrupt 6 count 2151296140
interrupt 7 count 2151296104
interrupt 8 count 16
interrupt 9 count 1048576
interrupt 10 count 0
interrupt 11 count 2432062003
interrupt 12 count 1366556672
interrupt 13 count 2154442496
interrupt 14 count 0
interrupt 15 count 2154442368
interrupt 16 count 2154441280
interrupt 17 count 0
interrupt 18 count 0
interrupt 19 count 0
interrupt 20 count 0
interrupt 21 count 0
interrupt 22 count 0
interrupt 23 count 0
interrupt 24 count 0
interrupt 25 count 2153896656
interrupt 26 count 16973824
interrupt 27 count 0
interrupt 28 count 0
interrupt 29 count 4
interrupt 30 count 0
interrupt 31 count 0
interrupt 0 count 2154424320
interrupt 1 count 2153942784
interrupt 2 count 0
interrupt 3 count 0
interrupt 4 count 2153893344
interrupt 5 count 2153893344
interrupt 6 count 2151296140
interrupt 7 count 2151296104
interrupt 8 count 6
interrupt 9 count 1048576
interrupt 10 count 0
interrupt 11 count 2432062003
interrupt 12 count 1366622208
interrupt 13 count 2154440320
interrupt 14 count 0
interrupt 15 count 2154442304
interrupt 16 count 2154439936
interrupt 17 count 0
interrupt 18 count 0
interrupt 19 count 0
interrupt 20 count 0
interrupt 21 count 0
interrupt 22 count 0
interrupt 23 count 0
interrupt 24 count 0
interrupt 25 count 2153896608
interrupt 26 count 16973824
interrupt 27 count 0
interrupt 28 count 0
interrupt 29 count 4
interrupt 30 count 0
interrupt 31 count 0
interrupt 0 count 2154424320
interrupt 1 count 2153942784
interrupt 2 count 0
interrupt 3 count 0
interrupt 4 count 2153893344
interrupt 5 count 2153893344
interrupt 6 count 2151296140
interrupt 7 count 2151296104
interrupt 8 count 6
interrupt 9 count 1048576
interrupt 10 count 0
interrupt 11 count 2432062003
interrupt 12 count 1366622208
interrupt 13 count 2154440320
interrupt 14 count 0
interrupt 15 count 2154442304
interrupt 16 count 2154439936
interrupt 17 count 0
interrupt 18 count 0
interrupt 19 count 0
interrupt 20 count 0
interrupt 21 count 0
interrupt 22 count 0
interrupt 23 count 0
interrupt 24 count 0
interrupt 25 count 2153896608
interrupt 26 count 16973824
interrupt 27 count 0
interrupt 28 count 0
interrupt 29 count 4
interrupt 30 count 0
interrupt 31 count 0
### after 9hrs, storm on int2 hits
interrupt storm detected on "int2"; throttling interrupt source
interrupt storm detected on "int2"; throttling interrupt source
interrupt storm detected on "int2"; throttling interrupt source
interrupt storm detected on "int2"; throttling interrupt source
# sysctl -a|grep interrupt
interrupt 0 count 2154400768
interrupt 1 count 2153943040
interrupt 2 count 0
interrupt 3 count 0
interrupt 4 count 2153893728
interrupt 5 count 2153893728
interrupt 6 count 2151296140
interrupt 7 count 2151296104
interrupt 8 count 16
interrupt 9 count 1048576
interrupt 10 count 0
interrupt 11 count 2432062003
interrupt 12 count 1366556672
interrupt 13 count 2154442496
interrupt 14 count 0
interrupt 15 count 2154442368
interrupt 16 count 2154441280
interrupt 17 count 0
interrupt 18 count 0
interrupt 19 count 0
interrupt 20 count 0
interrupt 21 count 0
interrupt 22 count 0
interrupt 23 count 0
interrupt 24 count 0
interrupt 25 count 2153896656
interrupt 26 count 16973824
interrupt 27 count 0
interrupt 28 count 0
interrupt 29 count 4
interrupt 30 count 0
interrupt 31 count 0
interrupt 0 count 2154400768
interrupt 1 count 2153943040
interrupt 2 count 0
interrupt 3 count 0
interrupt 4 count 2153893728
interrupt 5 count 2153893728
interrupt 6 count 2151296140
interrupt 7 count 2151296104
interrupt 8 count 16
interrupt 9 count 1048576
interrupt 10 count 0
interrupt 11 count 2432062003
interrupt 12 count 1366556672
interrupt 13 count 2154442496
interrupt 14 count 0
interrupt 15 count 2154442368
interrupt 16 count 2154441280
interrupt 17 count 0
interrupt 18 count 0
interrupt 19 count 0
interrupt 20 count 0
interrupt 21 count 0
interrupt 22 count 0
interrupt 23 count 0
interrupt 24 count 0
interrupt 25 count 2153896656
interrupt 26 count 16973824
interrupt 27 count 0
interrupt 28 count 0
interrupt 29 count 4
interrupt 30 count 0
interrupt 31 count 0
interrupt 0 count 2154424320
interrupt 1 count 2153942784
interrupt 2 count 0
interrupt 3 count 0
interrupt 4 count 2153893344
interrupt 5 count 2153893344
interrupt 6 count 2151296140
interrupt 7 count 2151296104
interrupt 8 count 6
interrupt 9 count 1048576
interrupt 10 count 0
interrupt 11 count 2432062003
interrupt 12 count 1366622208
interrupt 13 count 2154440320
interrupt 14 count 0
interrupt 15 count 2154442304
interrupt 16 count 2154439936
interrupt 17 count 0
interrupt 18 count 0
interrupt 19 count 0
interrupt 20 count 0
interrupt 21 count 0
interrupt 22 count 0
interrupt 23 count 0
interrupt 24 count 0
interrupt 25 count 2153896608
interrupt 26 count 16973824
interrupt 27 count 0
interrupt 28 count 0
interrupt 29 count 4
interrupt 30 count 0
interrupt 31 count 0
interrupt 0 count 2154424320
interrupt 1 count 2153942784
interrupt 2 count 0
interrupt 3 count 0
interrupt 4 count 2153893344
interrupt 5 count 2153893344
interrupt 6 count 2151296140
interrupt 7 count 2151296104
interrupt 8 count 6
interrupt 9 count 1048576
interrupt 10 count 0
interrupt 11 count 2432062003
interrupt 12 count 1366622208
interrupt 13 count 2154440320
interrupt 14 count 0
interrupt 15 count 2154442304
interrupt 16 count 2154439936
interrupt 17 count 0
interrupt 18 count 0
interrupt 19 count 0
interrupt 20 count 0
interrupt 21 count 0
interrupt 22 count 0
interrupt 23 count 0
interrupt 24 count 0
interrupt 25 count 2153896608
interrupt 26 count 16973824
interrupt 27 count 0
interrupt 28 count 0
interrupt 29 count 4
interrupt 30 count 0
interrupt 31 count 0
interrupt storm detected on "int2"; throttling interrupt source
interrupt storm detected on "int2"; throttling interrupt source
interrupt storm detected on "int2"; throttling interrupt source
interrupt storm detected on "int2"; throttling interrupt source
# ifconfig arge0 down
# vmstat -ia
interrupt total rate
sint0: 0 0
sint1: 0 0
int0 ath0 2938765 80
int1: 0 0
int2 arge0 335493 9
int3 arge1 0 0
int4 apb0 1936 0
int5 clock0 1129353 30
apb intr3: uart0 1936 0
apb intr2: gpio0 0 0
apb irq5: pmc 0 0
Total 4407483 120
-------------- next part --------------
A non-text attachment was scrubbed...
Name: if_arge-interrupt_counters.patch
Type: text/x-patch
Size: 1959 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-mips/attachments/20140726/cc68cdc2/attachment.bin>
More information about the freebsd-mips
mailing list