[Bug 236989] AWS EC2 lockups "Missing interrupt"
bugzilla-noreply at freebsd.org
bugzilla-noreply at freebsd.org
Mon May 11 20:56:10 UTC 2020
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236989
--- Comment #26 from Charles O'Donnell <cao at bus.net> ---
Not sure if this is expected behavior with the patched kernel.
Picking up from day 6 above:
===> Thu May 7 12:00:00 EDT 2020
mbuf_jumbo_page: 4096, 490945, 182, 31,45730287, 0, 0
mbuf_jumbo_9k: 9216, 145465, 7461, 259,66753693,1693, 0
mbuf_jumbo_16k: 16384, 81824, 0, 0, 0, 0, 0
dev.ena.0.queue7.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue6.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue5.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue4.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue3.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue2.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue1.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue0.rx_ring.mjum_alloc_fail: 0
===> Fri May 8 12:00:00 EDT 2020
mbuf_jumbo_page: 4096, 490945, 82, 71,52207112, 0, 0
mbuf_jumbo_9k: 9216, 145465, 7551, 351,75614231,2625, 0
mbuf_jumbo_16k: 16384, 81824, 0, 0, 0, 0, 0
dev.ena.0.queue7.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue6.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue5.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue4.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue3.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue2.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue1.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue0.rx_ring.mjum_alloc_fail: 0
===> Sat May 9 12:00:00 EDT 2020
mbuf_jumbo_page: 4096, 490945, 139, 69,56246475, 0, 0
mbuf_jumbo_9k: 9216, 145465, 7799, 339,83036796,2784, 0
mbuf_jumbo_16k: 16384, 81824, 0, 0, 0, 0, 0
dev.ena.0.queue7.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue6.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue5.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue4.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue3.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue2.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue1.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue0.rx_ring.mjum_alloc_fail: 0
===> Sun May 10 12:00:00 EDT 2020
mbuf_jumbo_page: 4096, 490945, 39, 51,59639876, 0, 0
mbuf_jumbo_9k: 9216, 145465, 7663, 222,85652082,2963, 0
mbuf_jumbo_16k: 16384, 81824, 0, 0, 0, 0, 0
dev.ena.0.queue7.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue6.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue5.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue4.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue3.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue2.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue1.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue0.rx_ring.mjum_alloc_fail: 0
===> Mon May 11 12:00:00 EDT 2020
mbuf_jumbo_page: 4096, 490945, 1598, 223,63502113, 0, 0
mbuf_jumbo_9k: 9216, 145465, 6230, 90,91087037,4248, 0
mbuf_jumbo_16k: 16384, 81824, 0, 0, 0, 0, 0
dev.ena.0.queue7.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue6.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue5.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue4.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue3.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue2.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue1.rx_ring.mjum_alloc_fail: 0
dev.ena.0.queue0.rx_ring.mjum_alloc_fail: 0
System spinning. Had to reboot.
Related /var/log/messages activity show below.
May 11 12:20:34 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:20:34 s4 kernel: ena0: Trigger reset is on
May 11 12:20:34 s4 kernel: ena0: device is going DOWN
May 11 12:20:39 s4 kernel: ena0: free uncompleted tx mbuf qid 1 idx 0x2e1
May 11 12:20:40 s4 kernel: ena0: device is going UP
May 11 12:20:40 s4 kernel: ena0: link is UP
May 11 12:21:55 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:21:55 s4 kernel: ena0: Trigger reset is on
May 11 12:21:55 s4 kernel: ena0: device is going DOWN
May 11 12:22:02 s4 kernel: ena0: free uncompleted tx mbuf qid 1 idx 0x172ena0:
free uncompleted tx mbuf qid 3 idx 0x66ena0: free>
May 11 12:22:03 s4 kernel:
May 11 12:22:03 s4 kernel: stray irq265
May 11 12:22:03 s4 kernel: ena0: ena0: device is going UP
May 11 12:22:03 s4 kernel: link is UP
May 11 12:22:29 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:22:29 s4 kernel: ena0: Trigger reset is on
May 11 12:22:29 s4 kernel: ena0: device is going DOWN
May 11 12:22:32 s4 kernel: ena0: free uncompleted tx mbuf qid 2 idx 0x32bena0:
free uncompleted tx mbuf qid 4 idx 0x338ena0: fre>
May 11 12:22:33 s4 kernel:
May 11 12:22:33 s4 kernel: stray irq265
May 11 12:22:33 s4 kernel: ena0: ena0: device is going UP
May 11 12:22:33 s4 kernel: link is UP
May 11 12:23:04 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:23:04 s4 kernel: ena0: Trigger reset is on
May 11 12:23:04 s4 kernel: ena0: device is going DOWN
May 11 12:23:11 s4 kernel: ena0: free uncompleted tx mbuf qid 6 idx 0x10e
May 11 12:23:12 s4 kernel: ena0: ena0: device is going UP
May 11 12:23:12 s4 kernel: link is UP
May 11 12:25:33 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:25:33 s4 kernel: ena0: Trigger reset is on
May 11 12:25:33 s4 kernel: ena0: device is going DOWN
May 11 12:25:38 s4 kernel: ena0: free uncompleted tx mbuf qid 0 idx 0x0
May 11 12:25:39 s4 kernel: ena0: ena0: device is going UP
May 11 12:25:39 s4 kernel: link is UP
May 11 12:28:36 s4 kernel: ena0: Keep alive watchdog timeout.
May 11 12:28:36 s4 kernel: ena0: Trigger reset is on
May 11 12:28:36 s4 kernel: ena0: device is going DOWN
May 11 12:28:39 s4 kernel: ena0: device is going UP
May 11 12:28:39 s4 kernel: ena0: link is UP
May 11 12:30:04 s4 kernel: nvme4: cpl does not map to outstanding cmd
May 11 12:30:04 s4 kernel: cdw0:00000000 sqhd:0013 sqid:0002 cid:0012 p:0 sc:00
sct:0 m:0 dnr:0
May 11 12:30:04 s4 kernel: nvme4: Missing interrupt
May 11 12:30:04 s4 kernel: nvme1: Missing interrupt
May 11 12:30:34 s4 kernel: nvme4: Resetting controller due to a timeout.
May 11 12:30:34 s4 kernel: nvme4: resetting controller
May 11 12:30:47 s4 kernel: nvme4: temperature threshold not supported
May 11 12:30:47 s4 kernel: nvme4: aborting outstanding i/o
May 11 12:30:47 s4 syslogd: last message repeated 3 times
May 11 12:31:26 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:31:26 s4 kernel: ena0: Trigger reset is on
May 11 12:31:26 s4 kernel: ena0: device is going DOWN
May 11 12:31:27 s4 kernel: ena0: free uncompleted tx mbuf qid 2 idx 0x20fena0:
free uncompleted tx mbuf qid 4 idx 0x8fena0: free>
May 11 12:31:28 s4 kernel:
May 11 12:31:28 s4 kernel: stray irq266
May 11 12:31:28 s4 kernel: stray irq272
May 11 12:31:28 s4 kernel: stray irq265
May 11 12:31:28 s4 kernel: ena0: device is going UP
May 11 12:31:28 s4 kernel: ena0: link is UP
May 11 12:32:46 s4 kernel: nvme1: Missing interrupt
May 11 12:32:46 s4 kernel: nvme4: cpl does not map to outstanding cmd
May 11 12:32:46 s4 kernel: cdw0:00000000 sqhd:0007 sqid:0001 cid:0015 p:1 sc:00
sct:0 m:0 dnr:0
May 11 12:32:46 s4 kernel: nvme4: Missing interrupt
May 11 12:32:46 s4 kernel: nvme2: Missing interrupt
May 11 12:33:16 s4 kernel: nvme4: Resetting controller due to a timeout.
May 11 12:33:16 s4 kernel: nvme4: resetting controller
May 11 12:33:17 s4 kernel: nvme4: temperature threshold not supported
May 11 12:33:17 s4 kernel: nvme4: aborting outstanding i/o
May 11 12:33:17 s4 syslogd: last message repeated 2 times
May 11 12:33:40 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:33:40 s4 kernel: ena0: Trigger reset is on
May 11 12:33:40 s4 kernel: ena0: device is going DOWN
May 11 12:33:43 s4 kernel: ena0: free uncompleted tx mbuf qid 3 idx 0x0
May 11 12:33:44 s4 kernel: ena0: ena0: device is going UP
May 11 12:33:44 s4 kernel: link is UP
May 11 12:34:15 s4 kernel: ena0: Keep alive watchdog timeout.
May 11 12:34:15 s4 kernel: ena0: Trigger reset is on
May 11 12:34:15 s4 kernel: ena0: device is going DOWN
May 11 12:34:21 s4 kernel: ena0: ena0: device is going UP
May 11 12:34:21 s4 kernel: link is UP
May 11 12:35:33 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:35:33 s4 kernel: ena0: Trigger reset is on
May 11 12:35:33 s4 kernel: ena0: device is going DOWN
May 11 12:35:42 s4 kernel: ena0: free uncompleted tx mbuf qid 3 idx 0x34
May 11 12:35:43 s4 kernel: ena0: ena0: device is going UP
May 11 12:35:43 s4 kernel: link is UP
May 11 12:36:16 s4 kernel: ena0: Keep alive watchdog timeout.
May 11 12:36:16 s4 kernel: ena0: Trigger reset is on
May 11 12:36:16 s4 kernel: ena0: device is going DOWN
May 11 12:36:21 s4 kernel: ena0: free uncompleted tx mbuf qid 7 idx 0x20a
May 11 12:36:22 s4 kernel: ena0: ena0: device is going UP
May 11 12:36:22 s4 kernel: link is UP
May 11 12:37:05 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:37:05 s4 kernel: ena0: Trigger reset is on
May 11 12:37:05 s4 kernel: ena0: device is going DOWN
May 11 12:37:09 s4 kernel: ena0: free uncompleted tx mbuf qid 4 idx 0x23f
May 11 12:37:10 s4 kernel: ena0: ena0: device is going UP
May 11 12:37:10 s4 kernel: link is UP
May 11 12:38:38 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:38:38 s4 kernel: ena0: Trigger reset is on
May 11 12:38:38 s4 kernel: ena0: device is going DOWN
May 11 12:38:43 s4 kernel: ena0: free uncompleted tx mbuf qid 5 idx 0x1b3
May 11 12:38:44 s4 kernel:
May 11 12:38:44 s4 kernel: too many stray irq 265's: not logging anymore
May 11 12:38:44 s4 kernel: ena0: ena0: device is going UP
May 11 12:38:44 s4 kernel: link is UP
May 11 12:39:17 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:39:17 s4 kernel: ena0: Trigger reset is on
May 11 12:39:17 s4 kernel: ena0: device is going DOWN
May 11 12:39:25 s4 kernel: ena0: free uncompleted tx mbuf qid 1 idx 0x238ena0:
free uncompleted tx mbuf qid 6 idx 0x249
May 11 12:39:26 s4 kernel: ena0: ena0: device is going UP
May 11 12:39:26 s4 kernel: link is UP
May 11 12:40:20 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:40:20 s4 kernel: ena0: Trigger reset is on
May 11 12:40:20 s4 kernel: ena0: device is going DOWN
May 11 12:40:24 s4 kernel: ena0: free uncompleted tx mbuf qid 0 idx 0x3c0
May 11 12:40:25 s4 kernel: ena0: device is going UP
May 11 12:40:25 s4 kernel: ena0: link is UP
May 11 12:41:17 s4 kernel: ena0: Keep alive watchdog timeout.
May 11 12:41:17 s4 kernel: ena0: Trigger reset is on
May 11 12:41:17 s4 kernel: ena0: device is going DOWN
May 11 12:41:22 s4 kernel: ena0: device is going UP
May 11 12:41:22 s4 kernel: ena0: link is UP
May 11 12:42:17 s4 kernel: ena0: Keep alive watchdog timeout.
May 11 12:42:17 s4 kernel: ena0: Trigger reset is on
May 11 12:42:17 s4 kernel: ena0: device is going DOWN
May 11 12:42:17 s4 sm-msp-queue[70209]: 04BGPgiQ066459: SYSERR(root): timeout
writing message to [127.0.0.1]
May 11 12:42:21 s4 kernel: ena0: ena0: device is going UP
May 11 12:42:21 s4 kernel: link is UP
May 11 12:43:01 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:43:01 s4 kernel: ena0: Trigger reset is on
May 11 12:43:01 s4 kernel: ena0: device is going DOWN
May 11 12:43:06 s4 kernel: ena0: free uncompleted tx mbuf qid 2 idx 0x359
May 11 12:43:07 s4 kernel:
May 11 12:43:07 s4 kernel: stray irq266
May 11 12:43:07 s4 kernel: ena0: ena0: device is going UP
May 11 12:43:07 s4 kernel: link is UP
May 11 12:45:23 s4 kernel: ena0: Keep alive watchdog timeout.
May 11 12:45:23 s4 kernel: ena0: Trigger reset is on
May 11 12:45:23 s4 kernel: ena0: device is going DOWN
May 11 12:45:33 s4 kernel: ena0: ena0: device is going UP
May 11 12:45:33 s4 kernel: link is UP
May 11 12:48:08 s4 kernel: ena0: Keep alive watchdog timeout.
May 11 12:48:08 s4 kernel: ena0: Trigger reset is on
May 11 12:48:08 s4 kernel: ena0: device is going DOWN
May 11 12:48:11 s4 kernel: ena0: ena0: device is going UP
May 11 12:48:11 s4 kernel: link is UP
May 11 12:48:51 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:48:51 s4 kernel: ena0: Trigger reset is on
May 11 12:48:51 s4 kernel: ena0: device is going DOWN
May 11 12:48:55 s4 kernel: ena0: free uncompleted tx mbuf qid 7 idx 0x266
May 11 12:48:55 s4 kernel: ena0: ena0: device is going UP
May 11 12:48:55 s4 kernel: link is UP
May 11 12:50:38 s4 kernel: ena0: Keep alive watchdog timeout.
May 11 12:50:38 s4 kernel: ena0: Trigger reset is on
May 11 12:50:38 s4 kernel: ena0: device is going DOWN
May 11 12:50:41 s4 kernel: ena0: ena0: device is going UP
May 11 12:50:41 s4 kernel: link is UP
May 11 12:51:43 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:51:43 s4 kernel: ena0: Trigger reset is on
May 11 12:51:43 s4 kernel: ena0: device is going DOWN
May 11 12:51:50 s4 kernel: ena0: free uncompleted tx mbuf qid 5 idx 0x358
May 11 12:51:51 s4 kernel: ena0: ena0: device is going UP
May 11 12:51:51 s4 kernel: link is UP
May 11 12:53:24 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:53:24 s4 kernel: ena0: Trigger reset is on
May 11 12:53:24 s4 kernel: ena0: device is going DOWN
May 11 12:53:28 s4 kernel: ena0: free uncompleted tx mbuf qid 5 idx 0x100
May 11 12:53:29 s4 kernel: ena0: ena0: device is going UP
May 11 12:53:29 s4 kernel: link is UP
May 11 12:56:19 s4 kernel: ena0: The number of lost tx completion is above the
threshold (129 > 128). Reset the device
May 11 12:56:19 s4 kernel: ena0: Trigger reset is on
May 11 12:56:19 s4 kernel: ena0: device is going DOWN
May 11 12:56:22 s4 kernel: ena0: free uncompleted tx mbuf qid 4 idx 0x3d8
May 11 12:56:23 s4 kernel: ena0: ena0: device is going UP
May 11 12:56:23 s4 kernel: link is UP
May 11 12:57:36 s4 kernel: .
--
You are receiving this mail because:
You are the assignee for the bug.
More information about the freebsd-virtualization
mailing list