MSI-X limitation in freebsd 8.2

[John Baldwin]

On Thursday, June 21, 2012 8:50:06 am Venkat Duvvuru wrote:
> While I'm able to ping to the machine..every command on the current active
> session is sluggish infact doesn't complete, unable to open another ssh
> session..an already opened ssh session doesn't react which is running
> "systat -vmstat"...
> 
> The only thing I could do is to successfully complete "top -P" command
> once..even top hung for the second time.

Hmm, is this a stock FreeBSD driver or an out-of-tree NIC driver?

> /Venkat
> 
> On Thu, Jun 21, 2012 at 5:46 PM, John Baldwin <jhb at freebsd.org> wrote:
> 
> > On Thursday, June 21, 2012 5:47:48 am Venkat Duvvuru wrote:
> > > John - Thanks for the reply.
> > >
> > > All the CPUs are ~100% idle. I don't see any interrupt storm on any of
> > the
> > > irqs (vmstat -i).
> > >
> > > One observation I made is that I see messages like these in dmesg
> > >
> > > ===> mem
> > 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf1c000-0xfaf1ffff
> > > irq 40 at device 0.1 on pci6
> > >
> > > Looking at the irq value I think it is the INTx irq range which shouldn't
> > > have probably got allocated as the device is msix capable and there are
> > > vectors allocated for these devices in the range (256-380).
> > >
> > > Could this be a problem?
> >
> > No, that line is output before the driver's attach routine is run, so it
> > will
> > always show INTx IRQ value even if it isn't used.
> >
> > > The scenario where I am hitting this problem is a setup with 4 NICs, each
> > > NIC with two ports and each port using up 4 msix vectors. The system is
> > > fine till some ports are up but once I ifup the 5th port, the system
> > > becomes sluggish.
> > >
> > > I'm not sure whether all the 30 vectors are from a single cpu..I don't
> > know
> > > how to get that information.
> >
> > Unfortunately there isn't an easy way.  I have this gdb script which can
> > display it from kgdb on x86:
> >
> > define irqs
> >    set $e = event_list->tqh_first
> >    while ($e != 0)
> >        if ($e->ie_source != 0 && $e->ie_handlers.tqh_first != 0)
> >            set $src = (struct intsrc *)$e->ie_source
> >            if ($src->is_pic->pic_enable_source == &ioapic_enable_source)
> >                set $_cpu = ((struct ioapic_intsrc *)$src)->io_cpu
> >            else
> >                if ($src->is_pic->pic_enable_source == &msi_enable_source)
> >                    set $_cpu = ((struct msi_intsrc *)$src)->msi_cpu
> >                else
> >                    set $_cpu = 0
> >                end
> >            end
> >            printf "CPU %d: %s\n", $_cpu, $e->ie_fullname
> >        end
> >        set $e = $e->ie_list.tqe_next
> >    end
> > end
> >
> > document irqs
> > Dump list of IRQs with associated CPU.
> > end
> >
> > However, unless the driver is using BUS_BIND_IRQ() or you are using cpuset
> > -x,
> > the interrupts should be round-robin assigned among CPUs.
> >
> > What exactly do you mean by sluggish?  Trying to interact with the box over
> > SSH is sluggish?  Is there a change in RTT if you are pinging the box, is
> > there a change in performance of TCP or UDP streams to/from the box?
> >
> > --
> > John Baldwin
> >
> 

-- 
John Baldwin