MSI-X limitation in freebsd 8.2

[Venkat Duvvuru]

and no matter in which order I bring up the interfaces, the system becomes
sluggish after the 5th interface is brought up. So doesn't look like a
specific driver issue.

On Fri, Jun 22, 2012 at 2:25 PM, Venkat Duvvuru
<venkatduvvuru.ml at gmail.com>wrote:

> This setup has variety of NICs, Intel's, Emulex's and all are stock
> drivers.
>
> /Venkat
>
>  On Thu, Jun 21, 2012 at 7:17 PM, John Baldwin <jhb at freebsd.org> wrote:
>
>> On Thursday, June 21, 2012 8:50:06 am Venkat Duvvuru wrote:
>> > While I'm able to ping to the machine..every command on the current
>> active
>> > session is sluggish infact doesn't complete, unable to open another ssh
>> > session..an already opened ssh session doesn't react which is running
>> > "systat -vmstat"...
>> >
>> > The only thing I could do is to successfully complete "top -P" command
>> > once..even top hung for the second time.
>>
>> Hmm, is this a stock FreeBSD driver or an out-of-tree NIC driver?
>>
>> > /Venkat
>> >
>> > On Thu, Jun 21, 2012 at 5:46 PM, John Baldwin <jhb at freebsd.org> wrote:
>> >
>> > > On Thursday, June 21, 2012 5:47:48 am Venkat Duvvuru wrote:
>> > > > John - Thanks for the reply.
>> > > >
>> > > > All the CPUs are ~100% idle. I don't see any interrupt storm on any
>> of
>> > > the
>> > > > irqs (vmstat -i).
>> > > >
>> > > > One observation I made is that I see messages like these in dmesg
>> > > >
>> > > > ===> mem
>> > > 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf1c000-0xfaf1ffff
>> > > > irq 40 at device 0.1 on pci6
>> > > >
>> > > > Looking at the irq value I think it is the INTx irq range which
>> shouldn't
>> > > > have probably got allocated as the device is msix capable and there
>> are
>> > > > vectors allocated for these devices in the range (256-380).
>> > > >
>> > > > Could this be a problem?
>> > >
>> > > No, that line is output before the driver's attach routine is run, so
>> it
>> > > will
>> > > always show INTx IRQ value even if it isn't used.
>> > >
>> > > > The scenario where I am hitting this problem is a setup with 4
>> NICs, each
>> > > > NIC with two ports and each port using up 4 msix vectors. The
>> system is
>> > > > fine till some ports are up but once I ifup the 5th port, the system
>> > > > becomes sluggish.
>> > > >
>> > > > I'm not sure whether all the 30 vectors are from a single cpu..I
>> don't
>> > > know
>> > > > how to get that information.
>> > >
>> > > Unfortunately there isn't an easy way.  I have this gdb script which
>> can
>> > > display it from kgdb on x86:
>> > >
>> > > define irqs
>> > >    set $e = event_list->tqh_first
>> > >    while ($e != 0)
>> > >        if ($e->ie_source != 0 && $e->ie_handlers.tqh_first != 0)
>> > >            set $src = (struct intsrc *)$e->ie_source
>> > >            if ($src->is_pic->pic_enable_source ==
>> &ioapic_enable_source)
>> > >                set $_cpu = ((struct ioapic_intsrc *)$src)->io_cpu
>> > >            else
>> > >                if ($src->is_pic->pic_enable_source ==
>> &msi_enable_source)
>> > >                    set $_cpu = ((struct msi_intsrc *)$src)->msi_cpu
>> > >                else
>> > >                    set $_cpu = 0
>> > >                end
>> > >            end
>> > >            printf "CPU %d: %s\n", $_cpu, $e->ie_fullname
>> > >        end
>> > >        set $e = $e->ie_list.tqe_next
>> > >    end
>> > > end
>> > >
>> > > document irqs
>> > > Dump list of IRQs with associated CPU.
>> > > end
>> > >
>> > > However, unless the driver is using BUS_BIND_IRQ() or you are using
>> cpuset
>> > > -x,
>> > > the interrupts should be round-robin assigned among CPUs.
>> > >
>> > > What exactly do you mean by sluggish?  Trying to interact with the
>> box over
>> > > SSH is sluggish?  Is there a change in RTT if you are pinging the
>> box, is
>> > > there a change in performance of TCP or UDP streams to/from the box?
>> > >
>> > > --
>> > > John Baldwin
>> > >
>> >
>>
>> --
>> John Baldwin
>>
>
>