amrd disk performance drop after running under high load

[Alexey Popov]

Hi

Kris Kennaway wrote:
>>>>>> So I can conclude that FreeBSD has a long standing bug in VM that 
>>>>>> could be triggered when serving large amount of static data (much 
>>>>>> bigger than memory size) on high rates. Possibly this only applies 
>>>>>> to large files like mp3 or video. 
>>>>> It is possible, we have further work to do to conclude this though.
>>>> I forgot to mention I have pmc and kgmon profiling for good and bad 
>>>> times. But I have not enough knowledge to interpret it right and not 
>>>> sure if it can help.
>>> pmc would be useful.
>> pmc profiling attached.
> OK, the pmc traces do seem to show that it's not a lock contention 
> issue.  That being the case I don't think the fact that different 
> servers perform better is directly related. 
But it was evidence of mbuf lock contention in mutex profiling, wasn't 
it? As far as I understand, mutex problems can exist without increasing 
CPU load in pmc stats, right?

> There is also no evidence of a VM problem.  What your vmstat and pmc 
> traces show is that your system really isn't doing much work at all, 
> relatively speaking.
> There is also still no evidence of a disk problem.  In fact your disk 
> seems to be almost idle in both cases you provided, only doing between 1 
> and 10 operations per second, which is trivial.
vmstat and network output graphs shows that the problem exists. If it is 
not a disk or network or VM problem, what else could be wrong?

> In the "good" case you are getting a much higher interrupt rate but with 
> the data you provided I can't tell where from.  You need to run vmstat 
> -i at regular intervals (e.g. every 10 seconds for a minute) during the 
> "good" and "bad" times, since it only provides counters and an average 
> rate over the uptime of the system.
I'll try this, but AFAIR there was no strangeness with interrupts.

I believe the reason of high interrupt rate in "good" cases is that 
server sends much traffic.

> What there is evidence of is an interrupt aliasing problem between em 
> and USB:
> irq16: uhci0                  1464547796       1870
> irq64: em0                    1463513610       1869
I tried disabling USB in kernel, this ussie was gone, but the main 
problem was left. Also I have this issue with interrupt aliasing on many 
servers without problems.

With best regards,
Alexey Popov