RELENG_4 -> 5 -> 6: significant performance regression

[Dmitry Pryanishnikov]

Hello!

On Thu, 27 Apr 2006, Kris Kennaway wrote:

  Thanks for your suggestions, they've made a difference (though not as big
as one could hope).

> On Thu, Apr 27, 2006 at 05:08:11PM +0300, Dmitry Pryanishnikov wrote:
>
>> makeoptions	CONF_CFLAGS=-fno-builtin
>
> Non-default option; this may conceivably affect performance.

  The reason why I've initially included this option is the following
comment (NOTES from RELENG_6):

#
# CONF_CFLAGS gives some extra compiler flags that are added to ${CFLAGS}
# after most other flags.  Here we use it to inhibit use of non-optimal
# gcc builtin functions (e.g., memcmp).
#

I've read this: using "CONF_CFLAGS=-fno-builtin" inhibits use of non-optimal
gcc builtin functions, so this option may be useful for getting max. 
performance. Are this comment and my interpretation still correct now?

>> options 	INVARIANTS
>> options 	INVARIANT_SUPPORT
>
> These definitely effect performance, much more in 5.x and 6.x (at the
> 10-20% level) than 4.x.

  It's a pity - those help to catch a lot of software bugs ;(

>> options 	QUOTA
>
> This definitely effects performance on 6.x since it makes your
> filesystem giant-locked, which may also interfere with your network
> processing.

  It's also a pity.

>>                  %Sys   %Intr   %Idl
>>RELENG_4 + rl0      14      14     72
>>RELENG_4 + fxp0     14      10     76
>>RELENG_5 + rl0      40      30     30
>>RELENG_5 + fxp0     35      25     40
>>RELENG_6 + rl0      45      40     15
>>RELENG_6 + fxp0     45      35     20
>
> Please retry without.  Also make sure there are no other diagnostic
> messages at boot time about e.g. mpsafenet being forced to 0.

I've removed all mentioned options and redone benchmarks. There are no
diagnostics about mpsafe* (last one I've seen due to IPSEC, and I've
replaced it with FAST_IPSEC+crypto before doing previous benchmarks).
New results are pretty close for RELENG_5 and RELENG_6:

                   %Sys   %Intr   %Idl  "time md5 -t" wall clock time
RELENG_5 + rl0      33      23     44   1:41
RELENG_5 + fxp0     30      20     50   1:36

RELENG_6 + rl0      34      24     42   1:43
RELENG_6 + fxp0     30      20     50   1:40

So performance now is much better then before removal of

makeoptions	CONF_CFLAGS=-fno-builtin
options 	INVARIANTS
options 	INVARIANT_SUPPORT
options 	QUOTA

(I'll try to find out which one of these takes which % of overhead when I
get free time), but still much worse then under RELENG_4, where this
particular (I'd say "quote common") usage pattern takes 24-28% of CPU time, 
while under RELENG_5 / 6 it takes >= 50% ;(

> Kris

Sincerely, Dmitry
-- 
Atlantis ISP, System Administrator
e-mail:  dmitry at atlantis.dp.ua
nic-hdl: LYNX-RIPE