Performance of SheevaPlug on 8-stable

[Maks Verver]

Hi everyone,

After a bit of patching and tinkering I got my SheevaPlug to boot
FreeBSD from a UFS2-formatted USB stick. To compare it with Linux I
decided to run nbench to see how FreeBSD compares with Ubuntu (which is
shipped with the SheevaPlug). To my surprise, the results were
atrocious! FreeBSD scores about 50 times worse than Ubuntu.

Of course, this performance difference is too large to be caused by
implementation differences. There must be something more fundemental
wrong here. To simplify things, I created a simple testcase that counts
up to the maximum value of an integer:

  int main() { int i = 0; do ++i; while(i > 0); return 0; }

This compiles to: (both on Linux and on FreeBSD)

0000848c <main>:
    848c:       e3a03000        mov     r3, #0  ; 0x0
    8490:       e2833001        add     r3, r3, #1      ; 0x1
    8494:       e3530000        cmp     r3, #0  ; 0x0
    8498:       cafffffc        bgt     8490 <main+0x4>
    849c:       e3a00000        mov     r0, #0  ; 0x0
    84a0:       e1a0f00e        mov     pc, lr

This stresses the CPU and not much else. Since there are three
instructions in the loop and the SheevaPlug runs at 1.2 GHz, I
expect this to take around (1<<31)*3/1.2e9 ~ 5.3687 seconds. On Ubuntu:

$ time ./test
real	0m5.422s
user	0m5.390s
sys	0m0.020s

Exactly as expected. On FreeBSD on the other hand:

%time ./test
286.000u 0.000s 4:47.22 99.8%   40+1321k 0+0io 0pf+0w

This takes almost five minutes, or over 50 times as long! All of it is
user-space CPU time. Does anybody have a suggestion why the CPU appears
to run so slowly in FreeBSD?

I pored over my kernel configuration but I don't see anything suspect. I
did (manually) apply Hans Petter Selasky's patch [1] to be able to boot
from USB, and consequently removed the NFS and BOOTP stuff from the
config provided at sys/arm/conf/SHEEVAPLUG. Furthermore I removed the
NO_SWAPPING and NO_FFS_SNAPSHOT options (because I plan to attach a USB
disk drive) and I left in the KDB and DDB options because as I think
they do not significantly affect performance. Is this correct?

Kind regards,
Maks Verver.

P.S.  The strange thing is that stuff like network performance is
perfectly fine. I can fetch FTP data at 11 MB/s, which is about the
maximum possible on the cheap 100 Mbit switch I use, and is even a few
percent better than Ubuntu. So it seems it's really the CPU that's the
bottleneck, for no apparent reason.

[1] http://p4db.freebsd.org/chv.cgi?CH=169183