Strange performance issue with grep -r -i as non-root user

[Jeremy Chadwick]

On Sat, Mar 05, 2011 at 03:01:40PM -1000, Clifton Royston wrote:
> On Sat, Mar 05, 2011 at 03:45:14PM -0800, Jeremy Chadwick wrote:
> > This is a strange one, and the more I started debugging it (starting
> > with truss, comparing fast vs. slow results, where all that appears
> > different is read() operations are taking a lot longer -- I haven't had
> > time to check with ktrace yet), the more strange it got: that's when I
> > found out the behaviour changes depending on if you're a user or root.
> > 
> > Easy to reproduce:
> > 
> > - grep -r string /usr/src, as non-root, is fast
> > - grep -r -i string /usr/src, as non-root, is 8x slower than without -i
> 
>   From your results below, I think you mean *80* x slower!

Oops; yes, typo on my part.  I was never any good at math either!  ;-)

> > - grep -r string /usr/src, as root, is fast
> > - grep -r -i string /usr/src, as root, is fast
> 
>   I can not reproduce this on 7.3-RELEASE-p4; I get consistent results
> between root and non-root, with -i being only marginally slower (about
> 15%) for each; results below.

Your results look more or less like what I see on the 4th system (the
7.0-STABLE one).  I believe the speed difference there (and on your
system) is justified, as I would imagine strcasecmp() a tiny bit slower
than strcmp().  But an 80x slowdown is completely unacceptable,
especially given the conditions.

My first thought was "compiler optimisation bug?", which I suppose could
still be the case, but I don't know how root vs. non-root would change
that behaviour, not to mention only when -i was specified.

Using 'truss -d' it looks like the slowdown is happening on read(2),
which makes me very concerned, as it could indicate something odd going
on with CAM?  Sadly I cannot (for many reasons) get rid of ahci.ko on
any of those 3 systems, so I can't compare stock ata(4) to ahci.ko
easily on the same system.

ktrace is next, but I have other things to do during the next few hours,
then I can spend some cycles on this.

>   Just to satisfy my peace of mind, can you do 'which grep' and 'sudo sh
> -c "which grep"' as jdc to make sure it's not path-related weirdness?

It's definitely not path weirdness, which is why I pointed out that the
greps on all the systems are identical.  Here's a more detailed version:

* System #1
  $ which grep ; grep --version | head -1 ; sudo sh -c 'which grep ; grep --version | head -1' ; ls -l `which grep`
  /usr/bin/grep
  grep (GNU grep) 2.5.1-FreeBSD
  /usr/bin/grep
  grep (GNU grep) 2.5.1-FreeBSD
  -r-xr-xr-x    9 root      wheel     86936  1 Mar 00:22 /usr/bin/grep

* System #2
  $ which grep ; grep --version | head -1 ; sudo sh -c 'which grep ; grep --version | head -1' ; ls -l `which grep`
  /usr/bin/grep
  grep (GNU grep) 2.5.1-FreeBSD
  /usr/bin/grep
  grep (GNU grep) 2.5.1-FreeBSD
  -r-xr-xr-x    9 root      wheel     86936 25 Feb 00:13 /usr/bin/grep

* System #3
  $ which grep ; grep --version | head -1 ; sudo sh -c 'which grep ; grep --version | head -1' ; ls -l `which grep`
  /usr/bin/grep
  grep (GNU grep) 2.5.1-FreeBSD
  /usr/bin/grep
  grep (GNU grep) 2.5.1-FreeBSD
  -r-xr-xr-x    9 root      wheel     86936 20 Oct 02:14 /usr/bin/grep

* System #4 (the one without the issue)
  $ which grep ; grep --version | head -1 ; sudo sh -c 'which grep ; grep --version | head -1' ; ls -l `which grep`
  /usr/bin/grep
  grep (GNU grep) 2.5.1-FreeBSD
  /usr/bin/grep
  grep (GNU grep) 2.5.1-FreeBSD
  -r-xr-xr-x    9 root      wheel     76728 19 Apr  2008 /usr/bin/grep

The file size difference should be expected given that it's a
significantly older system, and i386 as well.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |