sort(1) memory usage

[Wayne Sierke]

On Sun, 2008-02-03 at 14:13 +0100, Ed Schouten wrote:
> * Dag-Erling Smørgrav <des at des.no> wrote:
> > I've been trying to figure out why some periodic scripts consume so much
> > memory.  I've narrowed it down to sort(1).
> > 
> > At first, I thought the scripts were using it inefficiently, feeding it
> > more data than was really needed.  Then I discovered this:
> > 
> > des at ds4 ~% (sleep 10 | sort) & (sleep 5 ; top -o res | grep sort)
> > [1] 66024
> > 66024 des          1  -8    5 54796K 52680K piperd 1   0:00  0.88% sort
> > 
> > That's right - sort(1) consumes 50+ MB of memory doing *nothing*.
> > 
> > (roughly half that on a 32-bit box)
> > 
> > Something is rotten in the state of GNU...
> 
> On my i386 box it spends 27M, but when I replace `sort' with `sed',
> without any arguments, it's only 1.4 MB. I tried this on RELENG_6. I can
> also reproduce this on Linux.
> 

%uname -vm
FreeBSD 7.0-PRERELEASE #1: Fri Jan 25 01:08:47 CST 2008     root at freebsd7stable-1.vmware:/usr/obj/usr/src/sys/GENERIC-KTR-0x2000  i386
%( sleep 10 | sort ) & ( sleep 5 ; top -n 150 | grep sort )
[2] 38158
38158 ws            1  -8    0 29760K   736K piperd   0:00  0.00% sort
%su -
# ( sleep 10 | sort ) & ( sleep 5 ; top -n 150 | grep sort )
[2] 38165
38165 root          1  -8    0 29760K   732K piperd   0:00  0.00% sort


$ uname -vm
FreeBSD 6.3-PRERELEASE #1: Fri Dec 28 17:49:43 CST 2007     root at predator-ii.buffyverse:/usr/obj/usr/src/sys/LILLITH-IV  i386
$ (sleep 10 | sort) & (sleep 5; top -n 150 | grep sort)
68953 ws           1  -8    0 26988K   660K piperd   0:00  0.00% sort
$ su -
#  (sleep 10 | sort) & (sleep 5; top -n 150 | grep sort)
[1] 68981
68981 root         1  -8    0 26988K   660K piperd   0:00  0.00% sort


Next one is Ubuntu 7.04

$ uname -a
Linux developer 2.6.20-16-generic #2 SMP Thu Jun 7 20:19:32 UTC 2007 i686 GNU/Linux
$ (sleep 10 | sort) & (sleep 5; ps aux | grep -E "^USER|sort$")
[9] 10523
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ws       10526  0.0  0.0  28436   676 pts/4    S    01:29   0:00 sort


(I had to change the 'top' incantation to pick up the 'sort' process on
these busy boxes.)


Wayne