Slow creating lots of files...

[Marco Greene (Home)]

Doing some further research into my NetBackup slow restore performance...I
have noticed that if I create a lot of small files....it can take about 10
times as long as creating one big file of the same size.

With a sample set of 500MB...
Creating 500MB with of text files (about 30,000 of them) takes about 10
minutes
Creating 1 500 MB file takes about 1 minute.
Restoring 1 500MB file takes about 1 minute or so...a little more to allow
the robot to mount the tape/position the tape etc.
Restoring the 30,000 files with NetBackup takes about 20 minutes
Restoring the 30,000 files to an alternate location takes about 40 minutes

Now interms of backing this all up...well 2.5GB of it takes about 8 minutes.



My hardware platform is a Compaq DL360 with dual PIII/933MHz CPUs (only one
configured with the kernel to date).
Two 36GB (10K RPM) drives configured with Hardware RAID1
1 GB RAM.

(The OS is configured with a 2 GB swap space.) 

Here is the FreeBSD Slice...
# /dev/idad0s1c:
type: ESDI
disk: idad0s1
label:
flags:
bytes/sector: 512
sectors/track: 32
tracks/cylinder: 255
sectors/cylinder: 8160
cylinders: 8320
sectors/unit: 67891200
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0           # milliseconds
track-to-track seek: 0  # milliseconds
drivedata: 0

8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:  2097152        0    4.2BSD     2048 16384    89   # (Cyl.    0 - 257*)
  b:  4194304  2097152      swap                        # (Cyl.  257*- 771*)
  c: 67891200        0    unused        0     0         # (Cyl.    0 - 8319)
  e:  8388608  6291456    4.2BSD     2048 16384    89   # (Cyl.  771*-
1799*)
  f: 49152000 14680064    4.2BSD     2048 16384    89   # (Cyl. 1799*-
7822*)
  g:  4059136 63832064    4.2BSD     2048 16384    90   # (Cyl. 7822*-
8319*)



mislog01 # df -h
Filesystem      Size   Used  Avail Capacity  Mounted on
/dev/idad0s1a  1008M    50M   878M     5%    /
/dev/idad0s1g   1.9G   4.0K   1.8G     0%    /home
/dev/idad0s1e   3.9G   976M   2.7G    26%    /usr
/dev/idad0s1f    23G   2.5G    19G    12%    /var
procfs          4.0K   4.0K     0B   100%    /proc


Have any of you seen this before?  Is there some tuning that I can do?