Little research how rm -rf and tar kill server

[Mark Felder]

On Mon, Mar 30, 2015, at 20:24, Artem Kuchin wrote:
> 30.03.2015 19:09, Mark Felder пишет:
> >
> > On Mon, Mar 30, 2015, at 11:04, Artem Kuchin wrote:
> >> 30.03.2015 18:57, Mark Felder пишет:
> >>> On Mon, Mar 30, 2015, at 10:53, Artem Kuchin wrote:
> >>>> This is normal state, not under rm -rf
> >>>> Do you need it during  rm -rf  ?
> >>>>
> >>> No, but I wonder if changing the timer from LAPIC to HPET or possibly
> >>> one of the other timers makes the system more responsive under that
> >>> load. Would you mind testing that?
> >>>
> >>> You can switch the timer like this:
> >>>
> >>> sysctl kern.eventtimer.timer=HPET
> >>>
> >>> And then run some of your I/O tests
> >>>
> >> I see. I will test at night, when load goes down.
> >> I cannot say sure that's a right  way to dig, but i will test anything :)
> >>
> >> Just to remind: untar overloads the system, but untar + sync every 120s
> >> does not.
> >> That seems very strange to me.  I think the problem might be somewhere
> >> here.
> >>
> > I just heard from mav that there was a bottleneck in gmirror/graid with
> > regards to BIO_DELETE requests
> >
> > https://svnweb.freebsd.org/base?view=revision&revision=280757
> >
> 
> I applied this patch manually and rebuilt the kernel.
> Hit this bug
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195458
> on reboot, wasted 1 hour fsck-ing 2 times (was dirty after first fsck)
> and after boot tried doing
> rm -rf test1
> I coult not test anything, because it complete after 1 minute, instead 
> 15 minutes before.
> I copier the dir 4 times into subdirs and rm -rf full tree (4x larger) - 
> fast and smooth,
> mariadb did not tonice this, server were working fine.
> 
> However, i also noticed another thing:
> cp -Rp test test1
> also work a lot faster now, probably 3-5 times faster
> Maybe it is because fs is free of tons BIO_DELETE from other processes
> 
> 
> Then i did the untar test at maximum speed ( no pv to limit bandwidth):
> i see that mysql request became slower, but mysql sql request queue 
> built up slower now.
> However, when it reached 70 i stopped untar and mariadb could not 
> recover from condition
> until i executed sync. However, this time sync took only a second.
> I see big improvement, but i still don't understand why i need to issue 
> sync manually to push
> everything to recover from overload.
> 
> # man 2 sync
> a sync() system call is issued frequently by the user process syncer(4) 
> (about every 30 seconds).
> 
> it does not seem to be true
> 
> I checked syncer sysctl
> 
> # sysctl kern.filedelay
> kern.filedelay: 30
> # sysctl kern.dirdelay
> kern.dirdelay: 29
> # sysctl kern.metadelay
> kern.metadelay: 28
> 
> # ps ax | grep sync
> 23  -  DL     0:03.82 [syncer]
> 
> no clue why need manual sync
> 
> By the way: is there way to make sure  that SU+J is really working? 
> Maybe it is disabled for some reason
> and i don't know it. tunefs just shows stored setting, but, for example, 
> with dirty fs, journaling is not
> working in reality. Any way to get current status of SU journaling?
> 
> off topic: suggestion to move to ZFA was not so good, i see a "All 
> available memory used when deleting files from ZFS"
> topic. I'd rather have slow server when i can login and fix than halted 
> on panic. Just to point that ZFS still have plenty
> of unpredictable issues.
> 

This information is very good. Perhaps there is some more additional
tweaking that could be done. I will cc mav@ on this.