reason for "magic" crashes.

[Mateusz Guzik]

On Sun, Jun 24, 2012 at 07:05:35PM +0200, Wojciech Puchar wrote:
> i've got third crash third week in a row.
> 
> Every time in sunday after 18:00, every time with rsync process
> (which means rsync based backup that is done every day, not just in
> sunday!),
> 
> you may see a crash (viewed from KVM) at
> 
> http://www.tensor.gdynia.pl/~wojtek/crash.png
> 
> what is important - syncing disk doesn't go on, system hangs here.
> 
> For 99% system is not overheating at sunday, but i will be 100% sure
> as i added ipmitool sensor logged from cron every 5 minutes.
> 
> Please give me an idea what to check.
> 
> 
> There is nothing in cron that is done at sunday.
> 
> i don't run "periodic" stuff in /etc/crontab
> 

Compile the kernel with the following:

makeoptions     DEBUG="-O0 -g"

options         KDB                     # Enable kernel debugger support.
options         DDB                     # Support DDB.
options         GDB                     # Support remote GDB.
options         DEADLKRES               # Enable the deadlock resolver
options         INVARIANTS              # Enable calls of extra sanity checking
options         INVARIANT_SUPPORT       # Extra sanity checks of internal structures, required by INVARIANTS
options         WITNESS                 # Enable checks to detect deadlocks and cycles
options         WITNESS_SKIPSPIN        # Don't run witness on spinlocks for speed
options         DIAGNOSTIC

After kernel panic ddb prompt will be waiting for you. Type in:
dump <enter>
reset <enter>

Make sure you have swap that can handle crashdumps.

See this for more details:
http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html

You can check if everything works correctly by issuing panic manually:
sysctl debug.kdb.panic=1

then typing aforementioned ddb commands. After reboot you should get
core in /var/crash.

Also provide the following:
- system version
- filesystems involved in rsync with mount details (e.g. UFS with SU+J)
- dmesg

Hopefully this will be enough for someone to help.

-- 
Mateusz Guzik <mjguzik gmail.com>