/var full

[Paul Schmehl]

At 10PM (local time) this evening, a server started reporting that /var 
was full.  When I ssh'd in to the server to investigate, df said /var was 
at 2% full (5.1G) and dh reported the same (5.1G).  /var/log/dmesg.today 
is full of messages listing multiple entries with the same inode number 
followed by one entry listing dd as the culprit.

+pid 730 (mysqld), uid 88 inumber 7089166 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 7089166 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 7089166 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 7089166 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 7089166 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 7089166 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 27132107 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full
+pid 73338 (dd), uid 2 inumber 27131975 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full
+pid 730 (mysqld), uid 88 inumber 27132128 on /var: filesystem full
+pid 73365 (dd), uid 2 inumber 27131920 on /var: filesystem full

And so on.  What is going on?  The system seems to be functioning 
normally.  df and dh report normal disk usage.  Top looks fine. 
/var/log/dmesg.today shows 309 total lines and 98 unique lines.

Was this some sort of temporary glitch?  Or something more ominous?  Why 
would toor be running dd?  Is it some sort of file recovery routine 
triggered by filesystem full messages?

Paul Schmehl
If it isn't already obvious,
my opinions are my own and not
those of my employer.