ZFS - directory entry

[Dirk-Willem van Gulik]

> On 14 Dec 2016, at 19:51, Peter Jeremy <peter at rulingia.com> wrote:
> 
> On 2016-Dec-14 16:27:00 +0100, Dirk-Willem van Gulik <dirkx at webweaving.org> wrote:
>> A rather odd directory entry (in /root, the home dir of root/toor) appeared on a bog standard FreeBSD 10.2 (p18) lightly loaded machine under ZFS during/post a backup:
>> 
>> $ ls -la /root | tail -q
>> ----------   1 root  wheel  9223372036854775807 Jan  1  1970 ?%+?kD?H???x,?5?Dh;*s!?h???jw??????\h?:????????``?13?@?????OA????????Puux????<T]???R??Qv?g???]??%?R?
>> 
>> OS and ZFS is installed with a bog standard sysinstall. ‘SMART’ nor smartd have reported anything. nothing in dmesg, syslog of boot log. Any suggestions as how to debug or get to the root of this ? 
>> 
>> And in particular - what is a risk of a reboot (to get a kernel with debug, etc) causing the issue to ‘go away’ - and hence stopping the forensic ?
> 
> Do you have ECC RAM?  If not, it's possible this is an artifact of some RAM
> corruption, rather than on-disk corruption.
> 
> I'm surprised by the slow scrub, though they are very slow disks.  You might
> like to use gstat or zpool iostat to see if one of the disks is slower than
> the others - indicating a possible problem with it.

For the record - no such imbalance was found (all disks a specced performance; SMART data yielded nothing either).  Scrub found nothing. A zfs send & compares of snapshots prior and post the entry did not yield anything conclusive (But -1 entries in that directory).

A reboot did not fix the issue — i.e. it appeared resident on disk post reboot (and in zfs send). An extensive lowlevel/bios memtest (memtest.exe et.al. through PXE) did not find any HW issues; nor did a SMART level disk check on all disks.

Ultimately the ‘file’ was deleted with a simple ‘rm’. Took about 3 seconds to return with a prompt. And that was it.

A post remove “zfs send” followed by a "zfs scrub" found no ill effect/lost data (nor did tripwire throughout it all).

So very odd all in all - and mildly unsatisfying :)

Dw