stuck with ufs CHECK-HASH errors
Don Lewis
truckman at FreeBSD.org
Tue Dec 4 04:45:44 UTC 2018
I just ran into an issue with a vm guest running 13.0-CURRENT r341355
GENERIC amd64.
The problem started when the vm guest paniced and when I rebooted it
encountered an unexpected SU inconsitency and dropped into single user
mode. When I ran fsck manually, I *think* it asked if I wanted to enable
CHECK-HASH and I said yes. Now when I run fsck, it reports a bunch of
CHECK-HASH errors, but doesn't fix them.
I've copied the filesystem to a local file on the host that I'm
accessing via /dev/md0 so that I can easily capture the results.
# fsck /dev/md0
** /dev/md0
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
CG 0: BAD CHECK-HASH 0 vs 0x767f70f1
SUMMARY INFORMATION BAD
SALVAGE? [yn] y
ALLOCATED FILE 273 MARKED FREE
ALLOCATED FRAG 37855 MARKED FREE
BLK(S) MISSING IN BIT MAPS
SALVAGE? [yn] y
CG 1: BAD CHECK-HASH 0 vs 0x2a1c3669
CG 2: BAD CHECK-HASH 0 vs 0x4271e253
CG 3: BAD CHECK-HASH 0 vs 0x2b2ec911
CG 4: BAD CHECK-HASH 0 vs 0x650a2356
CG 5: BAD CHECK-HASH 0 vs 0x78a8ef2b
CG 6: BAD CHECK-HASH 0 vs 0xb9879f89
CG 7: BAD CHECK-HASH 0 vs 0x5e6cffed
CG 8: BAD CHECK-HASH 0 vs 0xacad5919
CG 9: BAD CHECK-HASH 0 vs 0x54f7e667
CG 10: BAD CHECK-HASH 0 vs 0x9b819e39
CG 11: BAD CHECK-HASH 0 vs 0xf4f4ab1a
CG 12: BAD CHECK-HASH 0 vs 0x2c9be1ec
CG 13: BAD CHECK-HASH 0 vs 0x52af104
CG 14: BAD CHECK-HASH 0 vs 0xcfc9d02d
CG 15: BAD CHECK-HASH 0 vs 0xaf439644
CG 16: BAD CHECK-HASH 0 vs 0x8ff167a7
CG 17: BAD CHECK-HASH 0 vs 0x24cf0d5f
CG 18: BAD CHECK-HASH 0 vs 0x7c9b00c5
CG 19: BAD CHECK-HASH 0 vs 0x8e58a45d
CG 20: BAD CHECK-HASH 0 vs 0x427a8a1
CG 21: BAD CHECK-HASH 0 vs 0x7b624c0c
CG 22: BAD CHECK-HASH 0 vs 0x4016d045
CG 23: BAD CHECK-HASH 0 vs 0xbeb868d1
CG 24: BAD CHECK-HASH 0 vs 0xbf3550a7
CG 25: BAD CHECK-HASH 0 vs 0xe1748785
CG 26: BAD CHECK-HASH 0 vs 0xacc5837a
CG 27: BAD CHECK-HASH 0 vs 0x397b8e79
CG 28: BAD CHECK-HASH 0 vs 0xef868063
CG 29: BAD CHECK-HASH 0 vs 0x28ab03a9
CG 30: BAD CHECK-HASH 0 vs 0x8c1c2f40
CG 31: BAD CHECK-HASH 0 vs 0x2825b898
CG 32: BAD CHECK-HASH 0 vs 0x57200d54
CG 33: BAD CHECK-HASH 0 vs 0x8ddd9f4e
CG 34: BAD CHECK-HASH 0 vs 0xa25fc59b
CG 35: BAD CHECK-HASH 0 vs 0xbaf45843
CG 36: BAD CHECK-HASH 0 vs 0xb82def37
CG 37: BAD CHECK-HASH 0 vs 0x320cb852
CG 38: BAD CHECK-HASH 0 vs 0xa8d433d9
CG 39: BAD CHECK-HASH 0 vs 0x4fb023bb
CG 40: BAD CHECK-HASH 0 vs 0x7606a0ac
CG 41: BAD CHECK-HASH 0 vs 0xb25d3f34
CG 42: BAD CHECK-HASH 0 vs 0xb43acd7f
CG 43: BAD CHECK-HASH 0 vs 0xd3c1ed7a
CG 44: BAD CHECK-HASH 0 vs 0xf5df5b0d
CG 45: BAD CHECK-HASH 0 vs 0x5f070643
CG 46: BAD CHECK-HASH 0 vs 0x1038a213
CG 47: BAD CHECK-HASH 0 vs 0x5e038a7b
CG 48: BAD CHECK-HASH 0 vs 0xe30f28bb
CG 49: BAD CHECK-HASH 0 vs 0x853827ef
720309 files, 2926458 used, 4686237 free (279357 frags, 550860 blocks, 3.7% fragmentation)
***** FILE SYSTEM IS CLEAN *****
***** FILE SYSTEM WAS MODIFIED *****
If I run fsck again, I get the same result:
# fsck /dev/md0
** /dev/md0
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
CG 0: BAD CHECK-HASH 0 vs 0x767f70f1
SUMMARY INFORMATION BAD
SALVAGE? [yn] y
ALLOCATED FILE 273 MARKED FREE
ALLOCATED FRAG 37855 MARKED FREE
BLK(S) MISSING IN BIT MAPS
SALVAGE? [yn] y
CG 1: BAD CHECK-HASH 0 vs 0x2a1c3669
CG 2: BAD CHECK-HASH 0 vs 0x4271e253
CG 3: BAD CHECK-HASH 0 vs 0x2b2ec911
CG 4: BAD CHECK-HASH 0 vs 0x650a2356
CG 5: BAD CHECK-HASH 0 vs 0x78a8ef2b
CG 6: BAD CHECK-HASH 0 vs 0xb9879f89
CG 7: BAD CHECK-HASH 0 vs 0x5e6cffed
CG 8: BAD CHECK-HASH 0 vs 0xacad5919
CG 9: BAD CHECK-HASH 0 vs 0x54f7e667
CG 10: BAD CHECK-HASH 0 vs 0x9b819e39
CG 11: BAD CHECK-HASH 0 vs 0xf4f4ab1a
CG 12: BAD CHECK-HASH 0 vs 0x2c9be1ec
CG 13: BAD CHECK-HASH 0 vs 0x52af104
CG 14: BAD CHECK-HASH 0 vs 0xcfc9d02d
CG 15: BAD CHECK-HASH 0 vs 0xaf439644
CG 16: BAD CHECK-HASH 0 vs 0x8ff167a7
CG 17: BAD CHECK-HASH 0 vs 0x24cf0d5f
CG 18: BAD CHECK-HASH 0 vs 0x7c9b00c5
CG 19: BAD CHECK-HASH 0 vs 0x8e58a45d
CG 20: BAD CHECK-HASH 0 vs 0x427a8a1
CG 21: BAD CHECK-HASH 0 vs 0x7b624c0c
CG 22: BAD CHECK-HASH 0 vs 0x4016d045
CG 23: BAD CHECK-HASH 0 vs 0xbeb868d1
CG 24: BAD CHECK-HASH 0 vs 0xbf3550a7
CG 25: BAD CHECK-HASH 0 vs 0xe1748785
CG 26: BAD CHECK-HASH 0 vs 0xacc5837a
CG 27: BAD CHECK-HASH 0 vs 0x397b8e79
CG 28: BAD CHECK-HASH 0 vs 0xef868063
CG 29: BAD CHECK-HASH 0 vs 0x28ab03a9
CG 30: BAD CHECK-HASH 0 vs 0x8c1c2f40
CG 31: BAD CHECK-HASH 0 vs 0x2825b898
CG 32: BAD CHECK-HASH 0 vs 0x57200d54
CG 33: BAD CHECK-HASH 0 vs 0x8ddd9f4e
CG 34: BAD CHECK-HASH 0 vs 0xa25fc59b
CG 35: BAD CHECK-HASH 0 vs 0xbaf45843
CG 36: BAD CHECK-HASH 0 vs 0xb82def37
CG 37: BAD CHECK-HASH 0 vs 0x320cb852
CG 38: BAD CHECK-HASH 0 vs 0xa8d433d9
CG 39: BAD CHECK-HASH 0 vs 0x4fb023bb
CG 40: BAD CHECK-HASH 0 vs 0x7606a0ac
CG 41: BAD CHECK-HASH 0 vs 0xb25d3f34
CG 42: BAD CHECK-HASH 0 vs 0xb43acd7f
CG 43: BAD CHECK-HASH 0 vs 0xd3c1ed7a
CG 44: BAD CHECK-HASH 0 vs 0xf5df5b0d
CG 45: BAD CHECK-HASH 0 vs 0x5f070643
CG 46: BAD CHECK-HASH 0 vs 0x1038a213
CG 47: BAD CHECK-HASH 0 vs 0x5e038a7b
CG 48: BAD CHECK-HASH 0 vs 0xe30f28bb
CG 49: BAD CHECK-HASH 0 vs 0x853827ef
720309 files, 2926458 used, 4686237 free (279357 frags, 550860 blocks, 3.7% fragmentation)
***** FILE SYSTEM IS CLEAN *****
***** FILE SYSTEM WAS MODIFIED *****
Note that the check hashes flags are set below:
# dumpfs /dev/md0 | head -20
dumpfs: /dev/md0: cylinder group checks failed
magic 19540119 (UFS2) time Mon Dec 3 20:36:43 2018
superblock location 65536 id [ 55dce11e c2eee767 ]
ncg 50 size 7864320 blocks 7612695
bsize 32768 shift 15 mask 0xffff8000
fsize 4096 shift 12 mask 0xfffff000
frag 8 shift 3 fsbtodb 3
minfree 8% optim time symlinklen 120
maxbsize 32768 maxbpg 4096 maxcontig 4 contigsumsize 4
nbfree 550860 ndir 11826 nifree 3292489 nffree 279357
bpg 20035 fpg 160280 ipg 80256 unrefs 0
nindir 4096 inopb 128 maxfilesize 2252349704110079
sbsize 4096 cgsize 32768 csaddr 5056 cssize 4096
sblkno 24 cblkno 32 iblkno 40 dblkno 5056
cgrotor 25 fmod 0 ronly 0 clean 1
metaspace 6408 avgfpdir 64 avgfilesize 16384
flags soft-updates
check hashes superblock cylinder-groups
fsmnt /
volname swuid 0 providersize 7864320
Digging into fsck_ffs pass5.c seems to show that the cylinder group
blocks are getting fixed and marked dirty, but I haven't been able to
figure out what actually is supposed to cause them to get written back
to disk.
In the meantime, my vm guest is stuck because the back cylinder group
hashes cause the cylinder groups to be treated as read-only, meaning
that I can't create any new files.
More information about the freebsd-fs
mailing list