[Bug 244899] zfs: xattr on a symlink target > 136 causes "bad file descriptor" (on 12.1) and panic on (13 CURRENT) in sa_build_index()

From: <bugzilla-noreply_at_freebsd.org>
Date: Sat, 22 Oct 2022 14:15:04 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244899

--- Comment #8 from Sean Champ <lab+bsd@thinkum.space> ---
For what it's worth, I started seeing this error again, for a significant
number of port builds, using zfs in poudriere. Example from an llvm13 build,
during the build-depends stage:
~~~~
===>   llvm13-13.0.1_3 depends on executable: ninja - not found
===>   Installing existing package /packages/All/ninja-1.11.1,2.pkg
[xmin.bld.cloud.thinkum.space] Installing ninja-1.11.1,2...
[xmin.bld.cloud.thinkum.space] `-- Installing python38-3.8.15...
[xmin.bld.cloud.thinkum.space] |   `-- Installing libffi-3.4.3...
[xmin.bld.cloud.thinkum.space] |   `-- Extracting libffi-3.4.3: .......... done
[xmin.bld.cloud.thinkum.space] |   `-- Installing mpdecimal-2.5.1...
[xmin.bld.cloud.thinkum.space] |   `-- Extracting mpdecimal-2.5.1: ..........
done
[xmin.bld.cloud.thinkum.space] |   `-- Installing readline-8.1.2...
[xmin.bld.cloud.thinkum.space] |   `-- Extracting readline-8.1.2: ..........
done
[xmin.bld.cloud.thinkum.space] `-- Extracting python38-3.8.15: ...
pkg-static: Fail to chown
/usr/local/lib/python3.8/idlelib/idle_test/.pkgtemp.test_squeezer.py.bFIrNzt0LFou:Bad
file descriptor
~~~~

uname in this instance:
~~~~
FreeBSD xmin.cloud.thinkum.space 13.1-STABLE FreeBSD 13.1-STABLE #0
build/stable/13-n252436-b63021e001d: Sun Oct  9 05:52:29 PDT 2022    
gimbal@xmin.cloud.thinkum.space:/usr/obj/xmin_FreeBSD-13.1-STABLE_amd64/usr/src/amd64.amd64/sys/XMIN
amd64
~~~~

i.e kernel was built from changeset: b63021e001d in the stable/13 branch. 

If it's not an absurd question, could this be related to to the number of open
file descriptors? 

At the time of the errors - with some hundred+ similar build failures while
using zfs in poudriere - I'd noticed that the single user gvfsd-trash process
had approx 10603 file descriptors showing under 'fstat -p'. I've terminated
this process and tried the poudriere build again. Those errors as I was seeing
here are not showing up now.

As a workaround, I'm going to use the following to try to disable gvfsd-trash
in effect:

~~~~
mkdir -p /usr/local/etc/gvfs/mounts
for F in /usr/local/share/gvfs/mounts/*.mount; do
    install -l rs $F /usr/local/etc/gvfs/mounts/$(basename $F);
done
rm /usr/local/etc/gvfs/mounts/trash.mount
~~~~

then in /etc/profile
~~~~
if [ -e /usr/local/etc/gvfs/mounts ]; then
  export GVFS_MOUNTABLE_DIR=/usr/local/etc/gvfs/mounts
fi
~~~~

Of course, this would not guarantee that /usr/local/etc/gvfs/mounts would stay
in sync with /usr/local/share/gvfs/mounts/ then. Alternately, one could patch
the devel/gvfs port to add an option to remove trash.mount before packaging

This correlation may not illustrate a causal relation - e.g between the number
of file descriptors in the gvfsd-trash process and the build failures under
poudriere with ZFS. After closing then disabling gvfsd-trash, I'm not seeing
those failures now, in the local ports build.

I'll try rebuilding the kernel with those options. My kernconf from sysctl
kern.conftxt:
~~~~
kern.conftxt: options   CONFIG_AUTOGENERATED
ident   XMIN
machine amd64
cpu     HAMMER
cpu     HAMMER
cpu     HAMMER
makeoptions     MODULES_EXTRA=acpi/acpi_rapidstart
makeoptions     WITH_CTF=1
makeoptions     DEBUG=-g
options IPI_PREEMPTION
options MSDOSFS_ICONV
options CD9660_ICONV
options LIBICONV
options FDESCFS
options FUSEFS
options GEOM_PART_VTOC8
options GEOM_PART_LDM
options GEOM_PART_BSD64
options GEOM_PART_APM
options X86BIOS
options ENABLE_ALART
options DUMMYNET
options VT_ALT_TO_ESC_HACK=1
options MSGBUF_SIZE=(32*PAGE_SIZE)
options PANIC_REBOOT_WAIT_TIME=-1
options HZ=1000
options IICHID_SAMPLING
options HID_DEBUG
options EVDEV_SUPPORT
options XENHVM
options USB_DEBUG
options ATH_ENABLE_11N
options AH_AR5416_INTERRUPT_MITIGATION
options IEEE80211_SUPPORT_MESH
options IEEE80211_DEBUG
options SC_PIXEL_MODE
options VESA
options PPS_SYNC
options COMPAT_LINUXKPI
options PCI_IOV
options PCI_HP
options IOMMU
options EARLY_AP_STARTUP
options SMP
options NETGDB
options NETDUMP
options DEBUGNET
options ZSTDIO
options GZIO
options EKCD
options KDB_TRACE
options KDB
options RCTL
options RACCT_DEFAULT_TO_DISABLED
options RACCT
options INCLUDE_CONFIG_FILE
options DDB_CTF
options KDTRACE_HOOKS
options KDTRACE_FRAME
options MAC
options CAPABILITIES
options CAPABILITY_MODE
options AUDIT
options KBD_INSTALL_CDEV
options PRINTF_BUFR_SIZE=128
options _KPOSIX_PRIORITY_SCHEDULING
options SYSVSEM
options SYSVMSG
options SYSVSHM
options STACK
options KTRACE
options SCSI_DELAY=5000
options COMPAT_FREEBSD12
options COMPAT_FREEBSD11
options COMPAT_FREEBSD10
options COMPAT_FREEBSD9
options COMPAT_FREEBSD7
options COMPAT_FREEBSD6
options COMPAT_FREEBSD5
options COMPAT_FREEBSD4
options COMPAT_FREEBSD32
options EFIRT
options GEOM_LABEL
options GEOM_RAID
options TMPFS
options PSEUDOFS
options PROCFS
options CD9660
options MSDOSFS
options NFS_ROOT
options NFSLOCKD
options NFSD
options NFSCL
options MD_ROOT
options QUOTA
options UFS_GJOURNAL
options UFS_DIRHASH
options UFS_ACL
options SOFTUPDATES
options FFS
options KERN_TLS
options SCTP_SUPPORT
options TCP_RFC7413
options TCP_HHOOK
options TCP_BLACKBOX
options TCP_OFFLOAD
options FIB_ALGO
options ROUTE_MPATH
options IPSEC_SUPPORT
options INET6
options INET
options VIMAGE
options PREEMPTION
options NUMA
options SCHED_ULE
options NEW_PCIB
options GEOM_PART_GPT
options GEOM_PART_MBR
options GEOM_PART_EBR
options GEOM_PART_BSD
options GEOM_PART_BSD
options GEOM_PART_EBR
options GEOM_PART_MBR
options GEOM_PART_GPT
options NEW_PCIB
options SCHED_ULE
options NUMA
options PREEMPTION
options VIMAGE
options INET
options INET6
options IPSEC_SUPPORT
options ROUTE_MPATH
options FIB_ALGO
options TCP_OFFLOAD
options TCP_BLACKBOX
options TCP_HHOOK
options TCP_RFC7413
options SCTP_SUPPORT
options KERN_TLS
options FFS
options SOFTUPDATES
options UFS_ACL
options UFS_DIRHASH
options UFS_GJOURNAL
options QUOTA
options MD_ROOT
options NFSCL
options NFSD
options NFSLOCKD
options NFS_ROOT
options MSDOSFS
options CD9660
options PROCFS
options PSEUDOFS
options TMPFS
options GEOM_RAID
options GEOM_LABEL
options EFIRT
options COMPAT_FREEBSD32
options COMPAT_FREEBSD4
options COMPAT_FREEBSD5
options COMPAT_FREEBSD6
options COMPAT_FREEBSD7
options COMPAT_FREEBSD9
options COMPAT_FREEBSD10
options COMPAT_FREEBSD11
options COMPAT_FREEBSD12
options SCSI_DELAY=5000
options KTRACE
options STACK
options SYSVSHM
options SYSVMSG
options SYSVSEM
options _KPOSIX_PRIORITY_SCHEDULING
options PRINTF_BUFR_SIZE=128
options KBD_INSTALL_CDEV
options AUDIT
options CAPABILITY_MODE
options CAPABILITIES
options MAC
options KDTRACE_FRAME
options KDTRACE_HOOKS
options DDB_CTF
options INCLUDE_CONFIG_FILE
options RACCT
options RACCT_DEFAULT_TO_DISABLED
options RCTL
options KDB
options KDB_TRACE
options EKCD
options GZIO
options ZSTDIO
options DEBUGNET
options NETDUMP
options NETGDB
options SMP
options EARLY_AP_STARTUP
options IOMMU
options PCI_HP
options PCI_IOV
options COMPAT_LINUXKPI
options PPS_SYNC
options VESA
options SC_PIXEL_MODE
options IEEE80211_DEBUG
options IEEE80211_SUPPORT_MESH
options AH_AR5416_INTERRUPT_MITIGATION
options ATH_ENABLE_11N
options USB_DEBUG
options XENHVM
options EVDEV_SUPPORT
options HID_DEBUG
options IICHID_SAMPLING
options SCHED_ULE
options NUMA
options PREEMPTION
options VIMAGE
options INET
options INET6
options IPSEC_SUPPORT
options ROUTE_MPATH
options FIB_ALGO
options TCP_OFFLOAD
options TCP_BLACKBOX
options TCP_HHOOK
options TCP_RFC7413
options SCTP_SUPPORT
options KERN_TLS
options FFS
options SOFTUPDATES
options UFS_ACL
options UFS_DIRHASH
options UFS_GJOURNAL
options QUOTA
options MD_ROOT
options NFSCL
options NFSD
options NFSLOCKD
options NFS_ROOT
options MSDOSFS
options CD9660
options PROCFS
options PSEUDOFS
options TMPFS
options GEOM_RAID
options GEOM_LABEL
options EFIRT
options COMPAT_FREEBSD32
options COMPAT_FREEBSD4
options COMPAT_FREEBSD5
options COMPAT_FREEBSD6
options COMPAT_FREEBSD7
options COMPAT_FREEBSD9
options COMPAT_FREEBSD10
options COMPAT_FREEBSD11
options COMPAT_FREEBSD12
options SCSI_DELAY=5000
options KTRACE
options STACK
options SYSVSHM
options SYSVMSG
options SYSVSEM
options _KPOSIX_PRIORITY_SCHEDULING
options PRINTF_BUFR_SIZE=128
options KBD_INSTALL_CDEV
options AUDIT
options CAPABILITY_MODE
options CAPABILITIES
options MAC
options KDTRACE_FRAME
options KDTRACE_HOOKS
options DDB_CTF
options INCLUDE_CONFIG_FILE
options RACCT
options RACCT_DEFAULT_TO_DISABLED
options RCTL
options KDB
options KDB_TRACE
options EKCD
options GZIO
options ZSTDIO
options DEBUGNET
options NETDUMP
options NETGDB
options SMP
options EARLY_AP_STARTUP
options IOMMU
options PCI_HP
options PCI_IOV
options COMPAT_LINUXKPI
options PPS_SYNC
options VESA
options SC_PIXEL_MODE
options IEEE80211_DEBUG
options IEEE80211_SUPPORT_MESH
options AH_AR5416_INTERRUPT_MITIGATION
options ATH_ENABLE_11N
options USB_DEBUG
options XENHVM
options EVDEV_SUPPORT
options HID_DEBUG
options IICHID_SAMPLING
options HZ=1000
options PANIC_REBOOT_WAIT_TIME=-1
options MSGBUF_SIZE=(32*PAGE_SIZE)
options HZ=1000
device  isa
device  mem
device  io
device  uart_ns8250
device  cpufreq
device  acpi
device  smbios
device  pci
device  fdc
device  ahci
device  ata
device  mvs
device  siis
device  ahc
device  ahd
device  esp
device  hptiop
device  isp
device  mpt
device  mps
device  mpr
device  sym
device  isci
device  ocs_fc
device  pvscsi
device  scbus
device  ch
device  da
device  sa
device  cd
device  pass
device  ses
device  amr
device  arcmsr
device  ciss
device  iir
device  ips
device  mly
device  twa
device  smartpqi
device  tws
device  aac
device  aacp
device  aacraid
device  ida
device  mfi
device  mlx
device  mrsas
device  pmspcv
device  twe
device  nvme
device  nvd
device  vmd
device  atkbdc
device  atkbd
device  psm
device  kbdmux
device  vga
device  splash
device  sc
device  vt
device  vt_vga
device  vt_efifb
device  vt_vbefb
device  agp
device  cbb
device  pccard
device  cardbus
device  uart
device  ppc
device  ppbus
device  lpt
device  ppi
device  puc
device  iflib
device  em
device  igc
device  ix
device  ixv
device  ixl
device  iavf
device  ice
device  vmx
device  axp
device  bxe
device  le
device  ti
device  mlx5
device  mlxfw
device  mlx5en
device  miibus
device  ae
device  age
device  alc
device  ale
device  bce
device  bfe
device  bge
device  cas
device  dc
device  et
device  fxp
device  gem
device  jme
device  lge
device  msk
device  nfe
device  nge
device  re
device  rl
device  sge
device  sis
device  sk
device  ste
device  stge
device  vge
device  vr
device  xl
device  wlan
device  wlan_wep
device  wlan_ccmp
device  wlan_tkip
device  wlan_amrr
device  an
device  ath
device  ath_pci
device  ath_hal
device  ath_rate_sample
device  ipw
device  iwi
device  iwn
device  malo
device  mwl
device  ral
device  wpi
device  crypto
device  aesni
device  loop
device  padlock_rng
device  rdrand_rng
device  ether
device  vlan
device  tuntap
device  md
device  gif
device  firmware
device  xz
device  bpf
device  uhci
device  ohci
device  ehci
device  xhci
device  usb
device  ukbd
device  umass
device  sound
device  snd_cmi
device  snd_csa
device  snd_emu10kx
device  snd_es137x
device  snd_hda
device  snd_ich
device  snd_via8233
device  mmc
device  mmcsd
device  sdhci
device  rtsx
device  virtio
device  virtio_pci
device  vtnet
device  virtio_blk
device  virtio_scsi
device  virtio_balloon
device  kvm_clock
device  hyperv
device  xenpci
device  netmap
device  evdev
device  uinput
device  hid
device  smbus
device  smb
device  intpm
device  imcsmb
device  ipmi
device  nvram
device  dpms
device  atpic
device  mptable
device  acpi_hp
~~~

This kernel build had also used the following, in files locally included under
/usr/src/sys/amd64/conf/

~~~~
nooptions       WITNESS
nooptions       WITNESS_KDB
nooptions       WITNESS_SKIPSPIN
nooptions       LOCK_PROFILING
nooptions       CALLOUT_PROFILING
nooptions       SLEEPQUEUE_PROFILING
nooptions       TURNSTILE_PROFILING
nooptions       UMTX_PROFILING
nooptions       MBUF_PROFILING
nooptions       INVARIANTS
nooptions       INVARIANT_SUPPORT
~~~~

I'll try adding the options recommended above
~~~~
makeoptions     BUILD_OPTIMIZED=NO
makeoptions     COPTFLAGS=-O0
~~~~

-- 
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.