16.0E ExpandSize? -- New Server
Larry Rosenman
ler at lerctr.org
Sun Feb 5 03:54:41 UTC 2017
I saw it was accepted upstream. Can it be committed to FreeBSD?
On Wed, Feb 01, 2017 at 02:43:51AM +0000, Steven Hartland wrote:
> Thanks I've put a PR in upstream to get some eyes on the fix.
> https://github.com/openzfs/openzfs/pull/296
>
> If no objections are raised to the approach I've used I'll commit the fix to
> HEAD too.
>
> On 01/02/2017 02:31, Larry Rosenman wrote:
> >
> > no grief that I can see:
> >
> > borg-new /home/ler $ sudo zdb
> > Password:
> > zroot:
> > version: 5000
> > name: 'zroot'
> > state: 0
> > txg: 96143
> > pool_guid: 11945658884309024932
> > hostid: 3619181042
> > hostname: ''
> > com.delphix:has_per_vdev_zaps
> > vdev_children: 1
> > vdev_tree:
> > type: 'root'
> > id: 0
> > guid: 11945658884309024932
> > create_txg: 4
> > children[0]:
> > type: 'raidz'
> > id: 0
> > guid: 7596925654112466913
> > nparity: 1
> > metaslab_array: 42
> > metaslab_shift: 36
> > ashift: 12
> > asize: 11947471798272
> > is_log: 0
> > create_txg: 4
> > com.delphix:vdev_zap_top: 35
> > children[0]:
> > type: 'disk'
> > id: 0
> > guid: 1443238581175429852
> > path: '/dev/mfid4p4'
> > whole_disk: 1
> > DTL: 137
> > create_txg: 4
> > com.delphix:vdev_zap_leaf: 131
> > children[1]:
> > type: 'disk'
> > id: 1
> > guid: 1865792721003775978
> > path: '/dev/mfid0p4'
> > whole_disk: 1
> > DTL: 133
> > create_txg: 4
> > com.delphix:vdev_zap_leaf: 37
> > children[2]:
> > type: 'disk'
> > id: 2
> > guid: 12541720522827927342
> > path: '/dev/mfid1p4'
> > whole_disk: 1
> > DTL: 132
> > create_txg: 4
> > com.delphix:vdev_zap_leaf: 38
> > children[3]:
> > type: 'disk'
> > id: 3
> > guid: 13053934791777776444
> > path: '/dev/mfid3p4'
> > whole_disk: 1
> > DTL: 136
> > create_txg: 4
> > com.delphix:vdev_zap_leaf: 135
> > children[4]:
> > type: 'disk'
> > id: 4
> > guid: 4432707573898874857
> > path: '/dev/mfid2p4'
> > whole_disk: 1
> > DTL: 130
> > create_txg: 4
> > com.delphix:vdev_zap_leaf: 40
> > children[5]:
> > type: 'disk'
> > id: 5
> > guid: 5106293125005422556
> > path: '/dev/mfid5p4'
> > whole_disk: 1
> > DTL: 129
> > create_txg: 4
> > com.delphix:vdev_zap_leaf: 41
> > features_for_read:
> > com.delphix:hole_birth
> > com.delphix:embedded_data
> > borg-new /home/ler $ sudo zpool list -v
> > NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
> > zroot 10.8T 94.3G 10.7T - 0% 0% 1.00x ONLINE -
> > raidz1 10.8T 94.3G 10.7T - 0% 0%
> > mfid4p4 - - - - - -
> > mfid0p4 - - - - - -
> > mfid1p4 - - - - - -
> > mfid3p4 - - - - - -
> > mfid2p4 - - - - - -
> > mfid5p4 - - - - - -
> > borg-new /home/ler $ sudo zpool get all
> > NAME PROPERTY VALUE SOURCE
> > zroot size 10.8T -
> > zroot capacity 0% -
> > zroot altroot - default
> > zroot health ONLINE -
> > zroot guid 11945658884309024932 default
> > zroot version - default
> > zroot bootfs zroot/ROOT/default local
> > zroot delegation on default
> > zroot autoreplace off default
> > zroot cachefile - default
> > zroot failmode wait default
> > zroot listsnapshots off default
> > zroot autoexpand off default
> > zroot dedupditto 0 default
> > zroot dedupratio 1.00x -
> > zroot free 10.7T -
> > zroot allocated 94.3G -
> > zroot readonly off -
> > zroot comment - default
> > zroot expandsize - -
> > zroot freeing 0 default
> > zroot fragmentation 0% -
> > zroot leaked 0 default
> > zroot feature at async_destroy enabled local
> > zroot feature at empty_bpobj active local
> > zroot feature at lz4_compress active local
> > zroot feature at multi_vdev_crash_dump enabled local
> > zroot feature at spacemap_histogram active local
> > zroot feature at enabled_txg active local
> > zroot feature at hole_birth active local
> > zroot feature at extensible_dataset enabled local
> > zroot feature at embedded_data active local
> > zroot feature at bookmarks enabled local
> > zroot feature at filesystem_limits enabled local
> > zroot feature at large_blocks enabled local
> > zroot feature at sha512 enabled local
> > zroot feature at skein enabled local
> > borg-new /home/ler $
> >
> >
> >
> > On 01/31/2017 5:22 pm, Steven Hartland wrote:
> >
> > > Yep
> > >
> > > On 31/01/2017 21:49, Larry Rosenman wrote:
> > > >
> > > > revert the other patch and apply this one?
> > > >
> > > >
> > > >
> > > > On 01/31/2017 3:47 pm, Steven Hartland wrote:
> > > >
> > > > Hmm, looks like there's also a bug in the way vdev_min_asize is
> > > > calculated for raidz as it can and has resulted in child
> > > > min_asize which won't provided enough space for the parent due
> > > > to the use of unrounded integer division.
> > > >
> > > > 1981411579221 * 6 = 11888469475326 < 11888469475328
> > > >
> > > > You should have vdev_min_asize: 1981411579222 for your children.
> > > >
> > > > Updated patch attached, however calculation still isn't 100%
> > > > reversible so may need work, however it does now ensure that the
> > > > children will provide enough capacity for min_asize even if all
> > > > of them are shrunk to their individual min_asize, which I
> > > > believe previously may not have been the case.
> > > >
> > > > This isn't related to the incorrect EXPANDSZ output, but would
> > > > be good if you could confirm it doesn't cause any issues for
> > > > your pool given its state.
> > > >
> > > > On 31/01/2017 21:00, Larry Rosenman wrote:
> > > >
> > > > borg-new /home/ler $ sudo ./vdev-stats.d
> > > > Password:
> > > > vdev_path: n/a, vdev_max_asize: 0, vdev_asize: 0,
> > > > vdev_min_asize: 0
> > > > vdev_path: n/a, vdev_max_asize: 11947471798272, vdev_asize:
> > > > 11947478089728, vdev_min_asize: 11888469475328
> > > > vdev_path: /dev/mfid4p4, vdev_max_asize: 1991245299712,
> > > > vdev_asize: 1991245299712, vdev_min_asize: 1981411579221
> > > > vdev_path: /dev/mfid0p4, vdev_max_asize: 1991246348288,
> > > > vdev_asize: 1991246348288, vdev_min_asize: 1981411579221
> > > > vdev_path: /dev/mfid1p4, vdev_max_asize: 1991246348288,
> > > > vdev_asize: 1991246348288, vdev_min_asize: 1981411579221
> > > > vdev_path: /dev/mfid3p4, vdev_max_asize: 1991247921152,
> > > > vdev_asize: 1991247921152, vdev_min_asize: 1981411579221
> > > > vdev_path: /dev/mfid2p4, vdev_max_asize: 1991246348288,
> > > > vdev_asize: 1991246348288, vdev_min_asize: 1981411579221
> > > > vdev_path: /dev/mfid5p4, vdev_max_asize: 1991246348288,
> > > > vdev_asize: 1991246348288, vdev_min_asize: 1981411579221
> > > > ^C
> > > >
> > > > borg-new /home/ler $
> > > >
> > > >
> > > > borg-new /home/ler $ sudo zpool list -v
> > > > Password:
> > > > NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
> > > > zroot 10.8T 94.3G 10.7T 16.0E 0% 0% 1.00x ONLINE -
> > > > raidz1 10.8T 94.3G 10.7T 16.0E 0% 0%
> > > > mfid4p4 - - - - - -
> > > > mfid0p4 - - - - - -
> > > > mfid1p4 - - - - - -
> > > > mfid3p4 - - - - - -
> > > > mfid2p4 - - - - - -
> > > > mfid5p4 - - - - - -
> > > > borg-new /home/ler $
> > > >
> > > >
> > > > On 01/31/2017 2:37 pm, Steven Hartland wrote:
> > > >
> > > > In that case based on your zpool history I suspect that
> > > > the original mfid4p4 was the same size as mfid0p4
> > > > (1991246348288) but its been replaced with a drive which
> > > > is (1991245299712), slightly smaller.
> > > >
> > > > This smaller size results in a max_asize of
> > > > 1991245299712 * 6 instead of original 1991246348288* 6.
> > > >
> > > > Now given the way min_asize (the value used to check if
> > > > the device size is acceptable) is rounded to the the
> > > > nearest metaslab I believe that replace would be allowed.
> > > > https://github.com/freebsd/freebsd/blob/master/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c#L4947
> > > >
> > > > Now the problem is that on open the calculated asize is
> > > > only updated if its expanding:
> > > > https://github.com/freebsd/freebsd/blob/master/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c#L1424
> > > >
> > > > The updated dtrace file outputs vdev_min_asize which
> > > > should confirm my suspicion about why the replace was
> > > > allowed.
> > > >
> > > > On 31/01/2017 19:05, Larry Rosenman wrote:
> > > >
> > > > I've replaced some disks due to failure, and some of
> > > > the pariition sizes are different.
> > > >
> > > >
> > > > autoexpand is off:
> > > >
> > > > borg-new /home/ler $ zpool get all zroot
> > > > NAME PROPERTY VALUE SOURCE
> > > > zroot size 10.8T -
> > > > zroot capacity 0% -
> > > > zroot altroot - default
> > > > zroot health ONLINE -
> > > > zroot guid 11945658884309024932 default
> > > > zroot version - default
> > > > zroot bootfs zroot/ROOT/default local
> > > > zroot delegation on default
> > > > zroot autoreplace off default
> > > > zroot cachefile - default
> > > > zroot failmode wait default
> > > > zroot listsnapshots off default
> > > > zroot autoexpand off default
> > > > zroot dedupditto 0 default
> > > > zroot dedupratio 1.00x -
> > > > zroot free 10.7T -
> > > > zroot allocated 94.3G -
> > > > zroot readonly off -
> > > > zroot comment - default
> > > > zroot expandsize 16.0E -
> > > > zroot freeing 0 default
> > > > zroot fragmentation 0% -
> > > > zroot leaked 0 default
> > > > zroot feature at async_destroy enabled local
> > > > zroot feature at empty_bpobj active local
> > > > zroot feature at lz4_compress active local
> > > > zroot feature at multi_vdev_crash_dump enabled local
> > > > zroot feature at spacemap_histogram active local
> > > > zroot feature at enabled_txg active local
> > > > zroot feature at hole_birth active local
> > > > zroot feature at extensible_dataset enabled local
> > > > zroot feature at embedded_data active local
> > > > zroot feature at bookmarks enabled local
> > > > zroot feature at filesystem_limits enabled local
> > > > zroot feature at large_blocks enabled local
> > > > zroot feature at sha512 enabled local
> > > > zroot feature at skein enabled local
> > > > borg-new /home/ler $
> > > >
> > > >
> > > > borg-new /home/ler $ gpart show
> > > > => 40 3905945520 mfid0 GPT (1.8T)
> > > > 40 1600 1 efi (800K)
> > > > 1640 1024 2 freebsd-boot (512K)
> > > > 2664 1432 - free - (716K)
> > > > 4096 16777216 3 freebsd-swap (8.0G)
> > > > 16781312 3889162240 4 freebsd-zfs (1.8T)
> > > > 3905943552 2008 - free - (1.0M)
> > > >
> > > > => 40 3905945520 mfid1 GPT (1.8T)
> > > > 40 1600 1 efi (800K)
> > > > 1640 1024 2 freebsd-boot (512K)
> > > > 2664 1432 - free - (716K)
> > > > 4096 16777216 3 freebsd-swap (8.0G)
> > > > 16781312 3889162240 4 freebsd-zfs (1.8T)
> > > > 3905943552 2008 - free - (1.0M)
> > > >
> > > > => 40 3905945520 mfid2 GPT (1.8T)
> > > > 40 1600 1 efi (800K)
> > > > 1640 1024 2 freebsd-boot (512K)
> > > > 2664 1432 - free - (716K)
> > > > 4096 16777216 3 freebsd-swap (8.0G)
> > > > 16781312 3889162240 4 freebsd-zfs (1.8T)
> > > > 3905943552 2008 - free - (1.0M)
> > > >
> > > > => 40 3905945520 mfid3 GPT (1.8T)
> > > > 40 1600 1 efi (800K)
> > > > 1640 1024 2 freebsd-boot (512K)
> > > > 2664 16777216 3 freebsd-swap (8.0G)
> > > > 16779880 3889165680 4 freebsd-zfs (1.8T)
> > > >
> > > > => 40 3905945520 mfid5 GPT (1.8T)
> > > > 40 1600 1 efi (800K)
> > > > 1640 1024 2 freebsd-boot (512K)
> > > > 2664 1432 - free - (716K)
> > > > 4096 16777216 3 freebsd-swap (8.0G)
> > > > 16781312 3889162240 4 freebsd-zfs (1.8T)
> > > > 3905943552 2008 - free - (1.0M)
> > > >
> > > > => 40 3905945520 mfid4 GPT (1.8T)
> > > > 40 1600 1 efi (800K)
> > > > 1640 1024 2 freebsd-boot (512K)
> > > > 2664 1432 - free - (716K)
> > > > 4096 16777216 3 freebsd-swap (8.0G)
> > > > 16781312 3889160192 4 freebsd-zfs (1.8T)
> > > > 3905941504 4056 - free - (2.0M)
> > > >
> > > > borg-new /home/ler $
> > > >
> > > >
> > > > this system was built last week, and I **CAN**
> > > > rebuild it if necessary, but I didn't do anything
> > > > strange (so I thought :) )
> > > >
> > > >
> > > >
> > > >
> > > > On 01/31/2017 12:30 pm, Steven Hartland wrote:
> > > >
> > > > Your issue is the reported vdev_max_asize >
> > > > vdev_asize:
> > > > vdev_max_asize: 11947471798272
> > > > vdev_asize: 11947478089728
> > > >
> > > > max asize is smaller than asize by 6291456
> > > >
> > > > For raidz1 Xsize should be the smallest disk
> > > > Xsize * disks so:
> > > > 1991245299712 * 6 = 11947471798272
> > > >
> > > > So your max asize looks right but asize looks
> > > > too big
> > > >
> > > > Expand Size is calculated by:
> > > > if (vd->vdev_aux == NULL && tvd != NULL &&
> > > > vd->vdev_max_asize != 0) {
> > > > vs->vs_esize = P2ALIGN(vd->vdev_max_asize -
> > > > vd->vdev_asize,
> > > > 1ULL << tvd->vdev_ms_shift);
> > > > }
> > > >
> > > > So the question is why is asize too big?
> > > >
> > > > Given you seem to have some random disk sizes do
> > > > you have auto expand turned on?
> > > >
> > > > On 31/01/2017 17:39, Larry Rosenman wrote:
> > > >
> > > > vdev_path: n/a, vdev_max_asize:
> > > > 11947471798272, vdev_asize: 11947478089728
> > > >
> > > >
> > > > -- Larry Rosenman
> > > > http://people.freebsd.org/~ler
> > > > <http://people.freebsd.org/%7Eler>
> > > > Phone: +1 214-642-9640 E-Mail:
> > > > ler at FreeBSD.org <mailto:ler at FreeBSD.org>
> > > > US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281
> > > >
> > > >
> > > > -- Larry Rosenman http://people.freebsd.org/~ler
> > > > <http://people.freebsd.org/%7Eler>
> > > > Phone: +1 214-642-9640 E-Mail:
> > > > ler at FreeBSD.org <mailto:ler at FreeBSD.org>
> > > > US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281
> > > >
> > > >
> > > > --
> > > > Larry Rosenman http://people.freebsd.org/~ler
> > > > <http://people.freebsd.org/%7Eler>
> > > > Phone: +1 214-642-9640 E-Mail: ler at FreeBSD.org
> > > > <mailto:ler at FreeBSD.org>
> > > > US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281
> >
> >
> > --
> > Larry Rosenman http://people.freebsd.org/~ler
> > <http://people.freebsd.org/%7Eler>
> > Phone: +1 214-642-9640 E-Mail: ler at FreeBSD.org
> > <mailto:ler at FreeBSD.org>
> > US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281
>
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 E-Mail: ler at lerctr.org
US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281
More information about the freebsd-fs
mailing list