Device timeouts(?) with LSI SAS3008 on mpr(4)

[Steven Hartland]

I assume da0 and da1 are a different disk then?

With regards your disk setup are all of you disks SSD's if so why do you 
have separate log and cache devices?

One thing you could try is to limit the delete size.

kern.geom.dev.delete_max_sectors limits the single request size allowed 
by geom but then individual requests can be built back up in cam so I 
don't think this will help you too much.

Instead I would try limiting the individual device delete_max, so add 
one line per disk into /boot/loader.conf of the form:
kern.cam.da.X.delete_max=1073741824

You can actually change these on the fly using sysctl, but in order to 
catch an cleanup done on boot loader.conf is the best place to tune them 
permanently.

I've attached a little c util which you can use to do direct disk 
deletes if you have a spare disk you can play with.

Be aware that most controller optimise delete's out if they know the 
cells are empty hence you do need to have written data to the sectors 
each time you test a delete.

As the requests go through geom anything over 
kern.geom.dev.delete_max_sectors will be split but then may well be 
recombined in CAM.

Another relevant setting is vfs.zfs.vdev.trim_max_active which can be 
used to limit the number of outstanding geom delete requests to the each 
device.

Oh one other thing, it would be interesting to see the output from 
camcontrol identify <device> e.g.
camcontrol identify da8
camcontrol identify da0

     Regards
     Steve

On 13/07/2015 10:25, Yamagi Burmeister wrote:
> On Mon, 13 Jul 2015 10:13:32 +0100
> Steven Hartland <killing at multiplay.co.uk> wrote:
>
>> What do you see from:
>> sysctl -a | grep -E '(delete|trim)'
> % sysctl -a | grep -E '(delete|trim)'
> kern.geom.dev.delete_max_sectors: 262144
> kern.cam.da.1.delete_max: 8589803520
> kern.cam.da.1.delete_method: ATA_TRIM
> kern.cam.da.8.delete_max: 12884705280
> kern.cam.da.8.delete_method: ATA_TRIM
> kern.cam.da.9.delete_max: 12884705280
> kern.cam.da.9.delete_method: ATA_TRIM
> kern.cam.da.3.delete_max: 12884705280
> kern.cam.da.3.delete_method: ATA_TRIM
> kern.cam.da.12.delete_max: 12884705280
> kern.cam.da.12.delete_method: ATA_TRIM
> kern.cam.da.7.delete_max: 12884705280
> kern.cam.da.7.delete_method: ATA_TRIM
> kern.cam.da.2.delete_max: 12884705280
> kern.cam.da.2.delete_method: ATA_TRIM
> kern.cam.da.11.delete_max: 12884705280
> kern.cam.da.11.delete_method: ATA_TRIM
> kern.cam.da.6.delete_max: 12884705280
> kern.cam.da.6.delete_method: ATA_TRIM
> kern.cam.da.10.delete_max: 12884705280
> kern.cam.da.10.delete_method: ATA_TRIM
> kern.cam.da.5.delete_max: 12884705280
> kern.cam.da.5.delete_method: ATA_TRIM
> kern.cam.da.0.delete_max: 8589803520
> kern.cam.da.0.delete_method: ATA_TRIM
> kern.cam.da.4.delete_max: 12884705280
> kern.cam.da.4.delete_method: ATA_TRIM
> vfs.zfs.trim.max_interval: 1
> vfs.zfs.trim.timeout: 30
> vfs.zfs.trim.txg_delay: 32
> vfs.zfs.trim.enabled: 1
> vfs.zfs.vdev.trim_max_pending: 10000
> vfs.zfs.vdev.bio_delete_disable: 0
> vfs.zfs.vdev.trim_max_active: 64
> vfs.zfs.vdev.trim_min_active: 1
> vfs.zfs.vdev.trim_on_init: 1
> kstat.zfs.misc.arcstats.deleted: 289783817
> kstat.zfs.misc.zio_trim.failed: 431
> kstat.zfs.misc.zio_trim.unsupported: 0
> kstat.zfs.misc.zio_trim.success: 6457142235
> kstat.zfs.misc.zio_trim.bytes: 88207753330688
>
>
>> Also while your seeing time-outs what does the output from gstat -d -p
>> look like?
> I'll try to get that data but it may take a while.
>
> Thank you,
> Yamagi
>

-------------- next part --------------
#include <stdio.h>
#include <stdlib.h>
#include <sys/ioctl.h>
#include <err.h>
#include <errno.h>
#include <fcntl.h>
#include <sys/disk.h>
#include <sys/time.h>
#include <libutil.h>
#include <unistd.h>

void
syntax()
{
	fprintf(stderr,"ioctl-delete <device> <startblock> <blockcount>\n");
	exit(1);
}


double
timediff(const struct timeval *t1, const struct timeval *t2)
{
    double ret;

    ret = t2->tv_sec - t1->tv_sec;
    ret += (t2->tv_usec - t1->tv_usec) * 0.000001;

    return ret;
}

int
main(int argc, char** argv)
{
	off_t ioarg[2];
	int fd;
	off_t offset, size;
	char *device;
	struct timeval start, end;
	double tdiff;
	char buf[8];
	uint64_t bsec;
	unsigned int sector_size = 512;

	if (4 != argc)
		syntax();

	device = argv[1];
	offset = strtoul(argv[2], NULL, 10);
	size = strtoul(argv[3], NULL, 10);

	fprintf(stderr, "deleting: %ld, %ld\n", offset, size);

	if ((fd = open(device, O_RDWR)) < 0)
		err(1, "device '%s' not found", device);

	if (ioctl(fd, DIOCGSECTORSIZE, &sector_size) <0)
		err(1, "delete failed");

	ioarg[0] = offset * sector_size;
	ioarg[1] = size * sector_size;

	gettimeofday(&start, NULL);
	if (ioctl(fd, DIOCGDELETE, ioarg) <0)
		err(1, "delete failed");
	gettimeofday(&end, NULL);
	tdiff = timediff(&start, &end);

	bsec = (int64_t)((long double)ioarg[1] / tdiff);
	humanize_number(buf, sizeof(buf), bsec, "/s", HN_AUTOSCALE, HN_B | HN_NOSPACE | HN_DECIMAL);

	printf("deleted %lu bytes in %f seconds, %ld bytes per second (%s)\n",
		ioarg[1], tdiff, bsec, buf);

	close(fd);
	exit(1);
}