performance tuning of iSCSI and Dell MD3000i

Fri Jan 22 02:05:28 UTC 2010

Daniel Braniss wrote:
> Hi Miroslav,
>> Hi,
>>
>> [1]
>> I am using iSCSI Initiator 2.2.3 (on 7-STABLE amd64) and Dell MD3000i as
>> storage for server distributing downloads of large files (10-200MB)
>> through Lighttpd daemon.
>> Is there any tips for performance tuning?
>>
>> My iscsi.conf is:
>> styx_storage_1 {
>>           initiatorname   = iqn.2005-01.il.ac.huji.cs::dust.example.com
>>           TargetName      =
>> iqn.1984-05.com.dell:powervault.md3000i.60026b900042587b000000004ae58efc
>>           TargetAddress   = 192.168.130.101:3260,1
>>           tags            = 64
>> }
>>
>> Lighttpd is serving about 200-300 client connections at speed 220Mb/s it
>> is about 215Mb/s on iSCSI network interface to MD3000i. (iostat showed
>> 100% busy of da0)
>> Is it really maximum achievable speed on this type of iSCSI storage?
>> (made of 4 SATA drives in RAID5)
>> Or can it be better with some tuned settings?
>>
>
> the limiting factor is mainly the network, I have no experience with the
> MD3000i,
> we have MD3000 and they are very fast, unfourtunately, I can't come up
> with real statistical values, since the raid+zfs are doing very aggresive
> caching and my benchmarks are useless.
> In any case, the values you are getting seem a bit low, I would first measure
> the speed on the server, if your net is GB, then you should expect something
> around 80-100 MB/s.

The network is gigabit. Internet connection is through 3Com 2928 switch, 
connection between server and MD3000i is crosslink with Jumbo Frames 
(9k) enabled on server and storage array too.

Internet NIC:
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 
1500 
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
         ether 00:22:11:64:0d:ff
         inet AA.BB.CC.DD netmask 0xffffff80 broadcast AA.BB.CC.EE
         media: Ethernet autoselect (1000baseTX <full-duplex>)
         status: active

iSCSI NIC:
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 
9000 
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
         ether 00:22:11:64:0e:01
         inet 192.168.130.2 netmask 0xffffff00 broadcast 192.168.130.255
         media: Ethernet autoselect (1000baseTX <full-duplex>)
         status: active

I can achieve higher speed with sequential read test by dd command. With 
dd I can read about 680Mbps (85MB/s), but I think the high concurrency 
workload is more stressing disk than network.

Is there some interesting sysctl which I can post you / tune network 
performance?

>> Have anyone experiences with MD3000i - if I add more disk drives in to
>> RAID5, will it help? How much?
>>
>>
>> [2]
>> Is somewhere final rc script for iscsi initiator which can be used to
>> mount filesystem from iSCSI target at boot time by standard rc
>> procedure? Why is it not included in the base system if iscsi_initiator
>> is there for a long time?
>
> I guess I should work on that :-(

Is the attached version the more recent one? I found it in the list
http://lists.freebsd.org/pipermail/freebsd-stable/2008-July/043611.html

Is it really necessary to run it after normal mount stage, so iSCSI 
devices must be defined in fstab.iscsi and fscked & mounted by 
rc.d/iscsi instead of standard system procedures? (I am just asking as I 
am not so skilled in rc.d scripting)
Did you talk about it with some freebsd-rc@ guys?

>> [3]
>> And one other thing, I got following error logged in messages after 2
>> days of running some performance tests:
> can you check the M3000i logs at that time? or else let me know
> what tests you are doing, so that i can try it here?

There is nothing in the Event Log of MD3000i in the time of this error.
The test was relatively simple. There is Lighttpd runing on the server 
(with server.max-worker = 4 to have higher throughput) and I run 
http_load (www/http_load) from another machine connected to the same 
switch.
http_load was ran in 4 instances with the following settings:

http_load -parallel 70 -seconds 43200 download.url

So it generates 280 parallel connections on to Lighttpd with URLs from 
download.url file.
There is 24376 of lines (URLs to files) in this file.

>> Jan 19 05:36:50 dust kernel:>>>  so_recv: have a problem, error=0
>> Jan 19 05:36:50 dust kernel: isc_in: sp->soc=0xffffff0028f14000
>> Jan 19 05:36:50 dust kernel: 0] isc_in: terminated, flags=39f so_count=2
>> so_state=2000 error=2080 proc=0xffffff0005ab1910
>> Jan 19 05:36:50 dust kernel: 0] isc_in: pid=963750 s]i gin_alfeudl(l3f0e)atu
>> Jan 19 05:36:50 dust kernel: r0e]:  ifslca_gi=n0:
>> Jan 19 05:36:50 dust kernel: dropped ISC_CON_RUNNING
>> Jan 19 05:36:50 dust iscontrol[96375]: trapped signal 30
>> Jan 19 05:36:52 dust kernel: 0] i_send: len=492 ahs_len=0 ds_len=441
>> buf=444 at 0xffffff0007068000
>> Jan 19 05:36:52 dust kernel: 0] i_setopt: maxRecvDataSegmentLength=65536
>> Jan 19 05:36:52 dust kernel: 0] i_setopt: opt.maXmitDataSegmentLength=65536
>> Jan 19 05:36:52 dust kernel: 0] i_setopt: opt.maxBurstLength=131072
>> Jan 19 05:36:52 dust kernel: 0] i_setopt:
>> opt.targetAddress='192.168.130.101'
>> Jan 19 05:36:52 dust kernel: 0] i_setopt:
>> opt.targetName='iqn.1984-05.com.dell:powervault.md3000i.60026b900042587b000000004ae58efc'
>> Jan 19 05:36:52 dust kernel: 0] i_setopt:
>> opt.initiatorName='iqn.2005-01.il.ac.huji.cs::dust.example.com'
>> Jan 19 05:36:52 dust kernel: 0] i_setopt: opt.headerDigest='None'
>> Jan 19 05:36:52 dust kernel: 0] i_setopt: opt.dataDigest='None'
>> Jan 19 05:36:52 dust kernel: 0] i_fullfeature: flag=2
>> Jan 19 05:36:52 dust kernel: 0] ism_restart: restart ...
>> Jan 19 05:36:52 dust kernel: 0] iscsi_requeue: last=30a30e4 n=30a30e4
>> Jan 19 05:36:52 dust kernel: 0] iscsi_requeue: last=30a30e4 n=30a30e5
[...]
>> Jan 19 05:36:52 dust kernel: 0] iscsi_requeue: last=30a30e4 n=30a312a
>> Jan 19 05:36:52 dust kernel: 0] ism_restart: restarted sn.cmd=0x30a312b
>> lastcmd=0x30a30e4
>>
>> Then transfer of files continued at slightly lower speed than before.
>
> it seems that there was a network glitch, but it recovered.

It is connected with direct crosslink cable and there is not "link state 
changed to DOWN / UP" message for the NIC so I don't know what network 
problem can happened.

Miroslav Lachman
-------------- next part --------------
#!/bin/sh

# PROVIDE: iscsi
# REQUIRE: NETWORKING
# BEFORE:  DAEMON
# KEYWORD: nojail shutdown

#
# Add the following lines to /etc/rc.conf to enable iscsi:
#
# iscsi_enable="YES"
# iscsi_fstab="/etc/fstab.iscsi"

. /etc/rc.subr
. /cs/share/etc/rc.subr

name=iscsi
rcvar=`set_rcvar`

command=/sbin/iscontrol

iscsi_enable=${iscsi_enable:-"NO"}
iscsi_fstab=${iscsi_fstab:-"/etc/fstab.iscsi"}
iscsi_exports=${iscsi_exports:-"/etc/exports.iscsi"}
iscsi_debug=${iscsi_debug:-0}
start_cmd="iscsi_start"
faststop_cmp="iscsi_stop"
stop_cmd="iscsi_stop"

start_precmd="iscontrol_precmd"
iscontrol_prog=${iscontrol_prog:-"iscontrol"}
iscontrol_log=${iscontrol_log:-"/var/log/$iscontrol_prog"}
iscontrol_syslog=${iscontrol_syslog:-"644  3	 100  *	 JC"}

iscontrol_precmd()
{
	setup_syslog "$iscontrol_prog" "$iscontrol_log" "$iscontrol_syslog"
}

iscsi_wait()
{
	dev=$1
	trap "echo 'wait loop cancelled'; exit 1" 2
	count=0
	while true; do
		if [ -c $dev ]; then
			break;
		fi
		if [ $count -eq 0 ]; then
			echo -n Waiting for ${dev}': '
		fi
		count=$((${count} + 1))
		if [ $count -eq 6 ]; then
			echo " Failed for dev=$dev"
			return 0
			break
		fi
		echo -n '.'
		sleep 5;
	done
	echo "$dev ok."
	return 1
}

iscsi_start()
{
	#
	# load needed modules
	for m in iscsi_initiator geom_label; do
	kldstat -qm $m || kldload $m
	done

	sysctl debug.iscsi_initiator=$iscsi_debug
	#
	# start iscontrol for each target
	if [ -n "${iscsi_targets}" ]; then
	for target in ${iscsi_targets}; do
		${command} ${rc_flags} -n ${target}
	done
	fi

	if [ -f "${iscsi_fstab}" ]; then
	while read spec file type opt t1 t2
	do
		case ${spec} in
		\#*|'')
		;;
		*)
			if iscsi_wait ${spec}; then
			break;
		fi
		echo type=$type spec=$spec file=$file
		fsck -p ${spec} && mkdir -p ${file} && mount ${spec} ${file}
		chmod 755 ${file} 
		;;
		esac
	done < ${iscsi_fstab} 
	fi

	if [ -f "${iscsi_exports}" ]; then
	cat ${iscsi_exports} >> /etc/exports
	#/etc/rc.d/mountd reload
	kill -1 `cat /var/run/mountd.pid`
	fi
}

iscsi_stop()
{
	echo 'iscsi stopping'
	while read spec file type opt t1 t2
	do
		case ${spec} in
		\#*|'')
		;;
		*)
			echo iscsi: umount $spec
			umount -fv $spec
		;;
		esac
	 done < ${iscsi_fstab} 
}

load_rc_config $name
run_rc_command "$1"