NFS poor performance in ipfw_nat

[KIRIYAMA Kazuhiko]

At Tue, 18 Sep 2018 14:02:18 +0300,
Andrey V. Elsukov wrote:
> 
> [1 Re: NFS poor performance in ipfw_nat <multipart/mixed (7bit)>]
> [1.1  <text/plain; utf-8 (quoted-printable)>]
> On 18.09.2018 01:53, KIRIYAMA Kazuhiko wrote:
> > Hi, all
> > 
> > I'm working on ipfw_nat box with port redirect for sunrpc
> > (111) and nfsd (2049):
> > 
> > # ifconfig 
> > em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> >         options=85259b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
> > igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> >         options=e505bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
> > # ipfw nat show config
> > ipfw nat 123 config if em0 log deny_in same_ports unreg_only reset redirect_port tcp 192.168.1.253:22 22253 redirect_port tcp 192.168.1.252:22 22252 redirect_port tcp 192.168.1.251:22 22251 redirect_port tcp 192.168.1.250:22 22250 redirect_port tcp 192.168.1.249:22 22249 redirect_port tcp 192.168.1.248:22 22248 redirect_port tcp 192.168.1.247:22 22247 redirect_port tcp 192.168.1.246:22 22246 redirect_port tcp 192.168.1.245:22 22245 redirect_port tcp 192.168.1.244:22 22244 redirect_port tcp 192.168.1.243:22 22243 > 
> > Is there any suggestions ?
> > 
> 
> Hi,
> 
> try to disable TSO on your NICs.

Thanks for your advice. Add '-lro' and '-tso' to ifconfig,
transfer rate up to almost native NIC speed:

# dd if=/dev/zero of=/.dake/tmp/foo.img bs=1k count=1m
1048576+0 records in
1048576+0 records out
1073741824 bytes transferred in 10.688162 secs (100460852 bytes/sec)
# 

BTW in VM on behyve, transfer rate to NFS mount of VM server
(bhyve) is appreciably low level:

# dd if=/dev/zero of=/.dake/tmp/foo.img bs=1k count=1m
1048576+0 records in
1048576+0 records out
1073741824 bytes transferred in 32.094448 secs (33455687 bytes/sec)
# ifconfig 
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 58:9c:fc:0c:10:58
        inet 192.168.1.193 netmask 0xffffff00 broadcast 192.168.1.255 
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
        inet 127.0.0.1 netmask 0xff000000 
        groups: lo 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
# df
Filesystem                            1K-blocks     Used       Avail Capacity  Mounted on
/dev/vtbd0p2                            5060988  4244412      411700    91%    /
devfs                                         1        1           0   100%    /dev
vm.pis:/.dake                       14773545597 51624777 14721920820     0%    /.dake
vm.pis:/ds/src/current/12.0/r338547 14749697607 27776787 14721920820     0%    /usr/src
vm.pis:/ds/obj/current/12.0/r338547 14772357057 50436237 14721920820     0%    /usr/obj
# cat /etc/rc.conf
hostname="tbedfc"
ifconfig_vtnet0="inet 192.168.1.193 netmask 255.255.255.0"
defaultrouter="192.168.1.254"
sshd_enable="YES"
nfs_client_enable="YES"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
moused_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"
dumpdev="AUTO"
# uname -a
FreeBSD tbedfc 12.0-ALPHA5 FreeBSD 12.0-ALPHA5 #0: Mon Sep 10 22:19:36 JST 2018     root at tbedfc:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
# 

It seems to be unconcerned to TSO or LRO ...

> 
> -- 
> WBR, Andrey V. Elsukov
> 
> [2 OpenPGP digital signature <application/pgp-signature (7bit)>]
> 

---
KIRIYAMA Kazuhiko