Network sysctl tuning [was Re: ZFSKnownProblems - needs revision?]

[Kip Macy]

I think most if not all of the gains are from increasing the maximum
tcp socket buffer sizes.

You might test it out with only those to confirm.

-Kip



On Thu, Apr 9, 2009 at 3:42 PM, Freddie Cash <fjwcash at gmail.com> wrote:
> On Wed, Apr 8, 2009 at 3:55 PM, Antony Mawer <fbsd-stable at mawer.org> wrote:
>> Freddie Cash wrote:
>> ...
>>> We've also heavily modified /etc/sysctl.conf and upped a bunch of the
>>> network-related sysctls.  Doing so increased our SSH throughput from ~30
>>> Mbits/sec across all connections to over 90 Mbits/sec per SSH connection.
>>
>> Are you able to share any of these with the list? It would be useful to
>> compare as a lot of these tunings people do individually and it would be
>> good to allow others to test in their environments to see if they help, as
>> well as potentially adding them to the tuning man-page.
>
> They're all taken from the HPN-SSH website and various google searches
> related to HPN-enabled OpenSSH.
>
> I don't know exactly what all the different, individual sysctls do,
> nor whether this is the most optimal setup, but here's the sysctl.conf
> that we use.  This is on 2 systems using a quad-port gigabit NIC where
> the top two ports are connected via lagg(4) and the bottom two ports
> are connected via lagg(4), with the two laggX interfaces on separate
> networks.
>
> I did a bunch of scp/sftp transfers of 100 MB files filled with random
> data pulled from /dev/random between these two boxes tweaking the
> options one at a time, but didn't do too much in the way of
> scientific/empirical measurements and comparisons beyond the
> throughput data that scp/sftp shows.
>
> If there are any glaring errors, gotchas, or "why would you ever do
> that"s, let me know.  :)
>
> # General network settings
> net.isr.direct=1                        # Whether to enable Direct
> Dispatch for netisr
>
>
> # IP options
> net.inet.ip.forwarding=0                # Whether to enable packet
> forwarding for NAT/routing
> net.inet.ip.process_options=0           # Disable processing of IP
> options (nothing uses this field)
> net.inet.ip.random_id=1                 # Randomise the IP header ID number
> net.inet.ip.redirect=0                  # Whether to allow redirect packets
> #net.inet.ip.stealth=0                  # Whether to appear in traceroute output
>
>
> # ICMP options
> net.inet.icmp.icmplim=200               # Limit ICMP packets to this
> many per second
> net.inet.icmp.drop_redirect=1           # Drop ICMP redirect packets
> net.inet.icmp.log_redirect=0            # Don't log ICMP redirect packets
>
>
> # TCP options
> net.inet.tcp.blackhole=1                # Drop packets destined to unused ports
> net.inet.tcp.inflight.enable=0          # Use automatic TCP window-scaling
> net.inet.tcp.log_in_vain=0              # Don't log the blackholed packets
> net.inet.tcp.path_mtu_discovery=1       # Use ICMP type 3 to find the MTU to use
> net.inet.tcp.recvbuf_max=16777216       # The max size of the receive
> buffer (16 MB)
> net.inet.tcp.recvspace=131072           # The initial size in bytes of
> the receive buffer
> net.inet.tcp.sack.enable=1              # Enable Selective ACKs
> net.inet.tcp.sendbuf_max=16777216       # The max size of the send buffer
> net.inet.tcp.sendspace=131072           # The initial size in bytes of
> the send buffer
> net.inet.tcp.syncookies=1               # Enable SYN cookie protection
> net.inet.tcp.rfc1323=1                  # Enable RFC1323 extensions
> (TCP window scaling)
>
>
> # UDP options
> net.inet.udp.blackhole=1                # Drop packets destined to unused ports
> net.inet.udp.checksum=1                 # Enable UDP checksums
> net.inet.udp.log_in_vain=0              # Don't log the blackholed packets
> net.inet.udp.recvspace=65536            # Size in bytes of the receive buffer
>
>
> # Debug options
> debug.minidump=1                        # Disable the small kernel
> core dump (only mem in use)
> debug.mpsafevfs=1                       # Enable threaded VFS subsystem
>
>
> # Kernel options
> kern.coredump=0                         # Disable kernel core dumps
> kern.ipc.maxsockbuf=4194304             # Set the max size of the
> socket buffers (4 MB)
> kern.ipc.somaxconn=1024                 # Expand the IP listen queue
> kern.maxvnodes=250000                   # Bump up the max number of vnodes
>
>
> # PCI bus options
> hw.pci.enable_msix=1                    # Enable Message Signalled
> Interrupts - Extended
> hw.pci.enable_msi=1                     # Enable Message Signalled Interrupts
> hw.pci.enable_io_modes=1                # Enable alternate I/O access modes
>
> --
> Freddie Cash
> fjwcash at gmail.com
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>



-- 
All that is necessary for the triumph of evil is that good men do nothing.
    Edmund Burke