jboss4 on freebsd

[Dmitriy]

-----Original Message-----
From: Stefan Lambrev <stefan.lambrev at moneybookers.com>
To: freebsd-net at FreeBSD.org
Date: Tue, 24 Jun 2008 15:58:30 +0300
Subject: jboss4 on freebsd

> 
> Greetings,
> 
> I'm experimenting with jboss4 cluster under freebsd 7 (amd64).
> In my configuration I have 2 jboss instances which are in cluster and 
> they communicate via separate network (used only for shared data)
> When I create some load on the application sometimes I see this error:
> 
> 2008-06-24 14:46:21,602 ERROR [org.jgroups.protocols.UDP] failed sending 
> message to 10.50.1.1:57680 (59800 bytes)
> java.io.IOException: No buffer space available
> 
> It looks very much, that jboss can't handle properly such error as on 
> linux there is no such thing as no network buffers ;) - 
> http://wiki.freebsd.org/AvoidingLinuxisms
> 
> But what really bothers me is that I see "No buffer space available" on 
> very low network IO -
> 
>             input          (em2)           output
>    packets  errs      bytes    packets  errs      bytes colls
>        144     0    2203390        292     0    2072771     0
>       1568     0    2329764         63     0       9099     0
>         76     0     231562         34     0     148306     0
>        563     0    1152531       1009     0    1768748     0
>       1625     0    2601502        104     0     229728     0
>         65     0     467296         85     0     441566     0
>        464     0     680082        973     0    1439442     0
>        357     0    1940361         55     0     222484     0
>       1651     0    2827932        145     0     450265     0
> 
> E.g. traffic between 1-3MB/s.
> 
> I'm using:
> em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
>         options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
>         ether 00:15:17:60:04:c8
>         inet 10.3.3.117 netmask 0xffffff00 broadcast 10.3.3.255
>         media: Ethernet autoselect (1000baseTX <full-duplex>)
>         status: active
> 
> em2: <Intel(R) PRO/1000 Network Connection 6.9.5> port 0x2020-0x203f mem 
> 0xb8820000-0xb883ffff,0xb8400000-0xb87fffff irq 18 at device 0.0 on pci5
> em2: Using MSI interrupt
> em2: [FILTER]
> 
> and my sysctl.conf is:
> kern.maxfiles=65000
> kern.ipc.shmmax=67108864
> kern.fallback_elf_brand=3
> kern.threads.max_threads_per_proc=6000
> kern.ipc.somaxconn=512
> #jboss extra
> net.inet.udp.maxdgram=73728
> kern.ipc.maxsockbuf=1048576
> net.inet.udp.recvspace=147456
> kern.ipc.maxsockets=49312
> 
> Any ideas how I can improve things?
> 

I guess you get the error from the NIC driver. There 2 conditions which can trigger it:
1. if( adapter->num_tx_desc_avail < EM_TX_OP_THRESHOLD )
2. if( nsegs > ( adapter->num_tx_desc_avail - 2 ) )

I have 2 suggestions how to fix your error:
1. try to disable TXCSUM ( #ifconfig em2 -txcsum )
2. increase the number of transfer descriptors available if your hardware supports it:
[man em(4) excerpt]
     Tunables can be set at the loader(8) prompt before booting the kernel or
     stored in loader.conf(5).
     hw.em.txd
             Number of transmit descriptors allocated by the driver.  The
             default value is 256.  The 82542 and 82543-based adapters can
             handle up to 256 descriptors, while others can have up to 4096.
     hw.em.tx_int_delay
             This value delays the generation of transmit interrupts in units
             of 1.024 microseconds.  The default value is 64.

     hw.em.tx_abs_int_delay
             If hw.em.tx_int_delay is non-zero, this tunable limits the maxi-
             mum delay in which a transmit interrupt is generated.
[/man]

Regards,
Dmitriy.