9.2 ixgbe tx queue hang

[Markus Gebert]

On 24.03.2014, at 16:21, Christopher Forgeron <csforgeron at gmail.com> wrote:

> This is regarding the TSO patch that Rick suggested earlier. (With many
> thanks for his time and suggestion)
> 
> As I mentioned earlier, it did not fix the issue on a 10.0 system. It did
> make it less of a problem on 9.2, but either way, I think it's not needed,
> and shouldn't be considered as a patch for testing/etc.
> 
> Patching TSO to anything other than a max value (and by default the code
> gives it IP_MAXPACKET) is confusing the matter, as the packet length
> ultimately needs to be adjusted for many things on the fly like TCP
> Options, etc. Using static header sizes won't be a good idea.
> 
> Additionally, it seems that setting nic TSO will/may be ignored by code
> like this in sys/netinet/tcp_output.c:
> 
> 10.0 Code:
> 
>  780                         if (len > tp->t_tsomax - hdrlen)
> {                 !!
>  781                                 len = tp->t_tsomax -
> hdrlen;               !!
>  782                                 sendalot =
> 1;
>  783                         }
> 
> 
> I've put debugging here, set the nic's max TSO as per Rick's patch ( set to
> say 32k), and have seen that tp->t_tsomax == IP_MAXPACKET. It's being set
> someplace else, and thus our attempts to set TSO on the nic may be in vain.
> 
> It may have mattered more in 9.2, as I see the code doesn't use
> tp->t_tsomax in some locations, and may actually default to what the nic is
> set to.
> 
> The NIC may still win, I didn't walk through the code to confirm, it was
> enough to suggest to me that setting TSO wouldn't fix this issue.


I just applied Rick’s ixgbe TSO patch and additionally wanted to be able to easily change the value of hw_tsomax, so I made a sysctl out of it.

While doing that, I asked myself the same question. Where and how will this value actually be used and how comes that tcp_output() uses that other value in struct tcpcb.

The only place tcpcb->t_tsomax gets set, that I have found so far, is in tcp_input.c’s tcp_mss() function. Some subfunctions get called:

tcp_mss() -> tcp_mss_update() -> tcp_maxmtu()

Then tcp_maxmtu() indeed uses the interface’s hw_tsomax value:

1746                                 cap->tsomax = ifp->if_hw_tsomax;

It get’s passed back to tcp_mss() where it is set on the  connection level which will be used in tcp_output() later on.

tcp_mss() gets called from multiple places, I’ll look into that later. I will let you know if I find out more.


Markus


> However, this is still a TSO related issue, it's just not one related to
> the setting of TSO's max size.
> 
> A 10.0-STABLE system with tso disabled on ix0 doesn't have a single packet
> over IP_MAXPACKET in 1 hour of runtime. I'll let it go a bit longer to
> increase confidence in this assertion, but I don't want to waste time on
> this when I could be logging problem packets on a system with TSO enabled.
> 
> Comments are very welcome..
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>