flowtable, collisions, locking and CPU affinity

[Gleb Smirnoff]

On Fri, Feb 07, 2014 at 04:12:56PM -0800, Adrian Chadd wrote:
A> I've been knee deep in the flowtable code looking at some of the less
A> .. predictable ways it behaves.
A> 
A> One of them is the collisions that do pop up from time to time.
A> 
A> I dug into it in quite some depth and found out what's going on. This
A> assumes it's a per-CPU flowtable.
A> 
A> * A flowtable lookup is performed, on say CPU #0
A> * the flowtable lookup fails, so it goes to do a flowtable insert
A> * .. but since in between the two, the flowtable "lock" is released so
A> it can do a route/adjacency lookup, and that grabs a lock
A> * .. then the flowtable insert is done on a totally different CPU
A> * .. which happens to _have_ the flowtable entry already, so it fails
A> as a collision which already has a matching entry.
A> 
A> Now, the reason for this is primarily because there's no CPU pinning
A> in the lookup path and if there's contention during the route lookup
A> phase, the scheduler may decide to schedule the kernel thread on a
A> totally different CPU to the one that was running the code when the
A> lock was entered.
A> 
A> Now, Gleb's recent changes seem to have made the instances of this
A> drop, but he didn't set out to fix it. So there's something about his
A> changes that has changed the locking/contention profile that I was
A> using to easily reproduce it.
A> 
A> In any case - the reason it's happening above is because there's no
A> actual lock held over the whole lookup/insert path. It's a per-CPU
A> critical enter/exit path, so the only way to guarantee consistency is
A> to use sched_pin() for the entirety of the function.
A> 
A> I'll go and test that out in a moment and see if it quietens the
A> collisions that I see in lab testing.
A> 
A> Has anyone already debugged/diagnosed this? Can anyone think of an
A> alternate (better) way to fix this?

Can't we just reuse the colliding entry?

Can you evaluate patch attached (against head) in your testing
conditions?

-- 
Totus tuus, Glebius.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: flowtable_collisions.diff
Type: text/x-diff
Size: 4357 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20140213/3af867b1/attachment.diff>