Lesson learned from working with slow HW crypto card

[Youlin Feng]

   This is a lesson I learned from working with a slow HW crypto card in
   the presence of the GbE interface. Hope it is helpful to others when
   they run into the same issue.

   We are using a 200Mpps HW crypto card on a FreeBSD 5.3 system as a GbE
   gateway. With ~700Mbps traffic inbound for encryption, the system
   dropped into a livelock state and the throughput is barely ~50Mbps.
   The console doesn't respond to key strokes and the system complains
   about the lack of memory.

   The problem turns out to be related to the fact that the crypto
   request queue doesn't have a limit on the queue length. Due to the
   async nature of HW crypto processing and the speed mismatch between
   the crypto card and the GbE interface, the request queue becomes so
   long that CPU will be pretty much monopolized by the crypto_proc
   thread running at SWI_NET pri level, leaving no cycles for lower pri
   kernel tasks and applications. Because the crypto callback thread can
   only run as fast as the HW crypto speed, memory resources are freed at
   a much slower rate than when they were allocated, leading to
   out-of-memory condition.

   The workaround I used is to cap the crypto request queue, ie crp_q.
   Capping it to 64 works fine with the 200Mbps crypto card, increasing
   the throughput to ~200Mbps. And livelock condition is also gone.

   The asymmetric op queue, ie crp_kq, can be left as unlimited because
   there are much fewer such requests.

   The limited request queue shouldn't have any impact on the SW crypto
   since it is a synchronous operation.