dummynet performance

[Luigi Rizzo]

On Thu, Apr 9, 2020 at 2:18 AM Özkan KIRIK <ozkan.kirik at gmail.com> wrote:
>
> Hello,
>
> I wonder if it is possible to update dummynet code multi-threading capable
> practically?
> My idea is below:
> - A new sysctl tunable will be defined as
> "net.inet.ip.dummynet.thread_count" (default 1)
> - To distribute tasks along threads, each sched instance can be assigned to
> different thread.
> - By default, all schedulers assigned to thread 0 if no thread_id is set.
> With a new option named "thr_id" affinity can be set. as below:
>   ipfw sched 1 config type QFQ   (by defaut thr_id = 0)
>   ipfw sched 2 config type QFQ thr_id 1
>   ipfw sched 3 config type QFQ thr_id 2
>   ipfw sched 4 config type FIFO thr_id 3
>   ipfw sched 5 config type FIFO thr_id 3  ( sched 4 & 5 assigned to same
> thread 3 )
>
> I think this approach is more easy then completely rewrite or other
> solutions.
>

This change may not be completely trivial given that dummynet was designed as
single threaded and protected by a single lock.

If you want to proceed, you should do as follows:
- some members of struct dn_parms need to be duplicated for each thread,
  including prev_t, evheap, curr_time, the locks, and perhaps others.
  Probably may be useful to create a new struct for them, with a back
  pointer to the parent dn_parms, and include a refcount.

- add another rwlock to protect access to dn_cfg

- dummynet_io() does the initial demultiplexing calling dn_ht_find(),
  currently protected DN_BH_WLOCK(). This could be changed to an RWLOCK
  and I believe this initial step could be protected by a read lock
  (maybe there are corner cases where you need to create a new entry
  so you need to drop the lock, acquire in write mode and retry...)

- once the demux has returned a dn_fsk, maybe you can use a refcount
  to avoid the object being destroyed, release the read lock on
  dn_cfg.fshash, acquire an exclusive lock on the thread handling the
  dn_fsk, and proceed from there.

- the kernel thread that process the heap should now run each on
  one instance of the evheap

cheers
luigi