Linux netmap memory allocation

[Charlie Smurthwaite]

On 01/01/18 21:05, Vincenzo Maffione wrote:


2018-01-01 17:14 GMT+01:00 Charlie Smurthwaite <charlie at atech.media<mailto:charlie at atech.media>>:

Hi,

Thank you for your reply. I was able to resolve this.

1) I do indeed open one FD per NIC
2) I no longer specify nr_arg1, nr_arg2 nor nr_arg3. Instead I just verify that all NICs return with identical nr_arg2 so that the memory is shared between them.
3) I properly initialized my memory, my failure to do so was causing me a lot of confusion,

The resulting memory space is large enough for all the NICs, and everything works perfectly with zero-copy forwarding, great!

The only thing I am still having trouble with is the ability to simultaneously trigger a TX and an RX sync on all NICs. I have tried select, poll, and epoll, and in all cases, RX rings are updated but TX rings are not and TX packets are not pushed out (this occurs using both native and emulated netmap modes). I notice the documentation says "Note that on epoll and kqueue, NETMAP_NO_TX_POLL and NETMAP_DO_RX_POLL only have an effect when some event is posted for the file descriptor.", but the behaviour seems the same on poll and select as well as epoll, perhaps this is a linux-specific implementation detail?

I have also found that all of these mechanisms seem to incur a very high cost in terms of CPU time (making them no more efficient than busy waiting at 1Mpps+). My current approach is as follows, but I feel like there should be a better option:

    for(int n=0; n<NIC_COUNT; n++) {
      // usleep(10); // More CPU time seems to be saved with a careful sleep than with select/poll/epoll
      ioctl(fds[n], NIOCTXSYNC);
      ioctl(fds[n], NIOCRXSYNC);
      rxring = rxrings[n];
      while (!nm_ring_empty(rxring)) {
        // Forward any packets waiting in this NIC's RX ring to the appropriate TX ring
      }
    }

If you are using poll() or select() you should not use ioctl(NIOC*XSYNC), as the txsync/rxsync operations are automatically performed within the poll()/select() syscall (at least assuming you did not specify NETMAP_NO_TX_POLL).
Also, whether netmap calls or does not call txsync/rxsync on certain rings depends on the parameters passed to nm_open().
Make sure you check for nm_ring_space(txring) when forwarding.

Cheers,
  Vincenzo


Hi Vincenzo,

Thanks again for your assistance. You state the following (as does the manual):

"If you are using poll() or select() you should not use ioctl(NIOC*XSYNC), as the txsync/rxsync operations are automatically performed within the poll()/select() syscall (at least assuming you did not specify NETMAP_NO_TX_POLL)."

However, this is not happening for me :(

I am using poll(), and I am not specifying NETMAP_NO_TX_POLL, and have found that sometimes frames and sent only when the TX buffer is full, and sometimes they are not sent at all. They are never sent as expected on every invocation of poll(). If I run ioctl(NIOCTXSYNC) manually, everything works correctly. I assume I have simply missed something from my nmreq.

You also mentioned: "whether netmap calls or does not call txsync/rxsync on certain rings depends on the parameters passed to nm_open()". I do not use the nm_open helper method, but I am extremely interested to know what parameters would affect this bahaviour, as this would seem very relevant to my problem.

If you are interested or if it helps explain my question, my complete code (hopefully well commented but far from complete) can be found here: https://github.com/catphish/netmap-router/blob/58a9b957c19b0a012088c491bd58bc3161a56ff1/router.c

Specifically, if the ioctl call at line 92 is removed, the code does not work (packets are not transmitted, or are only transmitted when the buffer is full, which of these 2 behaviours seems to be random), however I would expect it to work because I do not specify NETMAP_NO_TX_POLL, and I would therefore hope that the poll() call on line 80 would have the same effect.

I hope this all makes sense, and again, I hope I have simply missed something from the nmreq i pass to NIOCREGIF.

It is worth mentioning that with the exception of this problem / confusion, I am getting extremely good results from this code and netmap in general.

Charlie


Charlie Smurthwaite
Technical Director

tel. email. charlie at atech.media<mailto:charlie at atech.media> web. https://atech.media

This e-mail has been sent by aTech Media Limited (or one of its assoicated group companys, Dial 9 Communications Limited or Viaduct Hosting Limited). Its contents are confidential therefore if you have received this message in error, we would appreciate it if you could let us know and delete the message. aTech Media Limited is a UK limited company, registration number 5523199. Dial 9 Communications Limited is a UK limited company, registration number 7740921. Viaduct Hosting Limited is a UK limited company, registration number 8514362. All companies are registered at Unit 9 Winchester Place, North Street, Poole, Dorset, BH15 1NX.