Re: FreeBSD hugepages

On Thu, Jul 25, 2024 at 01:46:17PM -0500, Jake Freeland wrote:
> Hi there,
> 
> I have been steadily working on bringing Data Plane Development Kit (DPDK)
> on FreeBSD up to date with the Linux version. The most significant hurdle so
> far has been supporting concurrent DPDK processes, each with their own
> contiguous memory regions.
> 
> These contiguous regions are used by DPDK as a heap for allocating DMA
> buffers and other miscellaneous resources. Retrieving the underlying memory
> and mapping these regions is currently different on Linux and FreeBSD:
> 
> On Linux, hugepages are fetched from the kernel's pre-allocated hugepage
> pool and are mapped into virtual address space on DPDK initialization. Since
> the hugepages exist in a pool, multiple processes can reserve their own
> hugepages and operate concurrently.
> 
> On FreeBSD, DPDK uses an in-house contigmem kernel module that reserves a
> large contiguous region of memory on load. During DPDK initialization, the
> entire region is mapped into virtual address space. This leaves no memory
> for another independent DPDK process, so only one process can operate at a
> time.
> 
> I could modify the DPDK contigmem module to mimic Linux's hugepages, but I
> thought it would be better to integrate and upstream a hugepage-like
> interface directly in the FreeBSD kernel source. I am writing this email to
> see if anyone has any advice on the matter. I did not see any previous
> attempts at this in Phabriactor or the commit log, but it is possible that I
> missed it. I have read about transparent superpage promotion, but that seems
> like a different mechanism altogether.
> 
> At a quick glance, the implementation seems straightforward: read some
> loader tunables, allocate persistent hugepages at boot time, and create a
> pseudo filesystem that supports creating and mapping hugepages. I could be
> underestimating the magnitude of this task, but that is why I'm asking for
> thoughts and advice :)
> 
> For reference, here is Linux's documentation on hugepages:
> https://docs.kernel.org/admin-guide/mm/hugetlbpage.html

Are posix shm largepages objects enough (they were developed to support
DPDK).  Look for shm_create_largepage(3).