getting NUMA into the tree (userland most interesting for me)

[John Baldwin]

On Friday, February 20, 2015 12:17:09 AM K. Macy wrote:
> >>> Yes, I think we have a fair bit to do in the kernel before we are in a
> >>> position to export anything truly useful to userland unfortunately.  The
> >>> last time I talked with Jeff about projects/numa (after the first draft
> >>> of the wiki page) I came away with the impression that there might be
> >>> some things we can pull out of that branch, but that it isn't suitable
> >>> for merging upstream directly.  Jeff noted that he and Alan had gone
> >>> through several iterations of this already (I believe at least 3
> >>> completely different policy designs) all of which had their own issues.
> >>> 
> >>> Outside of the VM I think that we can keep the APIs somewhat stable by
> >>> having this opaque policy cookie to pass around that we can redefine
> >>> the guts of later.  However, various parts of the VM all have to handle
> >>> whatever the policy defines, and while the vm_phys bits and
> >>> contigmalloc() might be kind of obvious to implement, higher level VM
> >>> layers like kmem() and malloc() are more complicated.  One thing that
> >>> is in projects/numa is changes for UMA that we can hopefully reuse much
> >>> of, but I don't recall how much (if any) of kmem/malloc is in there. 
> >>> Also, while vm_phys is one of the first things to do, I know that Alan
> >>> and Jeff have pending patches to remove the cache queue (since it is
> >>> far less useful than it seems) which simplify vm_phys making it easier
> >>> to implement NUMA policies there, so I'm hoping we can get that in
> >>> sooner before having to start tearing up the VM too much.  This is why
> >>> the stuff I currently have is targeted non-VM bits like interrupts as
> >>> getting that correct is lower-hanging fruit that might provide some
> >>> gains regardless.  Even once vm_phys is done I think the first thing to
> >>> tackle next is contigmalloc to facilitate static bus_dma allocations
> >>> (descriptor rings and such) being local to a device.
> >> 
> >> Contigmalloc improvements and cache queue removal are in the
> >> phabricator queue now. They are also prerequisites for per-cpu free
> >> page caches which are a huge scalability improvement for some
> >> workloads such as Netflix's.
> >> 
> >> There is still a fair amount of scalability work  (including Jeffr's
> >> per-domain pagedaemon work) that really needs to happens before we can
> >> seriously think about a general user-level NUMA interface.
> > 
> > Is there anything wrong with maybe bringing over the basic low level
> > allocator changes from projects/numa so the basics are there?
> 
> I think they're probably predicated on the work that is being
> shepherded in now. Even if not, it would require someone to shepherd
> it in and the corresponding spare cycles from alc to review / revise /
> repeat - which seem to be in short supply.

Can you add entries for these to the wiki page with links to the phab reviews?  
I know there is an entry for the page cache queue removal already, but you 
could add one for contigmalloc right next to it.

-- 
John Baldwin