Re: CFT: snmalloc as libc malloc

On Thu, Feb 09, 2023 at 12:08:49PM +0000, David Chisnall wrote:
> Hi,
> 
> For the few yearsI've been running locally with snmalloc as the malloc in
> libc.  Eventually I'd like to propose this for upstreaming but it needs some
> wider testing first.
> 
> For those unfamiliar with snmalloc (https://github.com/microsoft/snmalloc),
> it is an allocator (or, rather, a toolkit for building allocators) from my
> team at Microsoft Research designed for both performance and security.  A
> few highlights:
> 
>  - Snmalloc uses a message-passing design, which makes allocating on one
> thread and freeing on another cheap.
>  - Very fast allocation performance
>  - Randomisation of relative locations of allocations
>  - Most metadata is stored out-of-band
>  - In-band metadata uses some lighweight encryption to protect against
> corruption.
>  - Support for CHERI.
> 
> In the (limited!) testing that I've done, it outperforms jemalloc and
> results in a smaller libc binary.
> 
> I've also previously managed to use it in the kernel, though that code
> hasn't been tested in a while (last used with FreeBSD 11):
> 
> https://github.com/microsoft/snmalloc/blob/main/src/snmalloc/pal/pal_freebsd_kernel.h
> 
> It is also used in the Verona process sandboxing work, which makes it easy
> to isolate a library in a capsicum Sandbox:
> 
> https://github.com/microsoft/verona/tree/master/experiments/process_sandbox
> 
> We test on FreeBSD in CI upstream and the code is actively maintained.
> We have implemented compatibility wrappers for all of the jemalloc
> non-standard APIs that FreeBSD's libc exposes.
> 
> In particular, snmalloc is designed to make it very cheap to find the start
> and end of an allocation, given a heap pointer.  This means that we can
> insert bounds checks in critical libc functions to prevent heap overflow.
> This is done in the branch for memcpy, which some investigation of a corpus
> of security vulnerabilities showed was the root cause of about 10% of
> arbitrary-code-execution vulnerabilities.
> 
> The bounds checks are controlled via an environment variable
> LIBC_BOUNDS_CHECKS.  Setting this to 0 disables checks, to 1 checks on
> destination arguments, and to 2 checks sources and destinations.  An ifunc
> resolver selects the correct memcpy implementation at load time.
> 
> I did have a version that checked a bunch of other libc functions (e.g.
> sprintf, puts) but it was quite hacky (and the way the ifunc resolves was
> implemented broke tcl).
> 
> The current branch puts two things behind the MALLOC_PRODUCTION toggle:
> 
>  - The additional security checks that detect corruption of malloc state.
>  - Pretty-printing errors.
> 
> We are currently separating the former into separate knobs upstream, some
> subset should probably be turned on by default in production.  The latter
> has less of a performance impact than it had and will probably be on for all
> configurations at some point once we've refactored slightly to ensure the
> compiler can tail call the failure function (which moves it entirely off the
> fast path).  With this enabled, you get errors that look like this:
> 
> Fatal Error!
> memcpy with source out of bounds of heap allocation:
>         range [0x14823c02440, 0x14823c0246a)
>         allocation [0x14823c02440, 0x14823c02450)
> range goes beyond allocation by 0x1a bytes
> 
> Abort trap (core dumped)
> 
> Without it, you just get an illegal instruction trap.
> 
> There are a few limitations in the current branch:
> 
>  - The memcpy integration is broken on non-amd64 platforms (patches welcome
> from people who can test these!).
>  - Only memcpy (not, for example, memmove) has bounds checks.
>  - The memcpy in rtld is naive, which may impact performance.
>  - MALLOC_PRODUCTION conflates too many things
> 
> The branch is here:
> 
> https://github.com/davidchisnall/freebsd-src/tree/snmalloc2
> 
> It adds snmalloc as a submodule in contrib.  FreeBSD is allergic to
> submodules, so upstreaming will need to replace this with something more
> complicated.  You should be able to cherry-pick the top commit on any
> vaguely-recent -CURRENT.

So I took a little bit of a different approach, which should provide
the same end result as your submodule approach. Note that I'm doing
this in HardenedBSD 14-CURRENT (the hardened/current/master branch).

1. git cherry-pick -xs a5c83c69817d03943b8be982dd815c7e263d1a83
2. git rm -f .gitmodules contrib/snmalloc
3. git commit
4. git subtree add -P contrib/snmalloc \
   git@github.com:microsoft/snmalloc.git main

I believe this should leave me with a tree that populates
contrib/snmalloc and pulls in your non-contrib/ changes, leading me to
end up in the same end state as your submodule approach.

I am seeing some build errors. I've uploaded a WITHOUT_CLEAN=yes log
to:

https://hardenedbsd.org/~shawn/2023-02-10_snmalloc-01.log.txt

Note that this is with llvm 15.0.7 that just landed in FreeBSD main.

Any non-XKCD[138]-conforming pointers would be appreciated. ;-)

Thanks,

-- 
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc