Re: zfs and git upload-pack

On 8/7/22 10:57, Philipp Takacs wrote:
> On Sun, 7 Aug 2022 09:54:41 -0700
> David Christensen <dpchrist@holgerdanske.com> wrote:
> 
>> On 8/7/22 01:28, Philipp wrote:
>>> Hi all
>>>
>>> I host a quite uncommon git repository mostly out of binary files. I
>>> have the problem every time this repo is cloned the host allocate
>>> memory and going to swap. This leads to the host being unusable and
>>> need to force rebooted.
>>>
>>> The repo is stored on a zfs and nullmounted in a jail to run the git
>>> service over ssh. The host is a FreeBSD 13.1 with 4GB RAM and 4GB
>>> swap.
>>>
>>> What I have noticed is that the biggest memory consumtion is from
>>> mmap() a pack file. For the given repo this has the size of 6,7G. I
>>> suspect this file is mapped in memory but not correctly
>>> handled/unmaped (by the kernel) when not enough memory is available.
>>>
>>> I have tested some options to solve/workaround this issue:
>>>
>>> * limit the zfs ARC size in loader.conf
>>> * zfs set primarycache none for the dataset
>>> * limit datasize, memoryuse and vmemoryuse via login.conf
>>> * limit git packedGitLimit
>>>
>>> None of them have solved the issue.
>>>
>>> On my last test I have noticed that the memory is not freed after
>>> the git commands are stopped. To free the memory I had to unmount
>>> the zfs dataset:
>>>
>>> satanist@confusion:~ $ freecolor -om
>>>                total       used       free     shared    buffers
>>> cached Mem:          3923       3881         41          0
>>> 0 0 Swap:         4096        456       3639
>>> satanist@confusion:~ $ sudo zfs umount zroot/data/git/bigrepository
>>> satanist@confusion:~ $ freecolor -om
>>>                total       used       free     shared    buffers
>>> cached Mem:          3923        278       3645          0
>>> 0 0 Swap:         4096        456       3639
>>>
>>> Any idea how to debug this or what else I could try solve this?
>>>
>>> Philipp
>>
>>
>> Add memory; the more, the better.
> 
> Can you borrow me your credit card?
> 
> Jokes aside. this repo gets cloned a few times a month. Currently the
> Host dies because one client try to clone this repo. This should be work
> with 4GB memory.


Please post console sessions that demonstrate cloning without failure 
and cloning with failure.


David