zfs and git upload-pack

Hi all

I host a quite uncommon git repository mostly out of binary files. I
have the problem every time this repo is cloned the host allocate
memory and going to swap. This leads to the host being unusable and
need to force rebooted.

The repo is stored on a zfs and nullmounted in a jail to run the git
service over ssh. The host is a FreeBSD 13.1 with 4GB RAM and 4GB swap.

What I have noticed is that the biggest memory consumtion is from mmap()
a pack file. For the given repo this has the size of 6,7G. I suspect
this file is mapped in memory but not correctly handled/unmaped (by the
kernel) when not enough memory is available.

I have tested some options to solve/workaround this issue:

* limit the zfs ARC size in loader.conf
* zfs set primarycache none for the dataset
* limit datasize, memoryuse and vmemoryuse via login.conf
* limit git packedGitLimit

None of them have solved the issue.

On my last test I have noticed that the memory is not freed after the
git commands are stopped. To free the memory I had to unmount the zfs
dataset:

satanist@confusion:~ $ freecolor -om
             total       used       free     shared    buffers
cached Mem:          3923       3881         41          0          0
       0 Swap:         4096        456       3639
satanist@confusion:~ $ sudo zfs umount zroot/data/git/bigrepository
satanist@confusion:~ $ freecolor -om
             total       used       free     shared    buffers
cached Mem:          3923        278       3645          0          0
       0 Swap:         4096        456       3639

Any idea how to debug this or what else I could try solve this?

Philipp