Re: zfs and git upload-pack

On 8/7/22 12:13, Philipp Takacs wrote:
> On Sun, 7 Aug 2022 11:12:20 -0700
> David Christensen <dpchrist@holgerdanske.com> wrote:
> 
>> On 8/7/22 10:57, Philipp Takacs wrote:
>>> On Sun, 7 Aug 2022 09:54:41 -0700
>>> David Christensen <dpchrist@holgerdanske.com> wrote:
>>>    
>>>> On 8/7/22 01:28, Philipp wrote:
>>>>> Hi all
>>>>>
>>>>> I host a quite uncommon git repository mostly out of binary
>>>>> files. I have the problem every time this repo is cloned the host
>>>>> allocate memory and going to swap. This leads to the host being
>>>>> unusable and need to force rebooted.
>>>>>
>>>>> The repo is stored on a zfs and nullmounted in a jail to run the
>>>>> git service over ssh. The host is a FreeBSD 13.1 with 4GB RAM and
>>>>> 4GB swap.
>>>>>
>>>>> What I have noticed is that the biggest memory consumtion is from
>>>>> mmap() a pack file. For the given repo this has the size of 6,7G.
>>>>> I suspect this file is mapped in memory but not correctly
>>>>> handled/unmaped (by the kernel) when not enough memory is
>>>>> available.
>>>>>
>>>>> I have tested some options to solve/workaround this issue:
>>>>>
>>>>> * limit the zfs ARC size in loader.conf
>>>>> * zfs set primarycache none for the dataset
>>>>> * limit datasize, memoryuse and vmemoryuse via login.conf
>>>>> * limit git packedGitLimit
>>>>>
>>>>> None of them have solved the issue.


I would restore them to previous values.


>>> this repo gets cloned a few times a month. Currently
>>> the Host dies because one client try to clone this repo.


What happens if the clone is attempted by a different user on the same 
workstation?


What happens if the clone is attempted from another workstation?


>> Please post console sessions that demonstrate cloning without failure
>> and cloning with failure.
> 
> Not sure what you mean. 


Please post client console sessions that demonstrate correct operation 
and failed operation.


It may be helpful to enable verbose and/or debug output.  If the output 
is large, reduce by hand, filter with grep(1), and/or put it into files, 
compress, upload, and post the URL's.


> This is a server, a client connect with a
> git client over ssh and use git-upload-pack 


https://git-scm.com/docs/git-upload-pack


> to receive the content of
> the repo. The communication of the git client and git-upload-pack works
> with stdin/stdout. I can give the logs of my git authorization handler
> (inside jail):

<snip>

What file?


> The last line mean the clone was finished[0]. But at this time
> everything else on the host was unusable. Here the corresponding content
> of /var/log/messages:

<snip>

> Between 12:00 and 14:00 the server started to be slow and running
> ssh/mosh session stopped working. Starting new sessions over ssh was
> not possible. The root login at 14:38 was me over ipmi try to somehow
> get the the host working again. But I could login and only run top then
> this session was also unusable. I have then restarted the server over
> ipmi.


It looks like the server is getting overloaded with incoming TCP 
packets, the  client is closing connections, the client is timing out 
when reconnecting, etc..  Near the end, I see jails being killed.  I do 
not see reasons why.  Perhaps there are clues in other logs.  Perhaps 
you can increase the logging verbosity of the Git and/or SSH services to 
obtain clues.


Start the following command in a terminal on the server to monitor ZFS 
disk activity (press Ctrl+C to exit):

# zpool iostat -v 60


Start the following command in another terminal on the server to monitor 
CPU and/or IO activity (press 'm' to switch between the two) (press 'q' 
to exit):

# top -S -s 60


David