Space problems with svn-cur

[Montgomery-Smith, Stephen]

On 03/13/2016 10:53 AM, Montgomery-Smith, Stephen wrote:
> On 03/13/2016 01:14 AM, Roman Kurakin wrote:
>> Hi,
>>
>> On 03/13/2016 04:01 AM, Montgomery-Smith, Stephen wrote:
>>> On 03/12/2016 03:59 AM, Julian H. Stacey wrote:
>>>> "Montgomery-Smith, Stephen" wrote:
>>>>> The svn repository is now getting so large that the CTM generating
>>>>> computer is running out of disk space when it creates the xEmpty deltas
>>>>> for svn-cur.  The easist band-aid fix is for me to simply stop
>>>>> generating these xEmpty deltas.  Since CTM is no longer FreeBSD
>>>>> approved, I'm going to assume that no-one new is going to join, and
>>>>> hence this is not a problem.
>>>>>
>>>>> Comments?
>>>> Hi Stephen
>>>> There's plenty of space on ctm.berklix.org
>>> The issue is that there isn't enough space to create the xEmpty deltas.
>>>   There is plenty of space to store the deltas.
>>>
>>> The svn repositories add up to about 35GB.  Fully compressed, they
>>> shrink to about 4GB.
>>>
>>> I'll look into other solutions.  But I also get a feeling I am putting a
>>> lot of effort into something that is only used by a very few people.
>> The xEmpty is the best way to start clean new copy on a new host.
>> The solution could be a big deltas relative the last xEmpty.
> 
> This is the problem.  Internally the svn xEmpty deltas look like this:
> 
> CTMTR <big number>
> <huge tar file of the entire repository>
> 
> where the big number is the size of the following huge tar file.  What I
> have been doing is to create the huge tar file in a tmp directory, and
> then find its size, and then copy this file onto the end of the xEmpty
> file.  This meant that tmp had to be able to take 25GB (the size of the
> ports repository).
> 
> Having big deltas relative to the last xEmpty is not a solution, because
> I still have to store the repository of every 100th delta.  That's 25GB
> whichever way you look at it.
> 
> What I am trying now is this.  I create the tar file twice - once to get
> the size, and the second time to append it to the xEmpty file.  This
> relies on tar creating the same size file both times.  Experiments I
> recently tried suggest this should be OK.

This seems to be working.  I created an xEmpty for 4406.

I also found out it takes about 5 hours to create an xEmpty for svn-cur.
 Almost all that time is spent by xz doing the compression.