Space problems with svn-cur

[Montgomery-Smith, Stephen]

On 03/13/2016 01:14 AM, Roman Kurakin wrote:
> Hi,
> 
> On 03/13/2016 04:01 AM, Montgomery-Smith, Stephen wrote:
>> On 03/12/2016 03:59 AM, Julian H. Stacey wrote:
>>> "Montgomery-Smith, Stephen" wrote:
>>>> The svn repository is now getting so large that the CTM generating
>>>> computer is running out of disk space when it creates the xEmpty deltas
>>>> for svn-cur.  The easist band-aid fix is for me to simply stop
>>>> generating these xEmpty deltas.  Since CTM is no longer FreeBSD
>>>> approved, I'm going to assume that no-one new is going to join, and
>>>> hence this is not a problem.
>>>>
>>>> Comments?
>>> Hi Stephen
>>> There's plenty of space on ctm.berklix.org
>> The issue is that there isn't enough space to create the xEmpty deltas.
>>   There is plenty of space to store the deltas.
>>
>> The svn repositories add up to about 35GB.  Fully compressed, they
>> shrink to about 4GB.
>>
>> I'll look into other solutions.  But I also get a feeling I am putting a
>> lot of effort into something that is only used by a very few people.
> The xEmpty is the best way to start clean new copy on a new host.
> The solution could be a big deltas relative the last xEmpty.

This is the problem.  Internally the svn xEmpty deltas look like this:

CTMTR <big number>
<huge tar file of the entire repository>

where the big number is the size of the following huge tar file.  What I
have been doing is to create the huge tar file in a tmp directory, and
then find its size, and then copy this file onto the end of the xEmpty
file.  This meant that tmp had to be able to take 25GB (the size of the
ports repository).

Having big deltas relative to the last xEmpty is not a solution, because
I still have to store the repository of every 100th delta.  That's 25GB
whichever way you look at it.

What I am trying now is this.  I create the tar file twice - once to get
the size, and the second time to append it to the xEmpty file.  This
relies on tar creating the same size file both times.  Experiments I
recently tried suggest this should be OK.

Note the xEmpty file is compressed on the fly, so this file doesn't take
a lot of space - only 4GB or so.

The other possibility is to ask for more disk space.

Stephen