svn-export Re: svn bdb checkout?

[Greg Larkin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 1/25/13 1:53 PM, Xyne wrote:
> Greg Larkin wrote:
> 
>> # python3 svn-export -r 310000 
>> http://svn.freebsd.org/ports/head/ports-mgmt ports-mgmt Exporting
>> new repository. Exported revision 310000 # python3 svn-export
>> --revision-file rev.dat 
>> http://svn.freebsd.org/ports/head/ports-mgmt ports-mgmt 
>> Determining current revision. Current revision is 310981. 
>> Determining changes [310000:310981]. 16 new/modified files, 5
>> deletions Traceback (most recent call last): File "svn-export",
>> line 459, in <module> svn.main() File "svn-export", line 432, in
>> main self.remove_old_files(deleted) File "svn-export", line 310,
>> in remove_old_files os.unlink(path) PermissionError: [Errno 1]
>> Operation not permitted: 'ports-mgmt/pkg/files' # exit
>> 
>> As far as I can tell, svn-export attempts to remove a directory
>> before the file contained within that directory.  It seems like 
>> PermissionError isn't the best error code for Python to throw in
>> that case, since it appears to mask the root problem.
> 
> On Arch Linux, Python raises the expected "IsADirectoryError"
> exception. Perhaps "PermissionError" is due to a bug in Python 3 on
> FreeBSD. It would not be the first exception bug that I have
> encountered (e.g. os.makedirs(..., exist_ok=True) raises
> "FileExists" when permissions differ on existing directories...
> very confusing the first time it shows up).
> 
> Regardless, svn-export now tries to remove a directory in both
> cases. Please updated to version 2013.1.25 and let me know if it
> works on FreeBSD.

Yes, it does work on FreeBSD now, but I suggest investigating whether
it's possible to reorder the removal list with files first, then
directories.  That way, you won't have to rely on an exception to
determine if a directory to be removed is not empty yet.

The os.walk() method looks like it would be useful here:
http://docs.python.org/3/library/os.html?highlight=os.walk#os.walk

> 
> 
>>> Incidentally, if anyone is looking at the code for svn-export,
>>> feel free to suggest a better way to incrementally export new 
>>> subdirectories (see the "get_new_files()" method).
>> 
>> I'll take a look at that soon to see if there are any possible 
>> optimizations.
> 
> Ok, thanks.

I looked at the method, and my first idea was to "svn cat" as many
files at once as possible, up to the maximum command line length.  You
will avoid spawning a lot of svn processes and opening new network
connections each time.  If the threads option is >1, you could divide
the maximum command line length by that number and keep filling in
filenames up to that limit in each child process.

I expect you would also get a nice performance boost if you changed
from spawning svn commands to the Python SVN bindings.  Here's an
example of how it works:
http://svn.apache.org/repos/asf/subversion/trunk/tools/examples/getfile.py

> 
> 
>> Once we create a port for svn-export, we typically point to the 
>> original download site in the port Makefile.  It's a good idea
>> for the port maintainer to find a mirror site and include that
>> too, but if not, all distro files are eventually mirrored to the
>> FreeBSD cluster.
>> 
>> Ideally, the source distros remain in place forever or at least
>> until the FreeBSD port is updated to the latest version, but if
>> that's not possible, I can mirror the files too.
> 
> I have changed the time-to-live to 6 months for old archives.

Excellent, thank you.

Hope that all helps,
Greg

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlEC/ooACgkQ0sRouByUApCa0wCfQr4rIT9oQrI3LkYPlMGbfCnj
KqoAniabxFC2xeNAzvdLHLYhXhxYs1xi
=cfUS
-----END PGP SIGNATURE-----