svn - but smaller?

John Mehr jcm at
Sun Mar 17 01:25:37 UTC 2013

On Sun, 17 Mar 2013 02:14:30 +1100 (EST)
 Ian Smith <smithi at> wrote:
> On Wed, 13 Mar 2013 21:11:28 -0500, John Mehr wrote:
> > On Wed, 13 Mar 2013 16:29:45 +1100 (EST)
> >  Ian Smith <smithi at> wrote:
> as per example, but it's the 
>closest to here 
> anyway; pings avg 217.5ms stddev 0.4ms, us-east avg 
>269.3ms sd 9.5ms. 
> fastest_cvsup used to find cvs mirrors in Australia at 
>around 55-70ms, 
> so for another while I thought RTT might be the issue - 
>but it's not.

I'm seeing ~110ms pings to

> > I'll definitely put in a verbosity level that does 
>exactly what you suggest. 
> > Sorry about that.
> Not at all, and thanks!  But neither, I now find, is 
>logging the extra 
> time issue; as mentioned elsewhere, running at -v0 makes 
>next to no 
> difference, 12:48 west vs 12:54 east on 4 do-nothing 
>runs, 700KB/s link, 
> not running in script(1) - though I think that matters 
>not, or little.
> So I began watching top more closely, also running 
>iostat -w10, noticing 
> that svnup was pegging CPU at 100% with ~75% _system_? 
>time for nearly 9 
> minutes of the run, doing little or no disk I/O and a 
>max of 130KB/s in, 
> 22.6KB/s out, no stress there, growing to ~20MB 
>resident, then doing a 
> flurry of disk I/O at about 3-5MB/s for the last 3m40s 
>odd.  IOW, svnup 
> here is mostly heavily CPU-bound, which I don't 
>understand, but then I 
> know nothing about what level of computation the 
>protocol demands.

Except for the time duration of the run, the behavior 
you're describing is normal.  Using the svn protocol, the 
program runs in three phases:

1) It issues a series of get-dir commands to the server to 
get a list of all the files/directories in the repository 
and stores the structure in memory.  There is some disk 
activity during this as it creates the directory tree (it 
uses one lstat per directory and creates the directory if 
it doesn't exist).  On my development machine, this phase 
takes about 25% of the total time of a do-nothing run.

2) Because the svn protocol doesn't include MD5 signatures 
and the last-author/committed date/committed revision 
fields in the results of the get-dir commands 
<gripe>although they *are* included in the http protocol 
responses</gripe> it has to issue a get-file request for 
each file to get this information.  This phase takes about 
50% of the total time of a do-nothing run.

3) The downloaded MD5 signatures are for versions of the 
files that do not have their $FreeBSD$ tags expanded, so 
each file on the local machine must be loaded into memory 
and have their $FreeBSD: ... $ tags collapsed and MD5 
summed to see if there's a match.  If there's no match, 
the file on the server is downloaded.  This phase takes up 
the remaining 25% and is I/O intensive.

This three step process is further complicated because of 
the huge time penalty that occurs when sending commands 
one-at-a-time.  To get around this, as many commands that 
can fit in 32KB are grouped together and sent in bulk.

> I've done some more controlled tests since, running both 
>iostat -w10 and 
> netstat -w10 -dn before, during and until a bit after 
>the run, attached, 
> hoping these might help pinpoint any bottlenecks?  This 
>was running -v1 
> default verbosity, hence the (not at all heavy) tty 
>output numbers.
> Just sometimes, running slower hardware can have 
>advantages!  On your 
> times, yours likely has more than 500% of my single-core 
>P3-M's 1133MHz 
> performance, which would tend to mask this issue pretty 
>well for you :)

I do all of my development and test runs on a nettop with 
an AMD E-350 (dual core, 1.6GHz) processor, and with the 
program only utilizing one core I'd be willing to bet the 
two chips are in roughly the same league.  Memory and hard 
drives probably aren't, and with 512MB of memory, how does 
your swap utilization look?

As soon as I can get a replacement CMOS battery, I'll be 
dusting off my old Athlon 2700+ system to run tests on it.

> John Baldwin has best described when c{,v}sup deletes 
>(only in-tree) 
> files, covered in section 'delete' in csup(1).  I think 
>this should 
> probably be default behaviour, even if svn proper would 
>do otherwise.

Will do.  Setting up both a default of deleting only files 
that appear in previous checkouts plus giving each user 
the option of skipping specific directories from any 
pruning is on the to-do list.  Thanks!

> cheers, Ian

More information about the freebsd-stable mailing list