svn - but smaller?

[Kevin Oberman]

On Sat, Mar 16, 2013 at 8:14 AM, Ian Smith <smithi at nimnet.asn.au> wrote:

> On Wed, 13 Mar 2013 21:11:28 -0500, John Mehr wrote:
>  > On Wed, 13 Mar 2013 16:29:45 +1100 (EST)
>  >  Ian Smith <smithi at nimnet.asn.au> wrote:
> [..]
>  > > I have a small test system on which I'd installed (two instances of)
> 9.1 so
>  > > a couple of days ago I fetched ports with portsnap, installed svnup,
> and
>  > > ran it using the (just what I needed) example command in svnup(1).
>
> Just to reiterate: this is vanilla releng/9.1, no ports but docs and
> svnup installed, no X, 512MB RAM, nothing happening but occasional ARP.
> For completeness, it's running powerd and acpi_ibm but not cam.ctl.
>
>  > > I get about 700KB/s here, and svnup took about 15 minutes to update
> 9.1
>  > > sources to 9-stable.  This is fine.  Last night I ran it again, but
> it took
>  > > 12:42 to make no changes.  This seemed puzzling, as you'd said only a
> few
>  > > minutes for subsequent updates, but the reason appears to be that in
> both
>  > > cases, I ran it in script(1), and the default verbosity of 1 includes
> a
>  > > listing of every directory and file examined, followed by <CR> then
> <erase
>  > > to eol> codes.  Even in less -r (raw) mode it still has to display
> and skip
>  > > through all the (now invisible) lines; bit messy.
>  > >
>  > > Even the second do-nothing run made a 2MB script file, the original
> with
>  > > all 9.1 to -stable updates being 3.4MB.  So I'd love the option to
> only
>  > > list the changes (- and +) and simply ignore unchanged dirs/files
> without
>  > > any display for use in script(1).  Apart from that, I'm happy.
>  >
>  > Which mirror are you using?  I ran several tests tonight repeatedly
> fetching
>  > 9/stable from svn0.us-west (so they would all be do-nothing runs) both
> inside
>  > and outside of a script(1) capture and on both an old SSD and on a ZFS
>  > mirrored array (to see if the target media made any difference) and
> they all
>  > completed in 2 minutes, 43 seconds +/- 2 seconds on my 350 KB/s
> connection.
>
> svn0.us-west.freebsd.org as per example, but it's the closest to here
> anyway; pings avg 217.5ms stddev 0.4ms, us-east avg 269.3ms sd 9.5ms.
> fastest_cvsup used to find cvs mirrors in Australia at around 55-70ms,
> so for another while I thought RTT might be the issue - but it's not.
>
>  > I'll definitely put in a verbosity level that does exactly what you
> suggest.
>  > Sorry about that.
>
> Not at all, and thanks!  But neither, I now find, is logging the extra
> time issue; as mentioned elsewhere, running at -v0 makes next to no
> difference, 12:48 west vs 12:54 east on 4 do-nothing runs, 700KB/s link,
> not running in script(1) - though I think that matters not, or little.
>
> So I began watching top more closely, also running iostat -w10, noticing
> that svnup was pegging CPU at 100% with ~75% _system_? time for nearly 9
> minutes of the run, doing little or no disk I/O and a max of 130KB/s in,
> 22.6KB/s out, no stress there, growing to ~20MB resident, then doing a
> flurry of disk I/O at about 3-5MB/s for the last 3m40s odd.  IOW, svnup
> here is mostly heavily CPU-bound, which I don't understand, but then I
> know nothing about what level of computation the protocol demands.
>
> I've done some more controlled tests since, running both iostat -w10 and
> netstat -w10 -dn before, during and until a bit after the run, attached,
> hoping these might help pinpoint any bottlenecks?  This was running -v1
> default verbosity, hence the (not at all heavy) tty output numbers.
>
> Just sometimes, running slower hardware can have advantages!  On your
> times, yours likely has more than 500% of my single-core P3-M's 1133MHz
> performance, which would tend to mask this issue pretty well for you :)
>
> John Baldwin has best described when c{,v}sup deletes (only in-tree)
> files, covered in section 'delete' in csup(1).  I think this should
> probably be default behaviour, even if svn proper would do otherwise.
>
> cheers, Ian
>

Ian,

Have you looked with either truss or ktrace? This looks a lot like some
silliness like repeated calls to expensive system service (e.g.
gettimeofday(3) or stat(3)). If it's something like that, It can possibly
be improved a lot fairly easily. My best guess is some sort of graph of the
ports tree is going on, but that is just a wild guess as I have never
looked at or installed the code.
-- 
R. Kevin Oberman, Network Engineer
E-mail: rkoberman at gmail.com