svn commit: r251886 - in head: contrib/apr contrib/apr-util contrib/serf contrib/sqlite3 contrib/subversion share/mk usr.bin usr.bin/svn usr.bin/svn/lib usr.bin/svn/lib/libapr usr.bin/svn/lib/libap...

[Peter Wemm]

On Sun, Jun 23, 2013 at 10:12 AM, Warner Losh <imp at bsdimp.com> wrote:
> On Jun 23, 2013, at 6:22 AM, Tijl Coosemans wrote:
> > On 2013-06-23 04:43, Garance A Drosehn wrote:
> >> On 6/22/13 2:38 PM, Tijl Coosemans wrote:
> >>> On 2013-06-20 21:34, Warner Losh wrote:
> >>>>
> >>>> I think insisting on a definitive statement on svn lite's mission
> >>>> statement is a way to obstruct progress. Sometimes you just need to
> >>>> things because they feel right, not because they have been through a
> >>>> 20-step approval process...
> >>>
> >>> For what it's worth, my reservations have always been because it
> >>> didn't feel right. I never asked for an approval process.
> >>>
> >>> I do think there should be a tool in base that can fetch source
> >>> updates and it would be nice if it could roll back changes and
> >>> even nicer if it could do bisects. But svn itself is not the
> >>> right tool for that.
> >>>
> >>> For instance, will you allow that svn is updated from version x.y
> >>> to x.(y+1) in a stable branch? If yes, then users might have to run
> >>> run "svn upgrade" which is a one way process, so how does importing
> >>> svn allow you to roll back changes or do bisects then? If no, then
> >>> who's volunteering to backport security fixes?
> >>
> >> What was added to the base system was 'svnlite', not 'svn' from
> >> the official subversion project.  The distinction is that
> >> 'svnlite' is a version meant only for access to the official
> >> FreeBSD repositories.  'svnlite' in the base system would only
> >> be upgraded when it is needed to match the FreeBSD respository.
> >> And if you need to run 'svn upgrade' to access the FreeBSD
> >> repository, then it doesn't make much difference if you have
> >> to do that with 'svnlite' or via the official 'svn' port.
> >>
> >> I'm not sure that my comments provide an answer to what you're
> >> concerned about, but anyone who wants the latest version of
> >> 'svn' to match their own repositories is still going to need
> >> to install the port.  We're not going to blindly update
> >> 'svnlite' every time a new version of 'svn' is released.
> >> 'svnlite' is going to be updated on *FreeBSD*'s schedule,
> >> not on the schedule of the subversion project.
> >>
> >> It is true that we're going to have to be careful when it does
> >> come time to switch to some new repo-format for the FreeBSD
> >> repository.  We might find ourselves in some kind of chicken-
> >> and-egg situation at that point.  And when we do make a
> >> significant upgrade to the FreeBSD repository, then we're
> >> going to have to upgrade 'svnlite' across multiple FreeBSD
> >> branches at the same time, including all -stable branches.
> >> But when that issue comes up it'll come up on our schedule,
> >> because we'll control both 'svnlite' and the FreeBSD repo.
> >
> > It is precisely the other way around. Once you do a FreeBSD release
> > (releng branch) that release will be stuck with the same version of
> > svnlite for years. You will end up with multiple releases with
> > multiple versions of svnlite that you cannot change. You have zero
> > control.
>
> Then they will never have to do svn update, since their checked out tree will never be obsolete in relationship to the version that's installed.
>
> > A port on the other hand is the same for all branches and releases
> > of FreeBSD. It is a single point where you have total control over
> > the version of svn for all users. Conceptually, a port corresponds
> > to the fact all branches and releases share the same subversion
> > repo.
>
> Except that you still need to do svn update on trees that are checked out from old repos.
>
> I'm having a real hard time seeing this as an issue based on my experience with the svn port since we made the switch. Then again, I've been talking to the svn repo over http, which is independent of the version of the repo on the other end...
>
> Warner

I've been resisting the urge to dive in.  I will say right up front
that a drive-by commit like this is Not The Way We Are Supposed To Do
Things(TM).  I know that I have ruffled some people's feathers, and
for that I'm sorry.  I've also got a lot of thank-you messages. I'll
try and explain myself.

[I know this is long. Please don't start replying until you've read it
to the end.]

The bottom line as to why I did this.. we are an open *source*
operating system.  After actually working fully from source on the
freebsd.org cluster, it has become rather apparent that convenience of
working with a pure-source environment has regressed considerably
since 2008 when we switched from cvs.


As an example of inconvenient,  take this clean 24 core system with 24
GB ram.  Suppose I have a problem that I want to do a bisection search
to see when it first appeared (right there, freebsd-update is
excluded).


# rm -rf /usr/local/* /var/db/pkg/* /var/db/ports/*

then a config-recursive, taking default options to match packages:
# cd /usr/ports/devel/subversion; time make config-recursive
(immediately hitting enter)
41.693u 4.778s 0:37.12 125.1%	30872+507k 185+13771io 360pf+0w
37 seconds just to configure the build options.

# cd /usr/ports/devel/subversion; time make install
[...]
Installing subversion-1.8.0_1... done
902.035u 213.337s 13:53.08 133.8%	21955+423k 1282+248411io 18705pf+0w

14 minutes, 30 seconds on a reasonably fast system, not counting
acquiring/extracting/updating the ports tree in the first place.
(portsnap takes over 20 hours just to verify its checksums on my
firewall, so please don't tell me portsnap is the solution for
everything!)

At the end the footprint is:
# pkg info
apr-1.4.6.1.4.1_3              Apache Portability Library
autoconf-2.69                  Automatically configure source code on
many Un*x platforms
autoconf-wrapper-20101119      Wrapper script for GNU autoconf
automake-1.12.6                GNU Standards-compliant Makefile generator
automake-wrapper-20101119      Wrapper script for GNU automake
db42-4.2.52_5                  The Berkeley DB package, revision 4.2
dialog4ports-0.1.5             Console Interface to configure ports
expat-2.0.1_2                  XML 1.0 parser written in C
gdbm-1.9.1                     GNU database manager
gettext-0.18.1.1_1             GNU gettext package
gmake-3.82_1                   GNU version of 'make' utility
help2man-1.43.2                Automatically generating simple manual
pages from program output
libiconv-1.14_1                A character set conversion library
libtool-2.4.2                  Generic shared library support script
m4-1.4.16_1,1                  GNU m4
p5-Locale-gettext-1.05_3       Message handling functions
perl-5.14.4                    Practical Extraction and Report Language
pkg-1.0.14                     New generation package manager
pkgconf-0.9.2_1                Utility to help to configure compiler
and linker flags
python27-2.7.5_1               Interpreted object-oriented programming language
serf-1.2.1                     Serf HTTP client library
sqlite3-3.7.17_1               SQL database engine in a C library
subversion-1.8.0_1             Version control system

highlights: db4.2(!!wtf?), gdbm, perl, python, make, m4

du -sh says: 238M /usr/local

The same machine (WITHOUT_CLANG) builds world *and* its kernel in:
3570.850u 845.997s 13:29.91 545.3%	6978+1075k 28182+1449464io 21379pf+0w

Same again but with clang enabled (and it's the default compiler. with
maximum working -j):
8276.034u 746.466s 31:24.78 478.7%	29824+610k 25348+1683059io 36894pf+0w

And svnlite, *with* clang to get the worst case:
/usr/src/usr.bin/svn# time make -s -j24 depend all
117.943u 15.330s 0:42.74 311.8%	32804+522k 2+37666io 1231pf+0w

du -sh /usr/bin/svnlite* says 18M

In summary, on the same machine, no third party binaries:
?? acquire ports tree (talk to my soekris firewall for worst-case examples)
?? deal with perl directory layout changes (actually pretty quick if
you just rm -rf /usr/local and start over)
14 minutes 30 seconds to build from svn ports
vs
13 minutes 30 seconds to buildworld without clang
31 minutes 24 seconds to buildworld with clang
vs
42 seconds to build svnlite (not actually lite, fully functional)

Don't get me wrong, I love what pkgng brings to the table.  But as a
"source code" consumer, it makes my skin crawl each time somebody
effectively says "oh, you shouldn't build from source, you should
install binaries to solve the problem".  Every time I hear that it
feels like we're giving up.  If I wanted binaries, I'd be running a
binary-only OS like Windows, Linux or OSX.

As for compatability and old releases:
Your svn-1.4 / 1.5 binaries from years ago still work on the project's
svn-1.8 infrastructure.
svnlite doesn't change what "svn up" does on your $path.
svnlite doesn't interfere with ports in ANY WAY.  If your perl install
broke and don't want to deal with it right now, you don't have to.
svnlite doesn't interfere with your 1.5 or 1.6 checkouts unless you
type it by name or compile it as "svn" with -DWITH_SVN=yes

What it brings:
svnlite restores the 'at-your-fingertips' convenience that we kind of
lost when we switched from cvs in 2008.
You can install an image in a VM or whatever and instantly participate
in development.
It brings WITH_SVN=yes in make.conf so you can get finger memory
compatibility if you wish.

I still want svnup to be finished and imported.   Just like when we
had cvs and csup, they're targeted at different people.

If a few extra seconds of buildworld time bother you, there *is* a
knob to turn it off.  But if we need to talk about that then we also
need to talk about multiple compilers, multiple installers, multiple
firewalls, multiple command line editors, multiple nfs servers,
multiple nfs clients, obsolete dns stacks, etc etc.

The bottom line is that this is as close to "free" as it can get.
It adds options for people.
It does not limit or restrict people's options
It does not interfere with people's existing svn checkouts (unless
they want to, options!)
And I am not willing to back down on it...
But I do apologize for the hypocrisy in committing a drive-by and the
upset that it caused a few people.

I don't know about everyone else but I've now spent more time talking
about this than I'll spend waiting for it to compile in the rest of my
life using FreeBSD as a source-code-based OS.
--
Peter Wemm - peter at wemm.org; peter at FreeBSD.org; peter at yahoo-inc.com; KI6FJV