The vim port needs a refresh

[Matthias Andree]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 28.05.2013 22:24, schrieb Lars Engels:

> Someone in this thread proposed to change the port to use phttpget, so I
> gave it a try using a German mirror nearby with 6 Mbit/s downlink:
> 
> $ time /usr/libexec/phttpget ftp.vim.ossmirror.de $(eval echo /pub/vim/patches/7.3/{$(make -C /usr/ports/editors/vim -VPATCHFILES | sed 's/\ /,/g')})
> http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.001: 200 OK
> http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.002: 200 OK
> http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.003: 200 OK
> http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.004: 200 OK
> http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.005: 200 OK
> [...]
> http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.974: 200 OK
> http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.984: 200 OK
> http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.985: 200 OK
> http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.1000: 200 OK
> 
> real	0m12.509s
> user	0m0.154s
> sys	0m0.089s
> 
> 
> That's really nice! 
> 
> Compare this to the current version using fetch(1):

This incurs massive overhead from the generation of the fetch URLs, on
top of "fetch" being slower.

You would have to come up with something similar to your phttpget line,
only that it runs fetch(1) instead.  Try:

$ make -C /usr/ports/editors/vim fetch FETCH_CMD=true

to see how slow URL generation is.

On one of my reasonably fast amd64 machines (dual-core 2.1 GHz AMD
BE-2350), residing on the T-Home domestic backbone near the Ruhr area in
Germany with a 3000/384 kBit/s ADSL that has roughly 50 ms ping round
trips, it generates approximately 1.7 fetch commands in 1 s, and see
below for fetch speed:

>  time make PATCH_SITES=http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/ fetch
> ===>  Found saved configuration for vim-7.3.669_1
> ===>   vim-7.3.1014 depends on file: /usr/local/sbin/pkg - found
> => 7.3.002 doesn't seem to exist in /usr/ports/distfiles/vim.
> => Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.002
> 7.3.002                                       100% of 1610  B   16 MBps 00m00s
> => 7.3.003 doesn't seem to exist in /usr/ports/distfiles/vim.
> => Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.003
> 7.3.003                                       100% of 1299  B 1281 kBps 00m00s
> => 7.3.004 doesn't seem to exist in /usr/ports/distfiles/vim.
> [...]
> => 7.3.984 doesn't seem to exist in /usr/ports/distfiles/vim.
> => Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.984
> 7.3.984                                       100% of 1706  B 2852 kBps 00m00s
> => 7.3.985 doesn't seem to exist in /usr/ports/distfiles/vim.
> => Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.985
> 7.3.985                                       100% of 1691  B   14 MBps 00m00s
> => 7.3.1000 doesn't seem to exist in /usr/ports/distfiles/vim.
> => Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.1000
> 7.3.1000                                      100% of 1637  B 1715 kBps 00m00s
> ===> Fetching all distfiles required by vim-7.3.1014 for building
> 
> Total time                              : 3:48.55s
> CPU utilisation (percentage)            : 54.5%

Now, using a similar approach as yours, I get, with fetch:

$ time fetch $(eval echo -O
http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/{$(make -C
/usr/ports/editors/vim -VPATCHFILES | tr ' ' ',')})
...
real    2m2.841s
user    0m1.083s
sys     0m5.238s

Which takes some 2 min of wasted CPU time out of your calculation.
Not that it helps overly in terms of wallclock time.

Using curl with -O cuts this down to:

real    1m5.195s
user    0m0.668s
sys     0m1.479s

And wget -nv to:

real    1m15.426s
user    0m0.845s
sys     0m1.536s


Finally, so you can compare gains whilst taking line speed out of the
picture, phttpget:

real    0m22.673s
user    0m0.530s
sys     0m0.659s


The computer on its link should be able to fetch a tarball in c. 11 s
wallclock time, we have 3.7 MB of payload in patches and the link
manages roughly 355 kB/s.

Now, any takers for "make make fetch more efficient" jobs?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlGlHjAACgkQvmGDOQUufZWaNgCgjERowRAqRo2Rgv8rQOZQiA7f
ahcAoMkjJxeFHGBe7hcZIjZb8rrLh+0/
=XoQd
-----END PGP SIGNATURE-----