The vim port needs a refresh

[Lars Engels]

On Tue, May 28, 2013 at 07:51:37PM +0200, Michael Gmelin wrote:
> On Tue, 28 May 2013 15:16:00 +0100
> RW <rwmaillists at googlemail.com> wrote:
> 
> > On Tue, 28 May 2013 15:14:52 +0200
> > John Marino wrote:
> > 
> > > All
> > > patches only take 74 seconds to download[2] so there is no sympathy
> > > for your obviously single data point anecdote, 
> > 
> > Well at the point you provided one data-point there was only one data
> > point. And it was like pulling teeth to get you to eliminate the
> > alternative explanations. Was it really too much to ask that you
> > provided some actual evidence. 
> > 
> > > you're clearly doing
> > > something wrong.  You need to stop complaining and start think about
> > > folks with slow connections[3] who also rebuild Vim frequently. 
> > 
> > Don't make things up. I never said anything about frequent rebuilds,
> > the patches all get redownloaded on the next rebuild. 
> 
> The real issue is not the number of patches, but the fact that every
> single patch is downloaded by invoking the fetch(1) command, creating
> lots of overhead not limited to the fetch command itself. The ports
> system wasn't designed for such an amount of distfiles in a single
> port I guess.
> 
> I just timed fetching the patches through ports vs. fetching over
> HTTP/1.1 using ftp/curl vs calling fetch directly. The VIM tarball was
> already downloaded, so this is really just the patches (plus
> downloading 6mb is barely noticeable on a fast line). It's a slow
> machine on a fast line.
> 
> Fetch:
> [user at server /usr/ports/editors/vim]$ time sudo make fetch
> ....
> real    4m57.327s
> user    0m17.010s
> sys     0m39.588s
> 
> Curl:
> [user at server /tmp]$ longcurlcommandline 
> ....
> real    0m15.291s
> user    0m0.026s
> sys     0m0.272s
> 
> Fetch on the command line (after initial make fetch, so this is only
> measuring transmission of the files):
> cd /usr/ports/editors/distfiles
> time for name in 7.3.*; do
>   fetch http://artfiles.org/vim.org/patches/7.3/$name
> done
> ....
> real    1m25.329s
> user    0m0.660s
> sys     0m3.174s
> 
> So just the fact we're invoking fetch for every file costs us about one
> minute - I assume the time lost is much bigger on a slow line with
> long latency. The remaining 3.5 minutes are spent somewhere in the
> ports infrastructure and clearly depend on the performance of the
> machine used. For comparison I timed "make fetch" on a reasonably fast
> server (good IO, fast datacenter connection), make fetch still took
> about 120 seconds(!).
> 
> So the bottomline is:
> - Using HTTP/1.1 and keepalive could safe a lot of time
> - The ports infrastructure creates a lot of overhead per patch file
> 
> Maybe there's something we can do to improve the situation.
> 
> Cheers,
> Michael
> 
> PS: I don't use vim myself and have no stake in this discussion
> whatsoever.

Someone in this thread proposed to change the port to use phttpget, so I
gave it a try using a German mirror nearby with 6 Mbit/s downlink:

$ time /usr/libexec/phttpget ftp.vim.ossmirror.de $(eval echo /pub/vim/patches/7.3/{$(make -C /usr/ports/editors/vim -VPATCHFILES | sed 's/\ /,/g')})
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.001: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.002: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.003: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.004: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.005: 200 OK
[...]
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.974: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.984: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.985: 200 OK
http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.1000: 200 OK

real	0m12.509s
user	0m0.154s
sys	0m0.089s


That's really nice! 

Compare this to the current version using fetch(1):

 time make PATCH_SITES=http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/ fetch
===>  Found saved configuration for vim-7.3.669_1
===>   vim-7.3.1014 depends on file: /usr/local/sbin/pkg - found
=> 7.3.002 doesn't seem to exist in /usr/ports/distfiles/vim.
=> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.002
7.3.002                                       100% of 1610  B   16 MBps 00m00s
=> 7.3.003 doesn't seem to exist in /usr/ports/distfiles/vim.
=> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.003
7.3.003                                       100% of 1299  B 1281 kBps 00m00s
=> 7.3.004 doesn't seem to exist in /usr/ports/distfiles/vim.
[...]
=> 7.3.984 doesn't seem to exist in /usr/ports/distfiles/vim.
=> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.984
7.3.984                                       100% of 1706  B 2852 kBps 00m00s
=> 7.3.985 doesn't seem to exist in /usr/ports/distfiles/vim.
=> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.985
7.3.985                                       100% of 1691  B   14 MBps 00m00s
=> 7.3.1000 doesn't seem to exist in /usr/ports/distfiles/vim.
=> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.1000
7.3.1000                                      100% of 1637  B 1715 kBps 00m00s
===> Fetching all distfiles required by vim-7.3.1014 for building

Total time                              : 3:48.55s
CPU utilisation (percentage)            : 54.5%


That's much slower using the same mirror.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-ports/attachments/20130528/1abbb22c/attachment.sig>