size of source tree

[Ian Smith]

In freebsd-questions Digest, Vol 514, Issue 11, Message: 1
On Sat, 12 Apr 2014 13:04:09 +0100 Arthur Chance <freebsd at qeng-ho.org> wrote:
 > On 12/04/2014 12:50, Matthew Seaman wrote:
 > > On 12/04/2014 10:09, Erich Dollansky wrote:
 > >> Hi,
 > >>
 > >> On Sat, 12 Apr 2014 09:26:23 +0100
 > >> Matthew Seaman <m.seaman at infracaninophile.co.uk> wrote:
 > >>
 > >>> On 12/04/2014 04:54, Erich Dollansky wrote:
 > >>>> The source tree dated 26.03.14 has a size of some 899GM while
 > >>>> today's source tree has some 1.8GB.
 > >>>>
 > >>>> Why is it suddenly so huge?
 > >>>>
 > >>>
 > >>> How were these source trees obtained?  A checkout from SVN will have a
 > >>> .svn directory containing pristine copies of all of the files, which
 > >>> will pretty much double the space requirement.
 > >>>
 > >> both have been obtained with svn. The smaller one was started last year
 > >> with the then current version of svn.
 > >>
 > >> Is there an option to turn this behaviour of keeping a copy off?
 > >
 > > Not if you want to use SVN.  You can use freebsd-update to get system
 > > sources -- but only from a release branch.
 > 
 > Wasn't svnup (${PORTS}/net/svnup) designed for this? It fetches from svn 
 > respositories but doesn't use the .svn directory.

Indeed.  I did some early testing (c. v0.56 IIRC) of svnup and sent John 
Mehr (cc'd) some observations and a few suggestions on a then 9.1 system 
that subsequently died.  Early versions only used svn: protocol and were 
pretty slow (and CPU-intensive) from here, before John rewrote the tree 
indexing code and added http: and https: protocols.

So the other night I installed it from 9.2-R ports, getting v1.0 (since 
upgraded to fix a bug with 10.0 I see).  I fetched the 'release' target 
to a work area so I could compare it with my virgin 9.2-R sources but 
silly me hadn't noticed that target fetched base/releng/9.2 rather than 
base/release/9.2.0, so all the ID headers were different of course :)

So I blew that away and did it again with release/9.2.0, after which 
diff -r /usr/src /tmp/release returned nothing, well passing that test.

Both times took about two hours to pull the sources (using https:) which 
still seemed kinda slow, with svnup never using over ~7% CPU, mostly 
half that, so I don't really understand why it takes tat long, unless 
the RTT to svn0.us-west.freebsd.org of ~215ms might be a factor?

I haven't tried svn for comparison but would expect to get more than 
~170KB/s out of my 8-10Mbps otherwise idle link.  Maybe someone could
comment on that, though I notice in the very next digest message:

 > From: Lena at lena.kiev.ua
 > [..]
 > rm -rf /usr/src
 > svn export svn://svn0.us-east.FreeBSD.org/base/releng/8.4 /usr/src
 > 
 > Takes about 30 minutes.

To which Erich later responded:

 > At your place, yes, not where I am most of the time.

So Erich, how does an initial two hours sound to you, to save 900MB?  
You might do better from Europe to us-east than from here, dunno.

However a subsequent repeat run (updating nothing) took only a bit over 
2 minutes, confirming earlier experience that the initial whole-tree 
fetch is slow but null or small incremental updates are quite fast 
enough for its intended use - ie a cvsup replacement for non-developers 
who want to upgrade their sources, track stable and such without the 
need for pushing patches (ie committing); for designed use I think it's
a useful and very lightweight tool; I'd recommend having a play with it.

As for overheads, svnup just keeps a full index of the src repository 
(which for 9.2.0 is 800MiB) in
 -rw-r--r--  1 root  wheel   3.5M Apr 14 02:37 /var/tmp/svnup/release

which is seriously light-weight vs the size doubling of /usr/src/.svn

I haven't tried svnup for ports; portsnap works well and quickly for me.

cheers, Ian