Looking for speed increases in "make index"
Stephen Montgomery-Smith
stephen at math.missouri.edu
Mon May 28 01:17:45 UTC 2007
On Mon, 28 May 2007, Michel Talon wrote:
> Stephen Montgomery-Smith said:
>
>> I suggest rewriting "make" so that variables are only evaluated on a
>> "need to know" basis. ....
>> or "I have tried to do this."
>
> Of course a lot of people have thinked about it, and quickly realized
> that it was not going to work. In the bsd.ports.mk, evaluation of one
> variable may be dependent on some conditional, which may itself be
> dependant on some other variable, appearing as some target. This
> constantly happens in bsd.ports.mk.
>
> If you think about that, you convince yourself that a "reduced make"
> needs to understand targets, needs to understand conditionals, and needs
> to evaluate not only the variable under consideration, but but possibly
> any other. In other words, you need a full blown make.
Actually I have thought quite a lot about this over the last couple of
weeks. You are correct in that the "make" really does have to be a full
blown make. What I am suggesting is something rather different - a rather
sophisticated make that works hard to only that which it really has to do.
And it certainly would require a lot of sophistication, precisely for the
reasons you state.
>
> To gain some performance, a first idea would be to simplify
> bsd.ports.mk. I am convinced that a substantial part of the 4000 lines
> are historical crap which serve no useful purpose. There are tons of
> variables who have probably purely anecdotical interest. There are
> targets which could be happily suppressed. Of course, due to the
> complexity of bsd.ports.mk, rewriting it is certainly not an easy task.
> There is a freebsd port whose aim is to rewrite it, i don't know how
> far they are. The NetBSD people have completely rewritten the system, i
> have no idea if the new one is faster. The OpenBSD people have also
> rewritten bsd.ports.mk, with apparently much faster makefile.
>
I rewrote a bsd.ports.mk in which ALL targets were removed, and all it did
was evalute variables. (I wrote a perl script that did this
automatically.) The net effect was no speed gain. In other words,
processing unnecessary targets is not the problem.
I really have looked at bsd.ports.mk a lot, and given that it is written
in the context of what make requires, it seems to be rather well and
efficiently written. I did find one speed increase in bsd.gnome.mk, which
I am kind of proud of, but even that only gained about a 5-10% speed
increase in "make index."
> A second idea would be to multithread make, since modern machines will
> have a lot of cores. At present make -j <something> forks submakes
> and waits for their completion. This doesn't help for the problem at
> hand. However, multithreading the execution of a single makefile is
> certainly not trivial due to the interdependencies.
>
I don't think multithreaded make will help in this situation.
> Anyways, one of the things which cannot be a big factor is reading
> bsd.ports.mk from hard disk. It is certainly cached in memory when you
> run make index. On the other hand it is so big that it probably doesn't
> fit in cache, or probably only fits in caches of machines generously
> endowed. I have remarked that the difference of execution speed of make
> index between machines of similar speed but very different cache size
> (i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less
> than 10 minutes on the big cache machine versus 30 minutes on the small
> cache one. If the cache effect is indeed dominant, then reducing the
> size of bsd.ports.mk by all possible means would be very beneficial.
>
My suspicion is that the timne taken to read bsd.ports.mk from the hard
disk is a rather small part of what takes it so long. Compare doing a
"make" with a simple "cat" and the time differences are very great. I
think make spends far more time processing the file than reading it.
> By the way an alternative system would be to use something other than
> make to do the job. This is what the Gentoo people are doing, using
> first a python based system, and now a C++ one (paludis). Here also i
> don't have any idea if it is faster.
>
It would be nice to find a solution that doesn't involve redesigning the
whole ports process from scratch.
>
>
>
> --
>
> Michel TALON
>
> _______________________________________________
> freebsd-hackers at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"
>
More information about the freebsd-ports
mailing list