Speeding up pkg_version and perhaps other port utilities

[Stephen Montgomery-Smith]

Stephen Montgomery-Smith wrote:
> 
> 
> On Sun, 20 May 2007, Doug Barton wrote:
> 
>> Alexander Leidinger wrote:
>>> Quoting Stephen Montgomery-Smith <stephen at math.missouri.edu> (Sat, 19 
>>> May 2007 23:48:52 -0500):
>>>
>>>> On my system, the program pkg_version can double its speed simply by 
>>>> replacing "make -V PKGNAME" by "make BEFOREPORTMK=yes -V PKGNAME"
>>
>> This is very cool! I've already added it to my devel version of 
>> portmaster in several places. I call make a LOT in portmaster, so I 
>> went through every invocation and added this wherever it worked (i.e., 
>> didn't cause a difference in (or absence of) output.
>>
> 
> However just because it works in a majority of cases doesn't mean that 
> it will work all the time.  You just don't know what important thing 
> might be in AFTERPORTMK, which 99% of the time you never need.
> 
> So I see two approaches:
> 
> 1.  Add more .if's to bsd.port.mk so that there is a more efficient 
> "make VARIABLEONLY=yes -V variable-name" that does all the variables, 
> but doesn't calculate any of the targets - this might make things a bit 
> faster, or it might not;

I don't think this will speed things up.  I hacked into the code for 
make, putting conditionals so that when the "-V" flag was set, that make 
wouldn't bother to calculate targets or dependencies or creations, but 
only variables.


> 2.  Profile bsd make and see if there are any bottlenecks.  I bet make 
> was never designed for speed in these kinds of situations.  But this 
> would be a long term project, albeit definitely worth doing.

It looks to me like the variables are stored as a linear list in the 
"make" program.  Thus if you have something like 500 variables (e.g. try 
a "make -d g1 -V XX | grep = | wc -l" in a port) it is going to take 
quite some time to search through all the variables to find one.  This 
is especially a problem for variable assignments using "?=" which first 
has to search to see if the variable is already defined.  And there are 
a lot of "?=" in bsd.port.mk, 298 of them.  I haven't done any profiling 
yet, but I bet that this is what is taking up all the time.  Probably 
the way to solve this is to rewrite src/usr.bin/make/var.c so that it 
uses some more sophisticated way of storing the variables - maybe 
Berkeley databases is the way to go.  I almost get the feeling that if 
make were completely rewritten in perl that it would go faster!