repeatedly opening the same .so(s) is slow?

[Eric Schuele]

Peter Jeremy wrote:
> On Mon, 2006-Mar-20 10:49:57 -0600, Eric Schuele wrote:
>> I have a port (gnucash) which takes 3-4 minutes to open on a 2.6GHz 
>> machine.  It used to take 15-20 seconds till all of the libtool changes.
> 
> It takes 15 minutes on a -current Athlon XP-1800 and about 2 minutes on
> a 2.2GHz AMD64 running -stable.
> 
>> I have no idea if the symptom is related to libtool or not.
> 
> I initially thought it was libtool related but now I'm uncertain.  I
> didn't just upgrade libtool, I upgraded quite a few other ports at the
> same time.  On the not-libtool side, ade@ says that he hasn't seen this
> behaviour with other libtool/libltdl ports and I've found that guile
> include it's own libltdl code (based on libtool).  I'm not sure if this
> is gnucash specific or affects other guile applications.
> 
>> Using truss, I can see that gnucash/guile is trying to open a dozen or 
>> two files, repeatedly.  It fails attempting to open it the first few 
>> times everytime it tries to access it, because it is traversing the 
>> LD_LIBRARY_PATH:
> 
> Worse than that, it's expanding LD_LIBRARY_PATH using additional
> paths embedded in the .la files that it's opening.
> 
>> Now I said a dozen or two files repeatedly.  It is 12-20 files maybe... 
>> but it is attempting to open them *hundreds of thousands of times*!  It 
>> goes on and on and on...
> 
> I took a complete ktrace of the startup and there are 24e6 NAMI events
> with the top files tested 2,000,000 times.
> 
>>  I have 
>> thought of placing symlinks in the folder(s) where it first looks for 
>> any given file, to make sure it finds the file... but this does not seem 
>> quite right either.
> 
> It's definitely a hack.  I tried something like this and it didn't
> help much.  The code still wants to open libraries multiple times.
> 
> I've been looking at adding caching to lt_dlopenext() and my first
> attempt went much faster but blew up because I wasn't correctly
> handling open/close/open sequences (libm is opened and then closed
> 42,000 times).  I think this is the way forward but need to find
> the time to understand ltdl.[ch] (~4800 lines).
> 
>> What I'm wondering is.... what is the lists opinion on how to best fix 
>> this type of a problem.  Is this even the cause of my long startup?
> 
> Any system calls involving opening pathnames are expensive, even with
> the namei cache.  Having 4 orders of magnitude too many is a destinct
> problem.
> 
>> I have spoken with one or two of the gnucash devs, they seem to think 
>> this is unique to FreeBSD, meaning they have not seen this problem on 
>> any other platform.  They said it might have to do with how FreeBSD 
>> handles how files are opened up many times recursively!?
> 
> Possibly Linux can more efficiently handle opening a non-existent file
> but the underlying problem is that there are far too many system calls
> being executed during the gnucash startup.  It would be interesting to
> get a truss of gnuash starting on another OS (unfortunately, I don't
> have access to any Linux systems) and/or some other guile applications.
> 

FWIW:
I spoke with some folks on a gnucash channel.  They ran a trace for me 
on gnucash, and it only attempted to load files 17 times or so.  Each 
time it loaded a file it hung onto it.  Sounds like the 'close' is not 
releasing the library like it does on fbsd.  Which is how it 'should' work.

>> If there is a more appropriate list, please let me know.
> 
> -ports may be better.
> 


-- 
Regards,
Eric