Re: Review D38047 ... and then there was one....

Ok, I looked a bit into this and for the case of 'getent *' it really is 
not (currently) a fair comparison to speed.

For 'getent password' the system currently works as follows, for each 
datasource in the list fully iterate over EVERY datasource, and 'cache' 
is a datasource, but so is ldap.

What you wind up getting is a list of EVERYTHNG in files, then a list of 
everything in cache, and then a list of everything in LDAP. (or 
whatever).   SO every time it will always go back to origin, so caching 
effectively doesn't matter except to duplicate the data.

I remember this when I was doing the initial development and I looked 
into ways to NOT have it do it but for some reason I didn't think it was 
possible without a substantial rewrite, I am taking another look to see 
if that is still true or if there is a way around it.

Going on my vague (it has been multiple years now), I think in the 
GENERAL case it is unavoidable.  The way NSCD typically operates is that 
looked up values are PUSHED into the cache from the client.  That is the 
client says 'do you have X'? nscd replies 'no', then the CLIENT falls 
back, does the lookup, get the value and pushes it into nscd.  nscd 
additionally has a 'perform_lookups' flag that will have it do the 
lookup itself and then tell the client the result.  The interaction of 
this variable behavior is that there is no way to programatically 
shortcircuit it without libc knowing how nscd is optionally configured.  
If libc knew that nscd would perform the lookups itself then it could 
for getent type calls just return immediately after the cache layer 
enumeration.  if libc knew that nscd would NOT perform lookups then it 
could bypass it and do the normal.


I guess I could implement it as follows:

nscd retruns NS_SUCCESS if it performs its own lookups and then in the 
case of getent NS_SUCCESS is treated as a return step for the cache 
layer only (since otherwise getent calls are treated as continue 
otherwise you'd never enumerate anything after files). and NS_NOTFOUND 
if it doesn't.. and then the libc layer would treat that as a continue.  
.. I think that may do it... I need to refamiliarize myself with that code.


In the meantime, checking basic lookups (not enumerations) is a more 
fair test.  Also keep in mind that without [notfound=return] that misses 
will always fall back to origin, which is probably what you want with 
nscd in the default configuration, but not with nscd doing its own lookups.

On 10/7/24 11:33, Marek Zarychta wrote:
>
> W dniu 7.10.2024 o 07:05, David Cross pisze:
>
>> How many entries are in your ldap structure?  I can attempt a replication here
>
> Hello David,
>
> I will rather not expose it publicly. Whole LDAP directory contains 
> few thousand entries - and it was was used for the tests mentioned in 
> this thread.
>
> With the filters applied I see below 1k entries, and then lookup with 
> nsdc running takes: first lookup 0.16s, next lookups 0.09s, while 
> without nscd it varies from 0.12 to 0.08 - so nscd performs OK.
>
> I have your patch applied and I am still testing it with 
> net/nss-pam-ldapd from ports with patch for login classes applied 
> (it's present in port but not enabled by default). So far it works 
> without issues.
>
> -- 
> Marek Zarychta