Re: Review D38047 ... and then there was one....

I have now fixed this as well.

WARNING, I did this as a manual patch to my tree and then put it back in 
git, I have not tried this EXACT code as is; I am working on some other 
changes  as well so things are a bit in flux; I will attach the diff to 
this, but caveat emptor.

The patch given is to be applied incrementally after the last patch.  
let met know how it works (or if it works!) for you.

On 10/8/24 16:17, David E. Cross wrote:
>
> Ok, I looked a bit into this and for the case of 'getent *' it really 
> is not (currently) a fair comparison to speed.
>
> For 'getent password' the system currently works as follows, for each 
> datasource in the list fully iterate over EVERY datasource, and 
> 'cache' is a datasource, but so is ldap.
>
> What you wind up getting is a list of EVERYTHNG in files, then a list 
> of everything in cache, and then a list of everything in LDAP. (or 
> whatever).   SO every time it will always go back to origin, so 
> caching effectively doesn't matter except to duplicate the data.
>
> I remember this when I was doing the initial development and I looked 
> into ways to NOT have it do it but for some reason I didn't think it 
> was possible without a substantial rewrite, I am taking another look 
> to see if that is still true or if there is a way around it.
>
> Going on my vague (it has been multiple years now), I think in the 
> GENERAL case it is unavoidable.  The way NSCD typically operates is 
> that looked up values are PUSHED into the cache from the client.  That 
> is the client says 'do you have X'? nscd replies 'no', then the CLIENT 
> falls back, does the lookup, get the value and pushes it into nscd.  
> nscd additionally has a 'perform_lookups' flag that will have it do 
> the lookup itself and then tell the client the result.  The 
> interaction of this variable behavior is that there is no way to 
> programatically shortcircuit it without libc knowing how nscd is 
> optionally configured.  If libc knew that nscd would perform the 
> lookups itself then it could for getent type calls just return 
> immediately after the cache layer enumeration.  if libc knew that nscd 
> would NOT perform lookups then it could bypass it and do the normal.
>
>
> I guess I could implement it as follows:
>
> nscd retruns NS_SUCCESS if it performs its own lookups and then in the 
> case of getent NS_SUCCESS is treated as a return step for the cache 
> layer only (since otherwise getent calls are treated as continue 
> otherwise you'd never enumerate anything after files). and NS_NOTFOUND 
> if it doesn't.. and then the libc layer would treat that as a 
> continue.  .. I think that may do it... I need to refamiliarize myself 
> with that code.
>
>
> In the meantime, checking basic lookups (not enumerations) is a more 
> fair test.  Also keep in mind that without [notfound=return] that 
> misses will always fall back to origin, which is probably what you 
> want with nscd in the default configuration, but not with nscd doing 
> its own lookups.
>
> On 10/7/24 11:33, Marek Zarychta wrote:
>>
>> W dniu 7.10.2024 o 07:05, David Cross pisze:
>>
>>> How many entries are in your ldap structure?  I can attempt a replication here
>>
>> Hello David,
>>
>> I will rather not expose it publicly. Whole LDAP directory contains 
>> few thousand entries - and it was was used for the tests mentioned in 
>> this thread.
>>
>> With the filters applied I see below 1k entries, and then lookup with 
>> nsdc running takes: first lookup 0.16s, next lookups 0.09s, while 
>> without nscd it varies from 0.12 to 0.08 - so nscd performs OK.
>>
>> I have your patch applied and I am still testing it with 
>> net/nss-pam-ldapd from ports with patch for login classes applied 
>> (it's present in port but not enabled by default). So far it works 
>> without issues.
>>
>> -- 
>> Marek Zarychta