Impact of having a large number of open file descriptors

[Robert Watson]

On Tue, 3 Jun 2008, Kris Kennaway wrote:

>> I think there's also considerable overlap with other kernel event systems, 
>> such as audit, and we might benefit from thinking seriously about enhancing 
>> those event systems rather than introducing a new one.  The design of 
>> fsevents is pretty much entirely dictated by the needs of Spotlight and 
>> later Time Machine.  In particular, it's not clear to me that the 
>> persistency requirements, which are a large part of the fsevents design, 
>> are important to us... or are they?
>
> Yes, I keep forgetting about audit for some reason :)  It might be that this 
> is already good enough for my use case, although having to maintain a path 
> -> inode mapping for millions of files will be potentially onerous (same for 
> kevent anyway though).  Persistency across reboots for unread events would 
> be nice but probably not essential (or worth the trouble).

One interesting design choice in fsevents is that the event record is 
submitted on file close, rather than on file open.  This doesn't properly 
handle writes due to memory mappings of the file, but does mean that you don't 
get the "Odd KDE PDF effect", in which KDE PDF viewer detects the open/write 
but not the close, so opens and refreshes the PDF before the file is 
completely rewritten, leading to a partial rendering.  Audit captures open(2) 
and close(2), but only open(2) has a path on FreeBSD.  This could be changed, 
however, to use a generated path on close(2), if available.

Right now, there are quite a few file systems, especially synthetic ones such 
as procfs and devfs, which don't use the name cache, meaning that the path 
lookup doesn't work for them.  Also, our name cache invalidation is pretty 
strict (to the point of being a performance problem in the NFS client), so 
path generation can be quite unreliable if it involves active directories.  A 
mini-project in which someone improves the reliability of path generation is 
probably a prerequisite for any real work in this area.

Robert N M Watson
Computer Laboratory
University of Cambridge