Locking fundamentals

[Attilio Rao]

2006/12/21, Suleiman Souhlal <ssouhlal at freebsd.org>:
> Attilio Rao wrote:
> > 2006/12/20, Duane Whitty <duane at dwlabs.ca>:
> >
> >> Hello again,
> >>
> >> It seems to me that understanding locking holds the key to
> >> understanding fbsd internals.
> >>
> >> Could someone review my understanding of fbsd locking fundamentals.
> >> (No assertions here, just questions)
> >>
> >>     lock_mgr
> >> --------------------
> >>  mutexes|sx_lock
> >> -------------------    ^
> >> atomic | mem barriers  |
> >
> >
> > Our current locking hierarchy is basically different:
> >
> > III level: lockmgr - sema - sx
> > II level: mutex (sleep/spin/pool) - rwlock - refcount - cv - msleep
> > I level: atomic instructions - memory barriers - sleepqueues/turnstiles
> >
> > (a lower lever means that the upper layer primitives use it as a base.
> > ie: sx locks are build using 1 pool
> > mutex and 2 condition variables).
> >
> > This scheme is far from being perfect due to the presence of 'level 3
> > primitives' which should never exist.
> > Currently, there is an ongoing efforts to take all the top layer
> > primitives to the level II.
> >
> > On the other side, level I primitives should never be used directly by
> > kernel code, but should only be used as a bottom layer for
> > syncronizing primitives. All you need to care is in the layer 2 and 3
> > (and possibly should switch to layer 2).
>
> I disagree. There are many uses of atomic operations in the kernel that are not for locks or refcounts. It's a bad idea to use locks if you can achieve the same thing locklessly, with atomic operations.

I can agree with you about this but atomic instructions/memory
barriers should be used very carefully (and if you know what you are
going to do). It is very simple to write down a wrong semantic using
them.

> I would personally also add "critical sections" (critical_enter()/critical_exit()) at level I. They can be used instead of locks when you know your data will only be accessed on one CPU, and you only need to protect it from (non-FAST) interrupt handlers.

>From this point of view, we would also add sched_pin()/sched_unpin()
which are used in order to avoid thread migration between CPUs
(particulary helpfull in the case we have to access safely to some
per-CPU datas).
However, probabilly one of the most important usage we do of
critical_section is in the spin mutex implementation (which linked to
interrupt disabling would avoid deadlock in spin mutex code).

Attilio


-- 
Peace can only be achieved by understanding - A. Einstein