regarding issues with futexes and multiple threads

[Alexander Best]

On Fri Mar  4 11, Chagin Dmitry wrote:
> On Fri, Mar 04, 2011 at 11:08:43AM +0000, Alexander Best wrote:
> > On Fri Mar  4 11, Chagin Dmitry wrote:
> > > On Thu, Mar 03, 2011 at 09:20:51PM +0000, Alexander Best wrote:
> > > > hi there,
> > > > 
> > > > i've been investigating this issue for quite a while now and simply wanted to
> > > > post some measurements i did.
> > > > 
> > > > basically there's a massive slowdown when a process triggers a lot of threads
> > > > which are doing futex operations. the following example is from darren hart's
> > > > futex test suite (performance/). the slowdown seems to SMP related. here are
> > > > the measurements running with SMP enabled and with kern.smp.disabled=1:
> > > > 
> > > 
> > > 1) slowdown is expected behaviour.
> > 
> > even if the slowdown is to be expected, looking at the screenshot with the
> > top(1) output it seems obvious that there are issues with the current futex
> > implementation.
> > 
> 
> why do you think it's a problem with the futexes?

i added the following debug printf's to linux_futex.c:

diff --git a/sys/compat/linux/linux_futex.c b/sys/compat/linux/linux_futex.c
index b69936c..3d5576f 100644
--- a/sys/compat/linux/linux_futex.c
+++ b/sys/compat/linux/linux_futex.c
@@ -510,6 +510,8 @@ linux_sys_futex(struct thread *td, struct linux_sys_futex_args *args)
                }
 
                error = futex_wait(f, wp, args->timeout, args->val3);
+               if (error)
+                       printf("LINUX_FUTEX_WAIT_BITSET finished with error = %d\n", error);
                break;
 
        case LINUX_FUTEX_WAKE:
@@ -541,6 +543,8 @@ linux_sys_futex(struct thread *td, struct linux_sys_futex_args *args)
                }
                td->td_retval[0] = futex_wake(f, args->val, args->val3);
                futex_put(f, NULL);
+               if (error)
+                       printf("LINUX_FUTEX_WAKE_BITSET finished with error = %d\n", error);
                break;
 
        case LINUX_FUTEX_CMP_REQUEUE:
@@ -601,6 +605,8 @@ linux_sys_futex(struct thread *td, struct linux_sys_futex_args *args)
                td->td_retval[0] = futex_requeue(f, args->val, f2, nrwake);
                futex_put(f2, NULL);
                futex_put(f, NULL);
+               if (error)
+                       printf("LINUX_FUTEX_CMP_REQUEUE finished with error = %d\n", error);
                break;
 
        case LINUX_FUTEX_WAKE_OP:
@@ -664,6 +670,8 @@ linux_sys_futex(struct thread *td, struct linux_sys_futex_args *args)
                        futex_put(f2, NULL);
                futex_put(f, NULL);
                td->td_retval[0] = ret;
+               if (error)
+                       printf("LINUX_FUTEX_WAKE_OP finished with error = %d\n", error);
                break;
 
        case LINUX_FUTEX_LOCK_PI:

...then i ran chrome with some flash website. this was the dmesg output (well
part of it):

LINUX_FUTEX_WAIT_BITSET finished with error = 60
LINUX_FUTEX_WAIT_BITSET finished with error = 60
LINUX_FUTEX_WAIT_BITSET finished with error = 60
LINUX_FUTEX_WAIT_BITSET finished with error = 60
LINUX_FUTEX_WAIT_BITSET finished with error = 60
LINUX_FUTEX_WAITL_IBNIUTXS_EFTU TfEiXn_iWsAhIeTd_ BwIiTtShE Te rfrionri s=h ed6 0wi
th error = 60
LINUX_FUTEX_WAIT_BITSET finished with error = 60
LINUX_FUTEX_WAIT_BITSET finished with error = 60
LINUX_FUTEX_WAIT_BITSET finished with error = 60
LINUX_FUTEX_WAIT_BITSET finished with error = 60
LINUX_FUTEX_WAIT_BITSET finished with error = 60
LINUX_FUTEX_WAIT_BITSET finished with error = 60
LINUX_FUTEX_WAIT_BITSET finished with error = 60
LINUX_FUTEX_WAIT_BITSET finished with error = 60
LINUX_FUTEX_WAIT_BITSET finished with error = 4
LINUX_FUTEX_WAIT_BITSET finisLhIeNdU Xw_iFtUhT EeXr_rWoArI T=_ B4IT
SET finished with erroLrI N=U X4_F
UTEX_WAIT_BITSET finLiIsNhUeXd_ FwUiTtEhX _eWrArIoTr_ B=I T4SE
T finished with errLoIrN U=X _4FU
TEX_WAIT_BITSET finLiIsNhUeXd_ FwUiTtEhX _eWrArIoTr_ B=I T4SE
T finished with erLrIoNrU X=_ F4UT
EX_WAIT_BITSET finiLsIhNeUdX _wFiUtThE Xe_rWrAoIrT _=B I4TS
ET finished with error = 4

... running the futex testsuite with #threads 1 and 32 showed no error = *
output, but i'll try running the suite with more testcases in a minute.

cheers.
alex

> 
> > > 2) is it possible to run this test suite on FreeBSD?
> > 
> > i used emulators/linux_dist-gentoo-stage3 to compile the testsuite and then ran
> > it natively under freebsd. if you want to i can send you the file. it's
> > 
> > futex_wait: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
> > 
> 
> ah, tests directly calls sys_futex syscall, so can't be used for umtx/futex comparison :)
> It would be great to have the same set of tests to compare the linuxulator threads
> and the native.
> 
> -- 
> Have fun!
> chd



-- 
a13x