LAM MPI on dual processor opteron box sees only one cpu...
Jeffrey Racine
jracine at maxwell.syr.edu
Mon Apr 12 06:04:31 PDT 2004
Hi Roland.
I do get CPU #1 launched. This is not the problem.
The problem appears to be with the way that current is scheduling.
With mpirun np 2 I get the job running on CPU 0 (two instances on one
proc). However, it turns out that with np 4 I get the job running on CPU
0 and 1 though with 4 instances (and associated overhead). Here is top
for np 4... notice that in the C column it is using both procs.
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU
COMMAND
96090 jracine 131 0 7148K 2172K CPU1 1 0:19 44.53% 44.53%
n_lam
96088 jracine 125 0 7148K 2172K RUN 0 0:18 43.75% 43.75%
n_lam
96089 jracine 136 0 7148K 2172K RUN 1 0:19 42.19% 42.19%
n_lam
96087 jracine 135 0 7188K 2248K RUN 0 0:19 41.41% 41.41%
n_lam
One run (once when I rebooted lam) did allocate the job correctly with
np 2, but this is not in general the case. On other systems I use,
however, they correctly farm out np 2 to CPU 0 and 1...
Thanks, and any suggestions welcome.
-- Jeff
On Sun, 2004-04-11 at 14:20 -0500, Roland Wells wrote:
> Jeffrey,
> I am not familiar with the LAM MPI issue, but in a dual proc box, you
> should also get an additional line towards the bottom in your dmesg,
> similar to:
>
> SMP: AP CPU #1 Launched!
>
> -Roland
> -----Original Message-----
> From: owner-freebsd-cluster at freebsd.org
> [mailto:owner-freebsd-cluster at freebsd.org] On Behalf Of Jeffrey Racine
> Sent: Saturday, April 10, 2004 5:22 PM
> To: freebsd-amd64 at freebsd.org; freebsd-cluster at freebsd.org
> Subject: LAM MPI on dual processor opteron box sees only one cpu...
>
>
> Hi.
>
> I am converging on getting a new dual opteron box running. Now I am
> setting up and testing LAM MPI, however, the OS is not farming out
> the job as expected, and only sees one processor.
>
> This runs fine on RH 7.3 and RH 9.0 both on a cluster and on a dual
> processor PIV desktop. I am running 5-current. Basically, mpirun -np 1
> binaryfile has the same runtime as mpirun -np 2 binaryfile, while on the
> dual PIV box it runs in half the time. When I check top, mpirun -np 2
> both run on CPU 0... here is the relevant portion from top with -np 2...
>
> 9306 jracine 4 0 7188K 2448K sbwait 0 0:03 19.53% 19.53% n_lam
> 29307 jracine 119 0 7148K 2372K CPU0 0 0:03 19.53% 19.53%
> n_lam
>
> I include output from laminfo, dmesg (cpu relevnt info), and lamboot -d
> bhost.lam... any suggestions most appreciated, and thanks in advance!
>
> -- laminfo
>
> LAM/MPI: 7.0.4
> Prefix: /usr/local
> Architecture: amd64-unknown-freebsd5.2
> Configured by: root
> Configured on: Sat Apr 10 11:22:02 EDT 2004
> Configure host: jracine.maxwell.syr.edu
> C bindings: yes
> C++ bindings: yes
> Fortran bindings: yes
> C profiling: yes
> C++ profiling: yes
> Fortran profiling: yes
> ROMIO support: yes
> IMPI support: no
> Debug support: no
> Purify clean: no
> SSI boot: globus (Module v0.5)
> SSI boot: rsh (Module v1.0)
> SSI coll: lam_basic (Module v7.0)
> SSI coll: smp (Module v1.0)
> SSI rpi: crtcp (Module v1.0.1)
> SSI rpi: lamd (Module v7.0)
> SSI rpi: sysv (Module v7.0)
> SSI rpi: tcp (Module v7.0)
> SSI rpi: usysv (Module v7.0)
>
> -- dmesg sees two cpus...
>
> CPU: AMD Opteron(tm) Processor 248 (2205.02-MHz K8-class CPU)
> Origin = "AuthenticAMD" Id = 0xf58 Stepping = 8
>
> Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,
> MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
> AMD Features=0xe0500800<SYSCALL,NX,MMX+,LM,3DNow!+,3DNow!>
> real memory = 3623813120 (3455 MB)
> avail memory = 3494363136 (3332 MB)
> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
> cpu0 (BSP): APIC ID: 0
> cpu1 (AP): APIC ID: 1
>
> -- bhost has the requisite information
>
> 128.230.130.10 cpu=2 user=jracine
>
> -- Here are the results from lamboot -d bhost.lam
>
> -bash-2.05b$ lamboot -d ~/bhost.lam
> n0<29283> ssi:boot: Opening
> n0<29283> ssi:boot: opening module globus
> n0<29283> ssi:boot: initializing module globus
> n0<29283> ssi:boot:globus: globus-job-run not found, globus boot will
> not run n0<29283> ssi:boot: module not available: globus n0<29283>
> ssi:boot: opening module rsh n0<29283> ssi:boot: initializing module rsh
> n0<29283> ssi:boot:rsh: module initializing n0<29283>
> ssi:boot:rsh:agent: rsh n0<29283> ssi:boot:rsh:username: <same>
> n0<29283> ssi:boot:rsh:verbose: 1000 n0<29283> ssi:boot:rsh:algorithm:
> linear n0<29283> ssi:boot:rsh:priority: 10 n0<29283> ssi:boot: module
> available: rsh, priority: 10 n0<29283> ssi:boot: finalizing module
> globus n0<29283> ssi:boot:globus: finalizing n0<29283> ssi:boot: closing
> module globus n0<29283> ssi:boot: Selected boot module rsh
>
> LAM 7.0.4/MPI 2 C++/ROMIO - Indiana University
>
> n0<29283> ssi:boot:base: looking for boot schema in following
> directories:
> n0<29283> ssi:boot:base: <current directory>
> n0<29283> ssi:boot:base: $TROLLIUSHOME/etc
> n0<29283> ssi:boot:base: $LAMHOME/etc
> n0<29283> ssi:boot:base: /usr/local/etc
> n0<29283> ssi:boot:base: looking for boot schema file:
> n0<29283> ssi:boot:base: /home/jracine/bhost.lam
> n0<29283> ssi:boot:base: found boot schema: /home/jracine/bhost.lam
> n0<29283> ssi:boot:rsh: found the following hosts:
> n0<29283> ssi:boot:rsh: n0 jracine.maxwell.syr.edu (cpu=2)
> n0<29283> ssi:boot:rsh: resolved hosts:
> n0<29283> ssi:boot:rsh: n0 jracine.maxwell.syr.edu --> 128.230.130.10
> (origin)n0<29283> ssi:boot:rsh: starting RTE procs
> n0<29283> ssi:boot:base:linear: starting
> n0<29283> ssi:boot:base:server: opening server TCP socket n0<29283>
> ssi:boot:base:server: opened port 49832 n0<29283> ssi:boot:base:linear:
> booting n0 (jracine.maxwell.syr.edu) n0<29283> ssi:boot:rsh: starting
> lamd on (jracine.maxwell.syr.edu) n0<29283> ssi:boot:rsh: starting on n0
> (jracine.maxwell.syr.edu): hboot -t -c lam-conf.lamd -d -I -H
> 128.230.130.10 -P 49832 -n 0 -o 0 n0<29283> ssi:boot:rsh: launching
> locally
> hboot: performing tkill
> hboot: tkill -d
> tkill: setting prefix to (null)
> tkill: setting suffix to (null)
> tkill: got killname
> back: /tmp/lam-jracine at jracine.maxwell.syr.edu/lam-killfile
> tkill: removing socket file ...
> tkill: socket
> file: /tmp/lam-jracine at jracine.maxwell.syr.edu/lam-kernel-socketd
> tkill: removing IO daemon socket file ...
> tkill: IO daemon socket
> file: /tmp/lam-jracine at jracine.maxwell.syr.edu/lam-io-socket
> tkill: f_kill = "/tmp/lam-jracine at jracine.maxwell.syr.edu/lam-killfile"
> tkill: nothing to kill:
> "/tmp/lam-jracine at jracine.maxwell.syr.edu/lam-killfile"
> hboot: booting...
> hboot: fork /usr/local/bin/lamd
> [1] 29286 lamd -H 128.230.130.10 -P 49832 -n 0 -o 0 -d n0<29283>
> ssi:boot:rsh: successfully launched on n0
> (jracine.maxwell.syr.edu)
> n0<29283> ssi:boot:base:server: expecting connection from finite list
> hboot: attempting to execute
> n-1<29286> ssi:boot: Opening
> n-1<29286> ssi:boot: opening module globus
> n-1<29286> ssi:boot: initializing module globus
> n-1<29286> ssi:boot:globus: globus-job-run not found, globus boot will
> not run n-1<29286> ssi:boot: module not available: globus n-1<29286>
> ssi:boot: opening module rsh n-1<29286> ssi:boot: initializing module
> rsh n-1<29286> ssi:boot:rsh: module initializing n-1<29286>
> ssi:boot:rsh:agent: rsh n-1<29286> ssi:boot:rsh:username: <same>
> n-1<29286> ssi:boot:rsh:verbose: 1000 n-1<29286> ssi:boot:rsh:algorithm:
> linear n-1<29286> ssi:boot:rsh:priority: 10 n-1<29286> ssi:boot: module
> available: rsh, priority: 10 n-1<29286> ssi:boot: finalizing module
> globus n-1<29286> ssi:boot:globus: finalizing n-1<29286> ssi:boot:
> closing module globus n-1<29286> ssi:boot: Selected boot module rsh
> n0<29283> ssi:boot:base:server: got connection from 128.230.130.10
> n0<29283> ssi:boot:base:server: this connection is expected (n0)
> n0<29283> ssi:boot:base:server: remote lamd is at 128.230.130.10:50206
> n0<29283> ssi:boot:base:server: closing server socket n0<29283>
> ssi:boot:base:server: connecting to lamd at 128.230.130.10:49833
> n0<29283> ssi:boot:base:server: connected n0<29283>
> ssi:boot:base:server: sending number of links (1) n0<29283>
> ssi:boot:base:server: sending info: n0
> (jracine.maxwell.syr.edu)
> n0<29283> ssi:boot:base:server: finished sending
> n0<29283> ssi:boot:base:server: disconnected from 128.230.130.10:49833
> n0<29283> ssi:boot:base:linear: finished n0<29283> ssi:boot:rsh: all RTE
> procs started n0<29283> ssi:boot:rsh: finalizing n0<29283> ssi:boot:
> Closing n-1<29286> ssi:boot:rsh: finalizing n-1<29286> ssi:boot: Closing
>
>
>
> _______________________________________________
> freebsd-cluster at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-cluster
> To unsubscribe, send any mail to
> "freebsd-cluster-unsubscribe at freebsd.org"
>
More information about the freebsd-amd64
mailing list