sheevaplug boot from nandfs hangs

[Ronald Klop]

On Sat, 15 Dec 2012 19:18:57 +0100, Ian Lepore  
<freebsd at damnhippie.dyndns.org> wrote:

> On Sat, 2012-12-15 at 18:51 +0100, Ronald Klop wrote:
>> On Sat, 15 Dec 2012 17:39:31 +0100, Ronald Klop
>> <ronald-freebsd8 at klop.yi.org> wrote:
>>
>> > On Thu, 13 Dec 2012 03:06:58 +0100, Ian Lepore
>> > <freebsd at damnhippie.dyndns.org> wrote:
>> >
>> >> On Thu, 2012-12-13 at 02:06 +0100, Ronald Klop wrote:
>> >>> Hello,
>> >>>
>> >>> I succeeded in installing and booting my kernel (SHEEVAPLUG  
>> 10-current
>> >>>  from 22 nov.) from nandfs. I followed all the advise at
>> >>> http://wiki.freebsd.org/NAND. But the rootfs will not mount/start.
>> >>>
>> >>> The end of the bootinfo is this:
>> >>> cryptosoft0: <software crypto>
>> >>> Timecounters tick every 1.000 msec
>> >>> ipfw2 initialized, divert loadable, nat loadable, default to accept,
>> >>> logging disabled
>> >>> usbus0: 480Mbps High Speed USB v2.0
>> >>> ugen0.1: <Marvell> at usbus0
>> >>> uhub0: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on
>> >>> usbus0
>> >>> uhub0: 1 port with 1 removable, self powered
>> >>> Root mount waiting for: usbus0
>> >>> ugen0.2: <USB 2.0> at usbus0
>> >>> umass0: <USB 2.0 USB Flash Drive, class 0/0, rev 2.00/11.00, addr  
>> 2> on
>> >>> usbus0
>> >>> umass0:  SCSI over Bulk-Only; quirks = 0x4000
>> >>> umass0:0:0:-1: Attached to scbus0
>> >>> Trying to mount root from nandfs:/dev/gnand0s.root []...
>> >>> WARNING: NANDFS is considered to be a highly experimental feature in
>> >>> FreeBSD.
>> >>> (probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 0 0 0 24 0
>> >>> (probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an
>> >>> error
>> >>> (probe0:umass-sim0:0:0:0): Retrying command
>> >>> da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
>> >>> da0: <USB 2.0 USB Flash Drive 1100> Removable Direct Access SCSI-0
>> >>> device
>> >>> da0: 40.000MB/s transfers
>> >>> da0: 3894MB (7975296 512 byte sectors: 255H 63S/T 496C)
>> >>>
>> >>> The kernel is responsive because if I remove or insert the usb  
>> stick it
>> >>> displays messages about it. (Not shown above)
>> >>> I'm connected via serial. What could be wrong? What information can  
>> I
>> >>> give
>> >>> more?
>> >>> I installed /dev/gnand0s.root by tarring the content of / on my usb
>> >>> stick
>> >>> to the mounted /dev/gnand0s.root. That looked ok.
>> >>>
>> >>> Regards,
>> >>> Ronald.
>> >>
>> >> Hmm, so the root mount happens, but init never gets started, or never
>> >> gets far enough to show signs of being started.  Mounting root is
>> >> invoked from sys/kern/init_main.c, I guess what I'd do next is add  
>> some
>> >> printfs in there to see if vfs_mountroot() is returning and if so how
>> >> much farther it gets.
>> >>
>> >> Hmmm, if your kernel is built with ALT_BREAK_TO_DEBUGGER then on the
>> >> serial console maybe you can CR ~ ^B and poke around.  Hmmm, or maybe
>> >> not, because I just tried that on my dreamplug and it just spewed  
>> this
>> >> many times:
>> >>
>> >> root at dpcur:/root # ~KDB: enter: Break to debugger
>> >> [ thread pid 10 tid 100002 ]
>> >> Stopped at      kdb_enter+0x48:panic: mtx_lock() by idle thread
>> >> 0xc3593c00 on sleep mutex pmap @
>> >> /local/build/staging/freebsd/dp10/src/sys/arm/arm/pmap.c:3662
>> >> panic: mtx_lock() by idle thread 0xc3593c00 on sleep mutex  
>> eventhandler
>> >> @  
>> /local/build/staging/freebsd/dp10/src/sys/kern/subr_eventhandler.c:251
>> >> KDB: enter: panic
>> >>
>> >> and apparently recursed until it ran out of stack and locked up.
>> >>
>> >> So I guess it'll be printf-debugging to the rescue. :)
>> >>
>> >> I can confirm that this should be working.  I've got a similar setup
>> >> going on Atmel arm chips, although I haven't sync'd up with -current  
>> for
>> >> a while, I'm at r241077 on that project.  Right now I still load the
>> >> kernel from sdcard but root gets mounted from the nand and the system
>> >> runs.  Performance is horrible (this is a low end arm, 180mhz), but  
>> it
>> >> works.
>> >>
>> >>
>> >> nand0: <AT91 Integrated NAND controller> mem 0xe0000000-0xefffffff on
>> >> atmelarm0
>> >> nandbus0: <NAND bus> on nand0
>> >> onand0: <ONFI compliant NAND> on nandbus0
>> >> onand0: Found BBT table for chip
>> >> Timecounters tick every 10.000 msec
>> >> usbus0: 12Mbps Full Speed USB v1.0
>> >> ugen0.1: <Atmel> at usbus0
>> >> uhub0: <Atmel OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on  
>> usbus0
>> >> mmcsd0: 1876MB <SD SU02G 8.0 SN 3406901 MFG 09/2010 by 48 SD> at mmc0
>> >> 22.5MHz/1bit/64-block
>> >> Root mount waiting for: usbus0
>> >> uhub0: 2 ports with 2 removable, self powered
>> >> Root mount waiting for: usbus0
>> >> ugen0.2: <Novatel Inc.> at usbus0
>> >> Root mount waiting for: usbus0
>> >> Root mount waiting for: usbus0
>> >> ugen0.3: <Unknown> at usbus0 (disconnected)
>> >> Trying to mount root from nandfs:/dev/gnand0s.root []...
>> >> WARNING: NANDFS is considered to be a highly experimental feature in
>> >> FreeBSD.
>> >> warning: no time-of-day clock registered, system time will not be set
>> >> accurately
>> >> Starting file system checks:
>> >>
>> >> -- Ian
>> >
>> > Hi,
>> >
>> > Thanks for the hint. Compiling a new kernel with printf now. I saw
>> > bootverbose also prints more information. How do I enable that on ARM
>> > from the bootprompt?
>> > NB: break to debugger works on my Sheevaplug. But I am not familiar  
>> with
>> > its commands to see what it is doing. Can I force a coredump and would
>> > somebody be interested to look at it?
>> >
>> > Ronald.
>>
>> I also defaulted bootverbose to 1 in init_main.c. Now I get this.
>>
>> ipfw2 initialized, divert loadable, nat loadable, default to accept,
>> logging disabled
>> lo0: bpf attached
>> GEOM: new disk gnand0
>> GEOM: new disk gnand.raw0
>> usbus0: 480Mbps High Speed USB v2.0
>> gnand0: slice 00000000-001fffff: u-boot (2047KB)
>> gnand0: slice 00200000-007fffff: fbsd-boot (6143KB)
>> gnand0: slice 00800000-1fffffff: root (516095KB)
>> ugen0.1: <Marvell> at usbus0
>> uhub0: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on  
>> usbus0
>> gnand.raw0: slice 00000000-001fffff: u-boot (2047KB)
>> gnand.raw0: slice 00200000-007fffff: fbsd-boot (6143KB)
>> gnand.raw0: slice 00800000-1fffffff: root (516095KB)
>> uhub0: 1 port with 1 removable, self powered
>> Trying to mount root from nandfs:/dev/gnand0s.root []...
>> WARNING: NANDFS is considered to be a highly experimental feature in
>> FreeBSD.
>> mountroot: unable to remount devfs under /dev (error 2).
>> mountroot: unable to unlink /dev/dev (error 2)
>> after vfs_mountroot()
>> start_init: trying /sbin/init
>>
>> The 'after vfs_mountroot()' is my own printf. I also verified sys_execve
>> of /sbin/init returned 0.
>> How do I go from here?
>
> Wow, this is all very familiar.  I had this exact situation a couple
> months ago as I was trying to get freebsd running on my pico-sam9g45
> eval board.  My printfs showed exactly the same thing, the execve
> returned 0, but I never got any indication that init was actually
> running, nor did it die as near as I could tell, the system just went
> comatose.
>
> I can't quite remember how I tracked down the problem from there.
>
> Oh hey, I just noticed in your verbose output, that "unable to remount
> devfs under /dev", that's significant.  init has this annoying tendancy
> to silently lock up if it can't open devices such as serial lines that
> are mentioned in /etc/ttys (also, if /etc/ttys is missing).
>
> I'll bet you just need to create a /dev mountpoint in your nandfs and
> everything will get better.  There may be other mountpoints (/tmp, /var,
> etc) that didn't get created properly too.
>
> -- Ian

Thank you for your sharp eyes! /dev was missing.
When I did tar cf - | tar xf - to copy world from usb disk to nandfs I  
already had some kind of feeling I was overlooking something.

root at sh10:~ # mount
/dev/gnand0s.root on / (nandfs, local)
devfs on /dev (devfs, local)

Ronald.