AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20090311140238.1cbfcc6b@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:mail.onstor.net
NSV:
SSH:
R:<sandrine.boulanger@onstor.com>,<brian.stark@onstor.com>,<caeli.collins@onstor.com>,<bill.duffy@onstor.com>,<dl-Customer-Engineering@onstor.com>,<ed.kwan@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@exch1.onstor.net/INBOX	0	102AB4F33EBBDB4C91915B145C8E9FB3127D02DC20@exch1.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Wed, 11 Mar 2009 14:02:45 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: Sandrine Boulanger <sandrine.boulanger@onstor.com>
Cc: Brian Stark <brian.stark@onstor.com>, Caeli Collins
 <caeli.collins@onstor.com>, Bill Duffy <bill.duffy@onstor.com>,
 dl-Customer-Engineering <dl-Customer-Engineering@onstor.com>, Ed Kwan
 <ed.kwan@onstor.com>
Subject: Re: can I borrow (again!) cslab6/7 for a cluster upgrade test from
 4.0/1/0 to 4.0.2.0?
Message-ID: <20090311140245.60dcbed3@ripper.onstor.net>
In-Reply-To: <102AB4F33EBBDB4C91915B145C8E9FB3127D02DC20@exch1.onstor.net>
References: <2779531E7C760D4491C96305019FEEB52AC724C9B8@exch1.onstor.net>
	<102AB4F33EBBDB4C91915B145C8E9FB3127D02DC20@exch1.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

OK, in this one, both CF slots are having problems, physical slot 0 is
disabled during bootup right off the bat, and the kernel (which is
loaded off ps0) tries to mount and use the root filesystem off of ps1.
But that has problems as well and the system craps out since it can't
find a useable root filesystem.  But everyone probably knew that
already.


On Wed, 11 Mar 2009 11:44:58 -0700 Sandrine Boulanger
<sandrine.boulanger@onstor.com> wrote:

> The whole system went weird at some point; the leds next to the
> flashes were not lit so I could not figure out who was primary or
> secondary. I'm good now on cslab7. On cslab6, the flash I was booting
> off kept crashing in loop with kernel panic
> 
> Root-NFS: No NFS server available, giving up.
> VFS: Unable to mount root fs via NFS, trying floppy.
> VFS: Cannot open root device "sda1" or unknown-block(2,0)
> Please append a correct "root=" boot option; here are the available
> partitions: Kernel panic - not syncing: VFS: Unable to mount root fs
> on unknown-block(2,0)
> 
> I removed that flash, do you want it? I've attached more console log
> when booting off that bad flash.
> 
> 
> _____________________________________________
> From: Brian Stark
> Sent: Wednesday, March 11, 2009 10:57 AM
> To: Sandrine Boulanger; Caeli Collins; Bill Duffy;
> dl-Customer-Engineering Cc: Ed Kwan; Andy Sharp
> Subject: RE: can I borrow (again!) cslab6/7 for a cluster upgrade
> test from 4.0/1/0 to 4.0.2.0?
> 
> OK, one of the theories that Andy and I were exploring was that the
> kernel actually mounted the secondary flash as the primary.  This
> could potentially happen if the kernel thought there was an issue
> with the real primary.  This fits with your observation that the
> system didn't go through an install wizard and wasn't running the
> version you expected.
> 
> I don't see anything in the SSC console output to suggest the kernel
> actually mounted the secondary as the primary, but maybe Andy has a
> different take.
> 
> On this system, can you send the results of the 'mount' command from
> the Linux prompt?
> 
> 
> 
> _____________________________________________
> From: Sandrine Boulanger
> Sent: Wednesday, March 11, 2009 10:20 AM
> To: Brian Stark; Caeli Collins; Bill Duffy; dl-Customer-Engineering
> Cc: Ed Kwan; Andy Sharp
> Subject: RE: can I borrow (again!) cslab6/7 for a cluster upgrade
> test from 4.0/1/0 to 4.0.2.0?
> 
> 
> 
> [X]
> From: Brian Stark
> Sent: Wednesday, March 11, 2009 10:11 AM
> To: Sandrine Boulanger; Caeli Collins; Bill Duffy;
> dl-Customer-Engineering Cc: Ed Kwan; Andy Sharp
> Subject: Re: can I borrow (again!) cslab6/7 for a cluster upgrade
> test from 4.0/1/0 to 4.0.2.0?
> 
> 
> Were any of the CFs swapped prior to the config reset?
> 
> > no, but the weird thing is, when I logged in after the reboot, it
> > did not go through the install wizard, and it was running a
> > different version so I suspect that for some reason, it went to the
> > secondary flash. On that flash, /dev/sdb is there. So I mounted the
> > other flash, and indeed, /dev/sdb is missing on the other one:
> 
> cslab6:~# mount /dev/sdb1 /mnt
> cslab6:~# ls /mnt/dev/sdb
> ls: /mnt/dev/sdb: No such file or directory
> 
> so I will save that flash.
> Do we have any console messages from cslab6 while it was booting?
> > yes, attached.
>  << File: cslab6-ssc-console >>
> 
> [X]
> From: Sandrine Boulanger
> To: Caeli Collins; Bill Duffy; dl-Customer-Engineering
> Cc: Ed Kwan; Brian Stark
> Sent: Wed Mar 11 10:04:01 2009
> Subject: RE: can I borrow (again!) cslab6/7 for a cluster upgrade
> test from 4.0/1/0 to 4.0.2.0?
> 
> After a system config reset it's working again...
> _____________________________________________
> From: Caeli Collins
> Sent: Wednesday, March 11, 2009 9:57 AM
> To: Sandrine Boulanger; Bill Duffy; dl-Customer-Engineering
> Cc: Ed Kwan; Brian Stark
> Subject: RE: can I borrow (again!) cslab6/7 for a cluster upgrade
> test from 4.0/1/0 to 4.0.2.0? Save that bad flash please - Brian will
> be interested. Caeli
> 
> 
> _____________________________________________
> From:   Sandrine Boulanger
> Sent:   Wednesday, March 11, 2009 9:40 AM
> To:     Bill Duffy; dl-Customer-Engineering
> Cc:     Ed Kwan
> Subject:        RE: can I borrow (again!) cslab6/7 for a cluster
> upgrade test from 4.0/1/0 to 4.0.2.0? Thanks, I'll build a new flash
> then. I need to go back to 4.0.1 anyway for the upgrade.
> _____________________________________________ From: Bill Duffy
> Sent: Wednesday, March 11, 2009 5:35 AM
> To: Sandrine Boulanger; dl-Customer-Engineering
> Cc: Ed Kwan
> Subject: RE: can I borrow (again!) cslab6/7 for a cluster upgrade
> test from 4.0/1/0 to 4.0.2.0? I booted off a known good compact flash
> and copied the customers config over.
> _____________________________________________ From: Sandrine Boulanger
> Sent: Tuesday, March 10, 2009 10:12 PM
> To: Sandrine Boulanger; dl-Customer-Engineering
> Cc: Ed Kwan
> Subject: RE: can I borrow (again!) cslab6/7 for a cluster upgrade
> test from 4.0/1/0 to 4.0.2.0? Cslab6 is having the problem again
> about the /dev/sdb missing. How did we solve it at the customer site
> the other day with mknod? cslab6:~# cslab6:~# ls /dev/sdb
> ls: /dev/sdb: No such file or directory
> cslab6:~#
> _____________________________________________
> From: Sandrine Boulanger
> Sent: Tuesday, March 10, 2009 6:29 PM
> To: dl-Customer-Engineering
> Subject: can I borrow (again!) cslab6/7 for a cluster upgrade test
> from 4.0/1/0 to 4.0.2.0? It should not take more than a day. Let me
> know by tomorrow morning if anybody is using it, thanks.
> 
