X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C882E9.63D9DCA2@onstor-exch02.onstor.net>; Mon, 10 Mar 2008 13:00:29 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C882E9.63D9DCA2"
Content-class: urn:content-classes:message
Subject: RE: system config reset
Date: Mon, 10 Mar 2008 13:00:29 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E08C0FAA6@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E08C0FA98@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: system config reset
Thread-Index: AciC11poquOJP/QvR52Q9BFCjacroAAARXxwAAC2LeAAACK/kAAANA6wAABYjWAAAFOZcAAAiBHAAADJQ5AAAMAkAAAAaOxQ
From: "Raj Kumar" <raj.kumar@onstor.com>
To: "Narain Ramadass" <narain.ramadass@onstor.com>,
	"Larry Scheer" <larry.scheer@onstor.com>,
	"Chris Vandever" <chris.vandever@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>
Cc: "Tim Gardner" <tim.gardner@onstor.com>,
	"Sripal Surendiran (HCL)" <sripal.surendiran@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C882E9.63D9DCA2
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

This is the workflow:

1.	this is a 2 node cluster, broke the cluster; system rebooted
2.	did system config reset, ran  into the issue
3.	did config reset 2 more times, same issue.



_____________________________________________
From: Narain Ramadass=20
Sent: Monday, March 10, 2008 12:52 PM
To: Larry Scheer; Raj Kumar; Chris Vandever; Andy Sharp
Cc: Tim Gardner; Sripal Surendiran (HCL)
Subject: RE: system config reset

Larry's statement may be accurate. FTI has not been ported into cougar
yet and in any case - Sripal is working off of a separate branch and not
working  off of dev. Could be some busted nfxsh code.=20

Raj - the workflow is not fully clear to me you did a flash_install and
you got into OCT (before the system booted). After that when the system
booted - you did not have any config? This works for others - but just
not on your filer? Lemme know when your free and we can take a look at
your filer.

Copying Sripal - the HCL authority on all these things.

Narain.=20

_____________________________________________=20
From: 	Larry Scheer =20
Sent:	Monday, March 10, 2008 12:30 PM
To:	Raj Kumar; Chris Vandever; Andy Sharp
Cc:	Narain Ramadass; Tim Gardner
Subject:	RE: system config reset

This pmtab looks like something the FTI project creates and it should
only happen on BSD system as I am told the FTI code has not been ported
to cougar, yet.

Maybe we have some busted nfxsh code that is getting FTI behavior on
cougars?

Copying Narain, maybe he can shed some light on this.

Larry

_____________________________________________
From: Raj Kumar=20
Sent: Monday, March 10, 2008 12:04 PM
To: Chris Vandever; Larry Scheer; Andy Sharp
Subject: RE: system config reset

Does pmtab gets wiped out during config reset? Didn't think so.

g11r10:~# cat /onstor/etc/pmtab
initwait: /onstor/bin/elog
initwait: /onstor/bin/sscccc

g11r10:~#=20
g11r10:~# ps ax | grep onstor
 4403 ?        Ss     0:00 /onstor/bin/sshd
 4591 ?        Ss     0:00 /onstor/bin/pm
 4603 ?        S      0:12 /onstor/bin/elog
 4612 ?        S      0:00 /onstor/bin/sscccc
 7025 ?        Ss     0:00 /bin/sh /onstor/bin/emrscron -g stats
 7335 ?        S      0:00 /bin/sh /onstor/bin/support.sh -e
nfxsh_connect  -g stats -s --
 9325 ?        Ss     0:00 /bin/sh /onstor/bin/emrscron -g h_res_stats
 9357 pts/0    R+     0:00 grep onstor
g11r10:~#

_____________________________________________
From: Chris Vandever=20
Sent: Monday, March 10, 2008 12:01 PM
To: Raj Kumar; Larry Scheer; Andy Sharp
Subject: RE: system config reset

What apps does ps show running?

Based on the elog, clustering hasn't even started (but as evidenced by
the second reboot, there are more messages missing from the elog than IN
the elog).  The messages that look like they're from clustering are
actually from libcluster being called by an app that starts prior to
clustering.  I see the 'system config reset' at 10:24 with the initial
reboot at 10:28.  The only app that made it to the log is elog:

Mar 10 10:24:37 g11r10 : 0:0:eventd:CRITICAL: Process-EVENT Node: Name
'local', State Down, Msg 'Node going down for reboot! ('system config
reset' issued from nfxsh).'
Mar 10 10:28:10 g11r10 pm: /onstor/bin/elog: finished initialization.
Mar 10 10:30:20 g11r10 : 0:0:cluster2:ERROR: Cluster_RetrieveConfig:
Cluster cfg file /onstor/conf/cluster.conf missing or corrupted or node
intentionally removed from cluster, defaulting to standalone mode, err 0

There's another boot at 10:35, and based on the subsequent reboot we
made it at least as far as sscccc (which is at the end of pmtab just
before sendmail):

Mar 10 10:35:28 g11r10 pm: /onstor/bin/elog: finished initialization.
Mar 10 10:36:28 g11r10 : 0:0:cluster2:ERROR: Cluster_RetrieveConfig:
Cluster cfg file /onstor/conf/cluster.conf missing or corrupted or node
intentionally removed from cluster, defaulting to standalone mode, err 0
Mar 10 10:38:00 g11r10 pm: pm_terminate: child 1732 (/onstor/bin/sscccc)
terminated
Mar 10 10:38:01 g11r10 pm: pm_terminate: child 1713 (/onstor/bin/elog)
terminated

ChrisV
_____________________________________________
From: Raj Kumar=20
Sent: Monday, March 10, 2008 11:41 AM
To: Chris Vandever; Larry Scheer; dl-Cougar
Subject: RE: system config reset

Yes, those messages are in elog after the 2nd attempt of config reset
(didn't see them after the last attempt though). Elogs at
/n/newcorevol/defect_22743

From /etc/onstor/initial-config option 3:

Current Settings:
   Node Name: g11r10
   Date & Time: Mon Mar 10 11:40:47 PDT 2008
   Network Settings:
      Mgmt port 1 IP: 10.2.10.11 NETMASK: 255.255.0.0
      Mgmt port 2 IP: address NETMASK: netmask
      Current default route: 10.2.0.1


Pending changes:

Press 'Enter' to continue...

_____________________________________________
From: Chris Vandever=20
Sent: Monday, March 10, 2008 11:37 AM
To: Raj Kumar; Larry Scheer; dl-Cougar
Subject: RE: system config reset

Can I get the full elogs?  They should contain a message like the
following:

Cluster_RetrieveConfig: Cluster cfg file cluster.conf missing or
corrupted or node intentionally removed from cluster, defaulting to
standalone mode, err 0

When cluster_contrl starts it should create the missing cluster.conf
file UNLESS it is unable to get an IP address for the local host.  Then,
it will complain:

ClusterCtrl_InitUbik: fail to find any IP address

And it will exit.  So, the question is, what happened to the IP address?

ChrisV

_____________________________________________
From: Raj Kumar=20
Sent: Monday, March 10, 2008 11:24 AM
To: Larry Scheer; dl-Cougar
Subject: RE: system config reset

I couldn't cut and paste all those screens but I did set all those
before exiting the script.

_____________________________________________
From: Larry Scheer=20
Sent: Monday, March 10, 2008 11:23 AM
To: Raj Kumar; dl-Cougar
Subject: RE: system config reset

Going strictly by the information you provided it is because you reset
the configuration and exited the configuration script without setting
any configuration information. You have no IP address, hostname, default
route, etc.

_____________________________________________
From: Raj Kumar=20
Sent: Monday, March 10, 2008 11:00 AM
To: dl-Cougar
Subject: FW: system config reset

Any idea?

_____________________________________________
From: Raj Kumar=20
Sent: Monday, March 10, 2008 10:51 AM
To: dl-QA
Subject: system config reset

Hi,

Did config reset on cougar soak g11r10 ( already tried twice). After
reset the filer's services doesn't come up because Cluster DB and
cluster.conf are missing. Any ideas?

g11r10:~# ls -l /onstor/conf/    =20
total 1433
-rw-r--r-- 1 root root 693561 Feb 19 20:43 R4.0.0.0-021908.bom
-rw-r--r-- 1 root root 693237 Feb 14 14:41 R4.0.0.0DBG-021408.bom
lrwxrwxrwx 1 root root     19 Feb 20 13:11 current.bom ->
R4.0.0.0-021908.bom
-rw-r--r-- 1 root root   2046 Feb  6 16:05 emrs_client.pem
-rw-r--r-- 1 root root   1363 Feb  6 16:05 emrs_server.crt
drwx------ 2 root root  12288 Feb  7 20:00 lost+found
lrwxrwxrwx 1 root root     22 Feb 20 13:06 previous.bom ->
R4.0.0.0DBG-021408.bom
-rw-r--r-- 1 root root  53742 Feb  6 16:05 sdm-devcap
g11r10:~#

     1. Configure Administrative Settings


     2. Configure Network Settings


     3. Display Current Settings


     4. Commit Changes


     5. Help


     6. Copy Configuration Files From Secondary Flash


     7. Exit


    Enter Selection: 7

Value Entered is 7

.
Setting up networking....
Configuring network interfaces...SIOCADDRT: Network is unreachable
run-parts: /etc/network/if-up.d/addroutes exited with return code 7
address: Host name lookup failure
ifconfig: `--help' gives usage information.
Failed to bring up eth1.
done.
Starting portmap daemon....
INIT: Entering runlevel: 2
Starting system log daemon: syslogd.
Starting kernel log daemon: klogd.
Starting portmap daemon...Already running..
Starting automounter: loading autofs4 kernel module, no automount maps
defined.
Setting NIS domainname to: NASgateway.
Starting NIS services: ypserv yppasswdd ypxfrd ypbind.
Starting MTA: exim4.
* ALERT: exim paniclog /var/log/exim4/paniclog has non-zero size, mail
system possibly broken
Starting internet superserver: inetd.
Starting OpenBSD Secure Shell server: sshd.
Starting NFS common utilities: statd.
Starting NTP server: ntpd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: crond.
Starting ONStor services: mgmtbus/onstor/bin/emrscron -f=20
 pm.

OnStor GNU/Linux 4.0 g11r10 duart0

g11r10 login: Mar 10 10:46:46 g11r10 : 0:0:cluster2:ERROR:
cluster_iUpdateRecordData: no reply bck -1=20
Mar 10 10:46:47 g11r10 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey:
no reply bck -1=20
Mar 10 10:46:47 g11r10 : 0:0:nfxsh:NOTICE: cmd[0]: elog display enable :
status[11]
Mar 10 10:47:36 g11r10 : 0:0:cluster2:ERROR: cluster_iGetRecordData: no
reply bck -1=20
Mar 10 10:48:09 g11r10 last message repeated 3 times
Mar 10 10:48:19 g11r10 last message repeated 2 times
Mar 10 10:48:29 g11r10 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey:
no reply bck -1=20
Mar 10 10:48:29 g11r10 : 0:0:cluster2:ERROR: cluster_iGetRecordData: no
reply bck -1=20
Mar 10 10:48:39 g11r10 last message repeated 2 times
Mar 10 10:48:50 g11r10 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey:
no reply bck -1=20
Mar 10 10:48:51 g11r10 : 0:0:cluster2:ERROR: cluster_iGetRecordData: no
reply bck -1=20
Mar 10 10:49:01 g11r10 last message repeated 2 times

Thanks.

--kumar :-)


------_=_NextPart_001_01C882E9.63D9DCA2
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7653.38">
<TITLE>RE: system config reset</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">This is the workflow:</FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">1.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> =
<FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">this is a 2 node =
cluster, broke the cluster; system rebooted</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">2.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">did system config reset, ran&nbsp; into the =
issue</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">3.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT> <FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">did config reset 2 more times, =
same issue.</FONT></SPAN>
</P>
<BR>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Narain Ramadass<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Monday, March 10, 2008 =
12:52 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Larry Scheer; Raj Kumar; =
Chris Vandever; Andy Sharp<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Tim Gardner; Sripal =
Surendiran (HCL)<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: system config reset</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">Larry's statement may be accurate. FTI has not been =
ported into cougar yet and in any case - Sripal is working off of a =
separate branch and not working&nbsp; off of dev. Could be some busted =
nfxsh code. </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">Raj - the workflow is not fully clear to me you did a =
flash_install and you got into OCT (before the system booted). After =
that when the system booted - you did not have any config? This works =
for others - but just not on your filer? Lemme know when your free and =
we can take a look at your filer.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">Copying Sripal - the HCL authority on all these =
things.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">Narain. </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D1 =
FACE=3D"Tahoma">_____________________________________________ =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">From: &nbsp;</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Larry Scheer&nbsp; </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Sent:&nbsp;&nbsp;</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Monday, March 10, 2008 12:30 PM</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">To:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Raj Kumar; Chris Vandever; Andy Sharp</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Cc:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Narain Ramadass; Tim Gardner</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Subject:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT>=
</B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT =
SIZE=3D1 FACE=3D"Tahoma">RE: system config reset</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">This pmtab looks like something the FTI project creates =
and it should only happen on BSD system as I am told the FTI code has =
not been ported to cougar, yet.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Maybe we have some busted nfxsh code that is getting FTI =
behavior on cougars?</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Copying Narain, maybe he can shed some light on =
this.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Larry</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Monday, March 10, 2008 =
12:04 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Chris Vandever; Larry =
Scheer; Andy Sharp<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: system config reset</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Does pmtab gets wiped out during config reset? =
Didn&#8217;t think so.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">g11r10:~# cat /onstor/etc/pmtab</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">initwait: /onstor/bin/elog</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">initwait: /onstor/bin/sscccc</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">g11r10:~# </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">g11r10:~# ps ax | grep onstor</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;4403 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
Ss&nbsp;&nbsp;&nbsp;&nbsp; 0:00 /onstor/bin/sshd</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;4591 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
Ss&nbsp;&nbsp;&nbsp;&nbsp; 0:00 /onstor/bin/pm</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;4603 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:12 /onstor/bin/elog</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;4612 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:00 =
/onstor/bin/sscccc</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;7025 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
Ss&nbsp;&nbsp;&nbsp;&nbsp; 0:00 /bin/sh /onstor/bin/emrscron -g =
stats</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;7335 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:00 /bin/sh /onstor/bin/support.sh -e =
nfxsh_connect&nbsp; -g stats -s --</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;9325 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
Ss&nbsp;&nbsp;&nbsp;&nbsp; 0:00 /bin/sh /onstor/bin/emrscron -g =
h_res_stats</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;9357 pts/0&nbsp;&nbsp;&nbsp; =
R+&nbsp;&nbsp;&nbsp;&nbsp; 0:00 grep onstor</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">g11r10:~#</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Chris Vandever<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Monday, March 10, 2008 =
12:01 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar; Larry Scheer; =
Andy Sharp<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: system config reset</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">What apps does ps show running?</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Based on the elog, clustering hasn&#8217;t even started =
(but as evidenced by the second reboot, there are more messages missing =
from the elog than IN the elog).&nbsp; The messages =
that</FONT></SPAN><SPAN LANG=3D"en-us"><I></I></SPAN><SPAN =
LANG=3D"en-us"><I> <FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">look</FONT></I></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial"> like =
they&#8217;re from clustering are actually from libcluster being called =
by an app that starts prior to clustering.&nbsp; I see the &#8216;system =
config reset&#8217; at 10:24 with the initial reboot at 10:28.&nbsp; The =
only app that made it to the log is elog:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Lucida Console">Mar 10 10:24:37 g11r10 : 0:0:eventd:CRITICAL: =
Process-EVENT Node: Name 'local', State Down, Msg 'Node going down for =
reboot! ('system config reset' issued from nfxsh).'</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Lucida Console">Mar 10 10:28:10 g11r10 pm: /onstor/bin/elog: =
finished initialization.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Lucida Console">Mar 10 10:30:20 g11r10 : 0:0:cluster2:ERROR: =
Cluster_RetrieveConfig: Cluster cfg file /onstor/conf/cluster.conf =
missing or corrupted or node intentionally removed from cluster, =
defaulting to standalone mode, err 0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">There&#8217;s another boot at 10:35, and based on the =
subsequent reboot we made it at least as far as sscccc (which is at the =
end of pmtab just before sendmail):</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Lucida Console">Mar 10 10:35:28 g11r10 pm: /onstor/bin/elog: =
finished initialization.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Lucida Console">Mar 10 10:36:28 g11r10 : 0:0:cluster2:ERROR: =
Cluster_RetrieveConfig: Cluster cfg file /onstor/conf/cluster.conf =
missing or corrupted or node intentionally removed from cluster, =
defaulting to standalone mode, err 0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Lucida Console">Mar 10 10:38:00 g11r10 pm: pm_terminate: child =
1732 (/onstor/bin/sscccc) terminated</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Lucida Console">Mar 10 10:38:01 g11r10 pm: pm_terminate: child =
1713 (/onstor/bin/elog) terminated</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">ChrisV</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Monday, March 10, 2008 =
11:41 AM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Chris Vandever; Larry =
Scheer; dl-Cougar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: system config reset</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Yes, those messages are in elog after the =
2</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><SUP><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">nd</FONT></SUP></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial"> attempt =
of config reset (didn&#8217;t see them after the last attempt though). =
Elogs at /n/newcorevol/defect_22743</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">From /etc/onstor/initial-config option =
3:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Current Settings:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp; Node Name: g11r10</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp; Date &amp; Time: Mon Mar 10 11:40:47 PDT =
2008</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp; Network Settings:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Mgmt port 1 IP: 10.2.10.11 =
NETMASK: 255.255.0.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Mgmt port 2 IP: address =
NETMASK: netmask</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Current default route: =
10.2.0.1</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Pending changes:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Press 'Enter' to continue...</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Chris Vandever<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Monday, March 10, 2008 =
11:37 AM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar; Larry Scheer; =
dl-Cougar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: system config reset</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Can I get the full elogs?&nbsp; They should contain a =
message like the following:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Lucida Console">Cluster_RetrieveConfig: Cluster cfg file =
cluster.conf missing or corrupted or node intentionally removed from =
cluster, defaulting to standalone mode, err 0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">When cluster_contrl starts it should create the missing =
cluster.conf file UNLESS it is unable to get an IP address for the local =
host.&nbsp; Then, it will complain:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Lucida Console">ClusterCtrl_InitUbik: fail to find any IP =
address</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">And it will exit.&nbsp; So, the question is, what =
happened to the IP address?</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">ChrisV</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Monday, March 10, 2008 =
11:24 AM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Larry Scheer; =
dl-Cougar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: system config reset</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">I couldn&#8217;t cut and paste all those screens but I =
did set all those before exiting the script.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Larry Scheer<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Monday, March 10, 2008 =
11:23 AM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar; dl-Cougar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: system config reset</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Going strictly by the information you provided it is =
because you reset the configuration and exited the configuration script =
without setting any configuration information. You have no IP address, =
hostname, default route, etc.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Monday, March 10, 2008 =
11:00 AM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Cougar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> FW: system config reset</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Any idea?</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Monday, March 10, 2008 =
10:51 AM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-QA<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> system config reset</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Hi,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Did =
config reset on cougar soak g11r10 ( already tried twice). After reset =
the filer&#8217;s services doesn&#8217;t come up because Cluster DB and =
cluster.conf are missing. Any ideas?</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">g11r10:~# ls -l /onstor/conf/&nbsp;&nbsp;&nbsp;&nbsp; =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">total =
1433</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">-rw-r--r-- 1 root root 693561 Feb 19 20:43 =
R4.0.0.0-021908.bom</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">-rw-r--r-- 1 root root 693237 Feb 14 14:41 =
R4.0.0.0DBG-021408.bom</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">lrwxrwxrwx 1 root root&nbsp;&nbsp;&nbsp;&nbsp; 19 Feb 20 =
13:11 current.bom -&gt; R4.0.0.0-021908.bom</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">-rw-r--r-- 1 root root&nbsp;&nbsp; 2046 Feb&nbsp; 6 16:05 =
emrs_client.pem</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">-rw-r--r-- 1 root root&nbsp;&nbsp; 1363 Feb&nbsp; 6 16:05 =
emrs_server.crt</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">drwx------ 2 root root&nbsp; 12288 Feb&nbsp; 7 20:00 =
lost+found</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">lrwxrwxrwx 1 root root&nbsp;&nbsp;&nbsp;&nbsp; 22 Feb 20 =
13:06 previous.bom -&gt; R4.0.0.0DBG-021408.bom</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">-rw-r--r-- 1 root root&nbsp; 53742 Feb&nbsp; 6 16:05 =
sdm-devcap</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">g11r10:~#</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 1. Configure Administrative =
Settings</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 2. Configure Network =
Settings</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 3. Display Current =
Settings</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 4. Commit =
Changes</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 5. Help</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 6. Copy Configuration Files From =
Secondary Flash</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 7. Exit</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp; Enter Selection: 7</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Value =
Entered is 7</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Setting up networking....</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Configuring network interfaces...SIOCADDRT: Network is =
unreachable</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Arial">run-parts: /etc/network/if-up.d/addroutes exited with =
return code 7</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Arial">address: Host name lookup failure</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Arial">ifconfig: `--help' gives usage =
information.</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Arial">Failed to bring up eth1.</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting portmap daemon....</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">INIT: =
Entering runlevel: 2</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting system log daemon: syslogd.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting kernel log daemon: klogd.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting portmap daemon...Already =
running..</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting automounter: loading autofs4 kernel module, no =
automount maps defined.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Setting NIS domainname to: NASgateway.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting NIS services: ypserv yppasswdd ypxfrd =
ypbind.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting MTA: exim4.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">* =
ALERT: exim paniclog /var/log/exim4/paniclog has non-zero size, mail =
system possibly broken</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting internet superserver: inetd.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting OpenBSD Secure Shell server: =
sshd.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting NFS common utilities: statd.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting NTP server: ntpd.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting deferred execution scheduler: =
atd.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting periodic command scheduler: =
crond.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting ONStor services: mgmtbus/onstor/bin/emrscron -f =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;pm.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">OnStor GNU/Linux 4.0 g11r10 duart0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">g11r10 login: Mar 10 10:46:46 g11r10 : =
0:0:cluster2:ERROR: cluster_iUpdateRecordData: no reply bck -1 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Mar =
10 10:46:47 g11r10 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no =
reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Mar =
10 10:46:47 g11r10 : 0:0:nfxsh:NOTICE: cmd[0]: elog display enable : =
status[11]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Mar =
10 10:47:36 g11r10 : 0:0:cluster2:ERROR: cluster_iGetRecordData: no =
reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Mar =
10 10:48:09 g11r10 last message repeated 3 times</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Mar =
10 10:48:19 g11r10 last message repeated 2 times</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Mar =
10 10:48:29 g11r10 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no =
reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Mar =
10 10:48:29 g11r10 : 0:0:cluster2:ERROR: cluster_iGetRecordData: no =
reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Mar =
10 10:48:39 g11r10 last message repeated 2 times</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Mar =
10 10:48:50 g11r10 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no =
reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Mar =
10 10:48:51 g11r10 : 0:0:cluster2:ERROR: cluster_iGetRecordData: no =
reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Mar =
10 10:49:01 g11r10 last message repeated 2 times</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">Thanks.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Courier =
New">--kumar</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT SIZE=3D2 FACE=3D"Wingdings">J</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

</BODY>
</HTML>
------_=_NextPart_001_01C882E9.63D9DCA2--
