X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C882F8.C320D9B6@onstor-exch02.onstor.net>; Mon, 10 Mar 2008 14:50:31 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C882F8.C320D9B6"
Content-class: urn:content-classes:message
Subject: RE: system config reset
Date: Mon, 10 Mar 2008 14:50:31 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E08C0FB6B@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E08C0FB68@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: system config reset
Thread-Index: AciC11poquOJP/QvR52Q9BFCjacroAAARXxwAAC2LeAAACK/kAAANA6wAABYjWAAAFOZcAAAiBHAAADJQ5AAAMAkAAAEKhYwAAAOXNA=
References: <BB375AF679D4A34E9CA8DFA650E2B04E08C0FA98@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E08C0FB68@onstor-exch02.onstor.net>
From: "Narain Ramadass" <narain.ramadass@onstor.com>
To: "Raj Kumar" <raj.kumar@onstor.com>,
	"Larry Scheer" <larry.scheer@onstor.com>,
	"Chris Vandever" <chris.vandever@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>
Cc: "Tim Gardner" <tim.gardner@onstor.com>,
	"Sripal Surendiran (HCL)" <sripal.surendiran@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C882F8.C320D9B6
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Sounds like a build problem with system config reset? Just looked at the
system config with Raj and he has pmtab.init labeled as pmtab when these
files (pmtab.all & pmtab.init) shld not even exist on Linux! Filer is up
and running now. This code path would be deprecated when FTI is brought
into dev - so unsure if this problem is to be tracked!=20

Certainly needs to be retried with FTI, but IMHO this problem would not
be seen with FTI.

Narain.


> _____________________________________________=20
> From: 	Raj Kumar =20
> Sent:	Monday, March 10, 2008 2:48 PM
> To:	Narain Ramadass; Larry Scheer; Chris Vandever; Andy Sharp
> Cc:	Tim Gardner; Sripal Surendiran (HCL)
> Subject:	RE: system config reset
>=20
> For now Narain suggested a workflow (copied pmtab.all  to pmtab &
> reboot). After reboot I got OCT and now filer is up. Anyways we still
> have a defect to follow up.
>=20
> _____________________________________________
> From: Narain Ramadass=20
> Sent: Monday, March 10, 2008 12:52 PM
> To: Larry Scheer; Raj Kumar; Chris Vandever; Andy Sharp
> Cc: Tim Gardner; Sripal Surendiran (HCL)
> Subject: RE: system config reset
>=20
> Larry's statement may be accurate. FTI has not been ported into cougar
> yet and in any case - Sripal is working off of a separate branch and
> not working  off of dev. Could be some busted nfxsh code.=20
>=20
> Raj - the workflow is not fully clear to me you did a flash_install
> and you got into OCT (before the system booted). After that when the
> system booted - you did not have any config? This works for others -
> but just not on your filer? Lemme know when your free and we can take
> a look at your filer.
>=20
> Copying Sripal - the HCL authority on all these things.
>=20
> Narain.=20
>=20
> _____________________________________________=20
> From: 	Larry Scheer =20
> Sent:	Monday, March 10, 2008 12:30 PM
> To:	Raj Kumar; Chris Vandever; Andy Sharp
> Cc:	Narain Ramadass; Tim Gardner
> Subject:	RE: system config reset
>=20
> This pmtab looks like something the FTI project creates and it should
> only happen on BSD system as I am told the FTI code has not been
> ported to cougar, yet.
>=20
> Maybe we have some busted nfxsh code that is getting FTI behavior on
> cougars?
>=20
> Copying Narain, maybe he can shed some light on this.
>=20
> Larry
>=20
> _____________________________________________
> From: Raj Kumar=20
> Sent: Monday, March 10, 2008 12:04 PM
> To: Chris Vandever; Larry Scheer; Andy Sharp
> Subject: RE: system config reset
>=20
> Does pmtab gets wiped out during config reset? Didn't think so.
>=20
> g11r10:~# cat /onstor/etc/pmtab
> initwait: /onstor/bin/elog
> initwait: /onstor/bin/sscccc
>=20
> g11r10:~#=20
> g11r10:~# ps ax | grep onstor
>  4403 ?        Ss     0:00 /onstor/bin/sshd
>  4591 ?        Ss     0:00 /onstor/bin/pm
>  4603 ?        S      0:12 /onstor/bin/elog
>  4612 ?        S      0:00 /onstor/bin/sscccc
>  7025 ?        Ss     0:00 /bin/sh /onstor/bin/emrscron -g stats
>  7335 ?        S      0:00 /bin/sh /onstor/bin/support.sh -e
> nfxsh_connect  -g stats -s --
>  9325 ?        Ss     0:00 /bin/sh /onstor/bin/emrscron -g h_res_stats
>  9357 pts/0    R+     0:00 grep onstor
> g11r10:~#
>=20
> _____________________________________________
> From: Chris Vandever=20
> Sent: Monday, March 10, 2008 12:01 PM
> To: Raj Kumar; Larry Scheer; Andy Sharp
> Subject: RE: system config reset
>=20
> What apps does ps show running?
>=20
> Based on the elog, clustering hasn't even started (but as evidenced by
> the second reboot, there are more messages missing from the elog than
> IN the elog).  The messages that look like they're from clustering are
> actually from libcluster being called by an app that starts prior to
> clustering.  I see the 'system config reset' at 10:24 with the initial
> reboot at 10:28.  The only app that made it to the log is elog:
>=20
> Mar 10 10:24:37 g11r10 : 0:0:eventd:CRITICAL: Process-EVENT Node: Name
> 'local', State Down, Msg 'Node going down for reboot! ('system config
> reset' issued from nfxsh).'
> Mar 10 10:28:10 g11r10 pm: /onstor/bin/elog: finished initialization.
> Mar 10 10:30:20 g11r10 : 0:0:cluster2:ERROR: Cluster_RetrieveConfig:
> Cluster cfg file /onstor/conf/cluster.conf missing or corrupted or
> node intentionally removed from cluster, defaulting to standalone
> mode, err 0
>=20
> There's another boot at 10:35, and based on the subsequent reboot we
> made it at least as far as sscccc (which is at the end of pmtab just
> before sendmail):
>=20
> Mar 10 10:35:28 g11r10 pm: /onstor/bin/elog: finished initialization.
> Mar 10 10:36:28 g11r10 : 0:0:cluster2:ERROR: Cluster_RetrieveConfig:
> Cluster cfg file /onstor/conf/cluster.conf missing or corrupted or
> node intentionally removed from cluster, defaulting to standalone
> mode, err 0
> Mar 10 10:38:00 g11r10 pm: pm_terminate: child 1732
> (/onstor/bin/sscccc) terminated
> Mar 10 10:38:01 g11r10 pm: pm_terminate: child 1713 (/onstor/bin/elog)
> terminated
>=20
> ChrisV
> _____________________________________________
> From: Raj Kumar=20
> Sent: Monday, March 10, 2008 11:41 AM
> To: Chris Vandever; Larry Scheer; dl-Cougar
> Subject: RE: system config reset
>=20
> Yes, those messages are in elog after the 2nd attempt of config reset
> (didn't see them after the last attempt though). Elogs at
> /n/newcorevol/defect_22743
>=20
> From /etc/onstor/initial-config option 3:
>=20
> Current Settings:
>    Node Name: g11r10
>    Date & Time: Mon Mar 10 11:40:47 PDT 2008
>    Network Settings:
>       Mgmt port 1 IP: 10.2.10.11 NETMASK: 255.255.0.0
>       Mgmt port 2 IP: address NETMASK: netmask
>       Current default route: 10.2.0.1
>=20
>=20
> Pending changes:
>=20
> Press 'Enter' to continue...
>=20
> _____________________________________________
> From: Chris Vandever=20
> Sent: Monday, March 10, 2008 11:37 AM
> To: Raj Kumar; Larry Scheer; dl-Cougar
> Subject: RE: system config reset
>=20
> Can I get the full elogs?  They should contain a message like the
> following:
>=20
> Cluster_RetrieveConfig: Cluster cfg file cluster.conf missing or
> corrupted or node intentionally removed from cluster, defaulting to
> standalone mode, err 0
>=20
> When cluster_contrl starts it should create the missing cluster.conf
> file UNLESS it is unable to get an IP address for the local host.
> Then, it will complain:
>=20
> ClusterCtrl_InitUbik: fail to find any IP address
>=20
> And it will exit.  So, the question is, what happened to the IP
> address?
>=20
> ChrisV
>=20
> _____________________________________________
> From: Raj Kumar=20
> Sent: Monday, March 10, 2008 11:24 AM
> To: Larry Scheer; dl-Cougar
> Subject: RE: system config reset
>=20
> I couldn't cut and paste all those screens but I did set all those
> before exiting the script.
>=20
> _____________________________________________
> From: Larry Scheer=20
> Sent: Monday, March 10, 2008 11:23 AM
> To: Raj Kumar; dl-Cougar
> Subject: RE: system config reset
>=20
> Going strictly by the information you provided it is because you reset
> the configuration and exited the configuration script without setting
> any configuration information. You have no IP address, hostname,
> default route, etc.
>=20
> _____________________________________________
> From: Raj Kumar=20
> Sent: Monday, March 10, 2008 11:00 AM
> To: dl-Cougar
> Subject: FW: system config reset
>=20
> Any idea?
>=20
> _____________________________________________
> From: Raj Kumar=20
> Sent: Monday, March 10, 2008 10:51 AM
> To: dl-QA
> Subject: system config reset
>=20
> Hi,
>=20
> Did config reset on cougar soak g11r10 ( already tried twice). After
> reset the filer's services doesn't come up because Cluster DB and
> cluster.conf are missing. Any ideas?
>=20
> g11r10:~# ls -l /onstor/conf/    =20
> total 1433
> -rw-r--r-- 1 root root 693561 Feb 19 20:43 R4.0.0.0-021908.bom
> -rw-r--r-- 1 root root 693237 Feb 14 14:41 R4.0.0.0DBG-021408.bom
> lrwxrwxrwx 1 root root     19 Feb 20 13:11 current.bom ->
> R4.0.0.0-021908.bom
> -rw-r--r-- 1 root root   2046 Feb  6 16:05 emrs_client.pem
> -rw-r--r-- 1 root root   1363 Feb  6 16:05 emrs_server.crt
> drwx------ 2 root root  12288 Feb  7 20:00 lost+found
> lrwxrwxrwx 1 root root     22 Feb 20 13:06 previous.bom ->
> R4.0.0.0DBG-021408.bom
> -rw-r--r-- 1 root root  53742 Feb  6 16:05 sdm-devcap
> g11r10:~#
>=20
>      1. Configure Administrative Settings
>=20
>=20
>      2. Configure Network Settings
>=20
>=20
>      3. Display Current Settings
>=20
>=20
>      4. Commit Changes
>=20
>=20
>      5. Help
>=20
>=20
>      6. Copy Configuration Files From Secondary Flash
>=20
>=20
>      7. Exit
>=20
>=20
>     Enter Selection: 7
>=20
> Value Entered is 7
>=20
> .
> Setting up networking....
> Configuring network interfaces...SIOCADDRT: Network is unreachable
> run-parts: /etc/network/if-up.d/addroutes exited with return code 7
> address: Host name lookup failure
> ifconfig: `--help' gives usage information.
> Failed to bring up eth1.
> done.
> Starting portmap daemon....
> INIT: Entering runlevel: 2
> Starting system log daemon: syslogd.
> Starting kernel log daemon: klogd.
> Starting portmap daemon...Already running..
> Starting automounter: loading autofs4 kernel module, no automount maps
> defined.
> Setting NIS domainname to: NASgateway.
> Starting NIS services: ypserv yppasswdd ypxfrd ypbind.
> Starting MTA: exim4.
> * ALERT: exim paniclog /var/log/exim4/paniclog has non-zero size, mail
> system possibly broken
> Starting internet superserver: inetd.
> Starting OpenBSD Secure Shell server: sshd.
> Starting NFS common utilities: statd.
> Starting NTP server: ntpd.
> Starting deferred execution scheduler: atd.
> Starting periodic command scheduler: crond.
> Starting ONStor services: mgmtbus/onstor/bin/emrscron -f=20
>  pm.
>=20
> OnStor GNU/Linux 4.0 g11r10 duart0
>=20
> g11r10 login: Mar 10 10:46:46 g11r10 : 0:0:cluster2:ERROR:
> cluster_iUpdateRecordData: no reply bck -1=20
> Mar 10 10:46:47 g11r10 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey:
> no reply bck -1=20
> Mar 10 10:46:47 g11r10 : 0:0:nfxsh:NOTICE: cmd[0]: elog display enable
> : status[11]
> Mar 10 10:47:36 g11r10 : 0:0:cluster2:ERROR: cluster_iGetRecordData:
> no reply bck -1=20
> Mar 10 10:48:09 g11r10 last message repeated 3 times
> Mar 10 10:48:19 g11r10 last message repeated 2 times
> Mar 10 10:48:29 g11r10 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey:
> no reply bck -1=20
> Mar 10 10:48:29 g11r10 : 0:0:cluster2:ERROR: cluster_iGetRecordData:
> no reply bck -1=20
> Mar 10 10:48:39 g11r10 last message repeated 2 times
> Mar 10 10:48:50 g11r10 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey:
> no reply bck -1=20
> Mar 10 10:48:51 g11r10 : 0:0:cluster2:ERROR: cluster_iGetRecordData:
> no reply bck -1=20
> Mar 10 10:49:01 g11r10 last message repeated 2 times
>=20
> Thanks.
>=20
> --kumar :-)
>=20

------_=_NextPart_001_01C882F8.C320D9B6
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7653.38">
<TITLE>RE: system config reset</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P><FONT COLOR=3D"#0000FF" SIZE=3D2 FACE=3D"Arial">Sounds like a build =
problem with system config reset? Just looked at the system config with =
Raj and he has pmtab.init labeled as pmtab when these files (pmtab.all =
&amp; pmtab.init) shld not even exist on Linux! Filer is up and running =
now. This code path would be deprecated when FTI is brought into dev - =
so unsure if this problem is to be tracked! </FONT></P>

<P><FONT COLOR=3D"#0000FF" SIZE=3D2 FACE=3D"Arial">Certainly needs to be =
retried with FTI, but IMHO this problem would not be seen with =
FTI.</FONT>
</P>

<P><FONT COLOR=3D"#0000FF" SIZE=3D2 FACE=3D"Arial">Narain.</FONT>
</P>
<BR>

<P><FONT SIZE=3D1 =
FACE=3D"Tahoma">_____________________________________________ </FONT>

<BR><B><FONT SIZE=3D1 FACE=3D"Tahoma">From: &nbsp;</FONT></B> <FONT =
SIZE=3D1 FACE=3D"Tahoma">Raj Kumar&nbsp; </FONT>

<BR><B><FONT SIZE=3D1 FACE=3D"Tahoma">Sent:&nbsp;&nbsp;</FONT></B> <FONT =
SIZE=3D1 FACE=3D"Tahoma">Monday, March 10, 2008 2:48 PM</FONT>

<BR><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">To:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></B> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Narain Ramadass; Larry Scheer; Chris Vandever; Andy =
Sharp</FONT>

<BR><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Cc:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></B> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Tim Gardner; Sripal Surendiran (HCL)</FONT>

<BR><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Subject:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT>=
</B> <FONT SIZE=3D1 FACE=3D"Tahoma">RE: system config reset</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">For now Narain =
suggested a workflow (copied pmtab.all&nbsp; to pmtab &amp; reboot). =
After reboot I got OCT and now filer is up. Anyways we still have a =
defect to follow up.</FONT></P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Narain Ramadass<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, March 10, 2008 12:52 PM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Larry Scheer; Raj Kumar; Chris Vandever; Andy Sharp<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Cc:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Tim Gardner; Sripal Surendiran (HCL)<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: system config reset</FONT>
</P>

<P><FONT COLOR=3D"#0000FF" SIZE=3D2 FACE=3D"Arial">Larry's statement may =
be accurate. FTI has not been ported into cougar yet and in any case - =
Sripal is working off of a separate branch and not working&nbsp; off of =
dev. Could be some busted nfxsh code. </FONT></P>

<P><FONT COLOR=3D"#0000FF" SIZE=3D2 FACE=3D"Arial">Raj - the workflow is =
not fully clear to me you did a flash_install and you got into OCT =
(before the system booted). After that when the system booted - you did =
not have any config? This works for others - but just not on your filer? =
Lemme know when your free and we can take a look at your =
filer.</FONT></P>

<P><FONT COLOR=3D"#0000FF" SIZE=3D2 FACE=3D"Arial">Copying Sripal - the =
HCL authority on all these things.</FONT>
</P>

<P><FONT COLOR=3D"#0000FF" SIZE=3D2 FACE=3D"Arial">Narain. </FONT>
</P>

<P><FONT SIZE=3D1 =
FACE=3D"Tahoma">_____________________________________________ </FONT>

<BR><B><FONT SIZE=3D1 FACE=3D"Tahoma">From: &nbsp;</FONT></B> <FONT =
SIZE=3D1 FACE=3D"Tahoma">Larry Scheer&nbsp; </FONT>

<BR><B><FONT SIZE=3D1 FACE=3D"Tahoma">Sent:&nbsp;&nbsp;</FONT></B> <FONT =
SIZE=3D1 FACE=3D"Tahoma">Monday, March 10, 2008 12:30 PM</FONT>

<BR><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">To:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></B> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Raj Kumar; Chris Vandever; Andy Sharp</FONT>

<BR><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Cc:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></B> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Narain Ramadass; Tim Gardner</FONT>

<BR><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Subject:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT>=
</B> <FONT SIZE=3D1 FACE=3D"Tahoma">RE: system config reset</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">This pmtab looks like =
something the FTI project creates and it should only happen on BSD =
system as I am told the FTI code has not been ported to cougar, =
yet.</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Maybe we have some =
busted nfxsh code that is getting FTI behavior on cougars?</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Copying Narain, maybe =
he can shed some light on this.</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Larry</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, March 10, 2008 12:04 PM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Chris Vandever; Larry Scheer; Andy Sharp<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: system config reset</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Does pmtab gets wiped =
out during config reset? Didn&#8217;t think so.</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">g11r10:~# cat =
/onstor/etc/pmtab</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">initwait: =
/onstor/bin/elog</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">initwait: =
/onstor/bin/sscccc</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">g11r10:~# </FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">g11r10:~# ps ax | =
grep onstor</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">&nbsp;4403 =
?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Ss&nbsp;&nbsp;&nbsp;&nbsp; =
0:00 /onstor/bin/sshd</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">&nbsp;4591 =
?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Ss&nbsp;&nbsp;&nbsp;&nbsp; =
0:00 /onstor/bin/pm</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">&nbsp;4603 =
?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:12 /onstor/bin/elog</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">&nbsp;4612 =
?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:00 /onstor/bin/sscccc</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">&nbsp;7025 =
?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Ss&nbsp;&nbsp;&nbsp;&nbsp; =
0:00 /bin/sh /onstor/bin/emrscron -g stats</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">&nbsp;7335 =
?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:00 /bin/sh /onstor/bin/support.sh -e =
nfxsh_connect&nbsp; -g stats -s --</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">&nbsp;9325 =
?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Ss&nbsp;&nbsp;&nbsp;&nbsp; =
0:00 /bin/sh /onstor/bin/emrscron -g h_res_stats</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">&nbsp;9357 =
pts/0&nbsp;&nbsp;&nbsp; R+&nbsp;&nbsp;&nbsp;&nbsp; 0:00 grep =
onstor</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">g11r10:~#</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Chris Vandever<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, March 10, 2008 12:01 PM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Raj Kumar; Larry Scheer; Andy Sharp<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: system config reset</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">What apps does ps =
show running?</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Based on the elog, =
clustering hasn&#8217;t even started (but as evidenced by the second =
reboot, there are more messages missing from the elog than IN the =
elog).&nbsp; The messages that<I> look</I> like they&#8217;re from =
clustering are actually from libcluster being called by an app that =
starts prior to clustering.&nbsp; I see the &#8216;system config =
reset&#8217; at 10:24 with the initial reboot at 10:28.&nbsp; The only =
app that made it to the log is elog:</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Lucida Console">Mar 10 =
10:24:37 g11r10 : 0:0:eventd:CRITICAL: Process-EVENT Node: Name 'local', =
State Down, Msg 'Node going down for reboot! ('system config reset' =
issued from nfxsh).'</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Lucida Console">Mar 10 =
10:28:10 g11r10 pm: /onstor/bin/elog: finished initialization.</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Lucida Console">Mar 10 =
10:30:20 g11r10 : 0:0:cluster2:ERROR: Cluster_RetrieveConfig: Cluster =
cfg file /onstor/conf/cluster.conf missing or corrupted or node =
intentionally removed from cluster, defaulting to standalone mode, err =
0</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">There&#8217;s another =
boot at 10:35, and based on the subsequent reboot we made it at least as =
far as sscccc (which is at the end of pmtab just before =
sendmail):</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Lucida Console">Mar 10 =
10:35:28 g11r10 pm: /onstor/bin/elog: finished initialization.</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Lucida Console">Mar 10 =
10:36:28 g11r10 : 0:0:cluster2:ERROR: Cluster_RetrieveConfig: Cluster =
cfg file /onstor/conf/cluster.conf missing or corrupted or node =
intentionally removed from cluster, defaulting to standalone mode, err =
0</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Lucida Console">Mar 10 =
10:38:00 g11r10 pm: pm_terminate: child 1732 (/onstor/bin/sscccc) =
terminated</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Lucida Console">Mar 10 =
10:38:01 g11r10 pm: pm_terminate: child 1713 (/onstor/bin/elog) =
terminated</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">ChrisV</FONT>

<BR><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, March 10, 2008 11:41 AM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Chris Vandever; Larry Scheer; dl-Cougar<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: system config reset</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Yes, those messages =
are in elog after the 2<SUP>nd</SUP> attempt of config reset =
(didn&#8217;t see them after the last attempt though). Elogs at =
/n/newcorevol/defect_22743</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">From =
/etc/onstor/initial-config option 3:</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Current =
Settings:</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">&nbsp;&nbsp; Node =
Name: g11r10</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">&nbsp;&nbsp; Date =
&amp; Time: Mon Mar 10 11:40:47 PDT 2008</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">&nbsp;&nbsp; Network =
Settings:</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Mgmt port 1 IP: 10.2.10.11 =
NETMASK: 255.255.0.0</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Mgmt port 2 IP: address =
NETMASK: netmask</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Current default route: =
10.2.0.1</FONT>
</P>
<BR>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Pending =
changes:</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Press 'Enter' to =
continue...</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Chris Vandever<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, March 10, 2008 11:37 AM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Raj Kumar; Larry Scheer; dl-Cougar<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: system config reset</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Can I get the full =
elogs?&nbsp; They should contain a message like the following:</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Lucida =
Console">Cluster_RetrieveConfig: Cluster cfg file cluster.conf missing =
or corrupted or node intentionally removed from cluster, defaulting to =
standalone mode, err 0</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">When cluster_contrl =
starts it should create the missing cluster.conf file UNLESS it is =
unable to get an IP address for the local host.&nbsp; Then, it will =
complain:</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Lucida =
Console">ClusterCtrl_InitUbik: fail to find any IP address</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">And it will =
exit.&nbsp; So, the question is, what happened to the IP address?</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">ChrisV</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, March 10, 2008 11:24 AM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Larry Scheer; dl-Cougar<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: system config reset</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">I couldn&#8217;t cut =
and paste all those screens but I did set all those before exiting the =
script.</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Larry Scheer<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, March 10, 2008 11:23 AM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Raj Kumar; dl-Cougar<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: system config reset</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Going strictly by the =
information you provided it is because you reset the configuration and =
exited the configuration script without setting any configuration =
information. You have no IP address, hostname, default route, =
etc.</FONT></P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, March 10, 2008 11:00 AM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> dl-Cougar<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> FW: system config reset</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Any idea?</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, March 10, 2008 10:51 AM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> dl-QA<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> system config reset</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Hi,</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Did config reset on cougar soak g11r10 =
( already tried twice). After reset the filer&#8217;s services =
doesn&#8217;t come up because Cluster DB and cluster.conf are missing. =
Any ideas?</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">g11r10:~# ls -l =
/onstor/conf/&nbsp;&nbsp;&nbsp;&nbsp; </FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">total 1433</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">-rw-r--r-- 1 root root 693561 Feb 19 =
20:43 R4.0.0.0-021908.bom</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">-rw-r--r-- 1 root root 693237 Feb 14 =
14:41 R4.0.0.0DBG-021408.bom</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">lrwxrwxrwx 1 root =
root&nbsp;&nbsp;&nbsp;&nbsp; 19 Feb 20 13:11 current.bom -&gt; =
R4.0.0.0-021908.bom</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">-rw-r--r-- 1 root root&nbsp;&nbsp; =
2046 Feb&nbsp; 6 16:05 emrs_client.pem</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">-rw-r--r-- 1 root root&nbsp;&nbsp; =
1363 Feb&nbsp; 6 16:05 emrs_server.crt</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">drwx------ 2 root root&nbsp; 12288 =
Feb&nbsp; 7 20:00 lost+found</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">lrwxrwxrwx 1 root =
root&nbsp;&nbsp;&nbsp;&nbsp; 22 Feb 20 13:06 previous.bom -&gt; =
R4.0.0.0DBG-021408.bom</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">-rw-r--r-- 1 root root&nbsp; 53742 =
Feb&nbsp; 6 16:05 sdm-devcap</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">g11r10:~#</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 1. Configure =
Administrative Settings</FONT>
</P>
<BR>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 2. Configure =
Network Settings</FONT>
</P>
<BR>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 3. Display =
Current Settings</FONT>
</P>
<BR>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 4. Commit =
Changes</FONT>
</P>
<BR>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 5. Help</FONT>
</P>
<BR>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 6. Copy =
Configuration Files From Secondary Flash</FONT>
</P>
<BR>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp; 7. Exit</FONT>
</P>
<BR>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;&nbsp;&nbsp; Enter Selection: =
7</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Value Entered is 7</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Setting up networking....</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Configuring network =
interfaces...SIOCADDRT: Network is unreachable</FONT>

<BR><B><FONT SIZE=3D2 FACE=3D"Arial">run-parts: =
/etc/network/if-up.d/addroutes exited with return code 7</FONT></B>

<BR><B><FONT SIZE=3D2 FACE=3D"Arial">address: Host name lookup =
failure</FONT></B>

<BR><B><FONT SIZE=3D2 FACE=3D"Arial">ifconfig: `--help' gives usage =
information.</FONT></B>

<BR><B><FONT SIZE=3D2 FACE=3D"Arial">Failed to bring up eth1.</FONT></B>

<BR><FONT SIZE=3D2 FACE=3D"Arial">done.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting portmap daemon....</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">INIT: Entering runlevel: 2</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting system log daemon: =
syslogd.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting kernel log daemon: =
klogd.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting portmap daemon...Already =
running..</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting automounter: loading autofs4 =
kernel module, no automount maps defined.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Setting NIS domainname to: =
NASgateway.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting NIS services: ypserv =
yppasswdd ypxfrd ypbind.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting MTA: exim4.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">* ALERT: exim paniclog =
/var/log/exim4/paniclog has non-zero size, mail system possibly =
broken</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting internet superserver: =
inetd.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting OpenBSD Secure Shell server: =
sshd.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting NFS common utilities: =
statd.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting NTP server: ntpd.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting deferred execution scheduler: =
atd.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting periodic command scheduler: =
crond.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Starting ONStor services: =
mgmtbus/onstor/bin/emrscron -f </FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;pm.</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">OnStor GNU/Linux 4.0 g11r10 =
duart0</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">g11r10 login: Mar 10 10:46:46 g11r10 : =
0:0:cluster2:ERROR: cluster_iUpdateRecordData: no reply bck -1 </FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Mar 10 10:46:47 g11r10 : =
0:0:cluster2:ERROR: cluster_getRecordIdByKey: no reply bck -1 </FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Mar 10 10:46:47 g11r10 : =
0:0:nfxsh:NOTICE: cmd[0]: elog display enable : status[11]</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Mar 10 10:47:36 g11r10 : =
0:0:cluster2:ERROR: cluster_iGetRecordData: no reply bck -1 </FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Mar 10 10:48:09 g11r10 last message =
repeated 3 times</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Mar 10 10:48:19 g11r10 last message =
repeated 2 times</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Mar 10 10:48:29 g11r10 : =
0:0:cluster2:ERROR: cluster_getRecordIdByKey: no reply bck -1 </FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Mar 10 10:48:29 g11r10 : =
0:0:cluster2:ERROR: cluster_iGetRecordData: no reply bck -1 </FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Mar 10 10:48:39 g11r10 last message =
repeated 2 times</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Mar 10 10:48:50 g11r10 : =
0:0:cluster2:ERROR: cluster_getRecordIdByKey: no reply bck -1 </FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Mar 10 10:48:51 g11r10 : =
0:0:cluster2:ERROR: cluster_iGetRecordData: no reply bck -1 </FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Mar 10 10:49:01 g11r10 last message =
repeated 2 times</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Courier New">Thanks.</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Courier New">--kumar</FONT> <FONT SIZE=3D2 =
FACE=3D"Wingdings">J</FONT>
</P>

</BODY>
</HTML>
------_=_NextPart_001_01C882F8.C320D9B6--
