X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C88615.8FB74246@onstor-exch02.onstor.net>; Fri, 14 Mar 2008 13:54:14 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C88615.8FB74246"
Content-class: urn:content-classes:message
Subject: strerror inconsistency?
Date: Fri, 14 Mar 2008 13:54:14 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E03E9A6C5@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: strerror inconsistency?
Thread-Index: AciGEbum16WVJI61SSC0JKPu+9ujKAAATF1QAAAvabA=
From: "Chris Vandever" <chris.vandever@onstor.com>
To: "Andy Sharp" <andy.sharp@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C88615.8FB74246
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

John K. got the following on 2 different bobcat linux nodes supposedly
running the same release:

=20

Mar 14 13:23:58 g1r5 : 0:0:cluster2:INFO: Cluster_SendMsgSock: sendto
failed, code 9 (Network is down)

=20

Mar 14 13:24:12 g5r3 : 0:0:cluster2:INFO: Cluster_SendMsgSock: sendto
failed, code 9 (Host is down)

=20

The code that generates this is:

=20

        errno =3D 0;

        stat =3D sendto(sock, msg, msgLen, 0, (struct sockaddr*)&to,
sizeof(to));

        if (stat !=3D msgLen) {

            /* The send failed, so retry it.

             */

            if (stat < 0) {

                CLUSTER_INFO(("%s: sendto failed, code %d (%s)\n",

                              __FUNCTION__,

                              errno,

                              strerror(errno)));

=20

So, (1) Why do the two nodes translate an errno of 9 differently?  (2)
Why do they translate 9 to ENETDOWN or EHOSTDOWN when 9 is EBADF?  I can
believe that perhaps I need to stash a copy of errno into a local
variable in case somewhere in the bowels of CLUSTER_INFO() (which maps
to an elog) we inadvertently step on errno, but I'd kind of expect the
printed int and string to at least be consistent.  Sometimes I can be
pretty dense, however...  :-(

=20

ChrisV

________________________________

From: Chris Vandever=20
Sent: Friday, March 14, 2008 1:40 PM
To: John Keiffer
Cc: dl-QA
Subject: RE: Bobcat cluster help?

=20

There's a problem with the networking on the ssc on one or both nodes:

=20

=20

Can each of them ping the other?

=20

(I find it very interesting that one node translates an errno of 9 to
"Network is down" while the other is "Host is down", both of which are
wrong.)

=20

ChrisV

________________________________

From: John Keiffer=20
Sent: Friday, March 14, 2008 1:27 PM
To: Chris Vandever
Cc: dl-QA
Subject: Bobcat cluster help?

=20

Hello Chris,

=20

My bobcat cluster is really having problems today. This morning when I
came in (Pleasanton office), both filer had crashed and were down to the
point I had to have them power cycled.

=20

I have since upgraded both Bobcats with the latest build from Larry
(I'll call it Sub12.v2). Currently, neither filer can do a vsvr show...

=20

G5r3 is up (though once the FC was in prom_init)

           =20

13:18:15 g5r3 diag> cluster show cluster

Cluster Name: g1r5       Cluster State:   On

NAS Gateways        IP              State   PCC

------------------------------------------------------

g1r5                10.2.1.21       UP      NO

g5r3                10.2.1.18       UP      YES

=20

G1r5 is up, but it only starts about 11 or 12 onstor processes.

=20

03/14/08 13:17:51 g1r5 diag> cluster show cluster

Cluster Name:        Cluster State:   Off

NAS Gateways        IP              State   PCC

------------------------------------------------------

g1r5                10.2.1.21       N/A     N/A

g5r3                10.2.1.18       N/A     N/A

=20

# onstor

      12

root     12246  0.0  0.2  2020   476 ??  Ss     1:14PM    0:00.06
/onstor/bin/sshd

root     16200  0.0  0.3   500   776 ??  Ss     1:15PM    0:00.40
/onstor/bin/pm

root     24192  0.0  0.6   436  1416 ??  S      1:15PM    0:01.18
/onstor/bin/elog

root      2737  0.0  0.6   796  1588 ??  S      1:15PM    0:00.88
/onstor/bin/ncmd

root     25055  0.0  0.6   496  1564 ??  S      1:15PM    0:00.56
/onstor/bin/eventd

root      7116  0.0  0.5   412  1308 ??  S      1:15PM    0:00.17
/onstor/bin/timekeeper

root      3309  0.0  0.2   156   468 ??  S      1:15PM    0:00.48
/onstor/bin/chassisd

root      3289  0.0  1.2  2888  2968 ??  S      1:15PM    0:03.17
/onstor/bin/sdm_cfgd

root     13035  0.0  0.9  1684  2196 ??  S      1:15PM    0:00.36
/onstor/bin/evm_cfgd

root     15469  0.0  0.4   516  1152 ??  S<     1:15PM    0:00.11
/onstor/bin/cluster_server

root     14965  0.1  0.8  1204  1936 ??  S<     1:15PM    0:00.45
/onstor/bin/cluster_contrl

root      8075  0.0  0.6  1520  1496 ??  S<     1:15PM    0:00.26
/onstor/bin/cluster_contrl

#

=20

Here are the repeating elogs from g1r5:

=20

Mar 14 13:23:58 g1r5 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1

Mar 14 13:23:58 g1r5 : 0:0:nfxsh:NOTICE: cmd[0]: -> EMRS: tried (5
times) without success to get the EMRS config from nfxsh : status[2]

Mar 14 13:23:58 g1r5 : 0:0:cluster2:INFO: Cluster_SendMsgSock: sendto
failed, code 9 (Network is down)

Mar 14 13:23:58 g1r5 : 0:0:cluster2:ERROR: ClusterContrl_GetRecordWhole:
ubik_call failed, recType 4(elogCfg), code -1, rc 30

Mar 14 13:23:58 g1r5 : 0:0:cluster2:INFO: cluster_lookup_sess: fail to
open sess for app cluster2, rc -20

=20

Here are the repeating elogs from g5r3:

=20

Mar 14 13:24:12 g5r3 : 0:0:cluster2:INFO: cluster_clientSendRmcRpc:
Error sending rpc to cluster2, flags 820a, name tape-driver, rc -20,
retrying...

Mar 14 13:24:12 g5r3 : 0:0:cluster2:INFO: Cluster_SendMsgSock: sendto
failed, code 9 (Host is down)

Mar 14 13:24:12 g5r3 : 0:0:cluster2:INFO: cluster_clientSendRmcRpc:
Error sending rpc to cluster2, flags 820a, name vsd, rc -20, retrying...

Mar 14 13:24:13 g5r3 : 0:0:cluster2:INFO: cluster_clientSendRmcRpc:
Error sending rpc to cluster2, flags 820a, name vtm, rc -20, retrying...

Mar 14 13:24:13 g5r3 : 0:0:cluster2:INFO: Cluster_SendMsgSock: sendto
failed, code 9 (Host is down)

=20

Thank you,

John Keiffer

=20

=20


------_=_NextPart_001_01C88615.8FB74246
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
 namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" =
name=3D"PlaceType"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"PlaceName"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"City"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"place"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:Wingdings;
	panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal;
	font-family:Arial;
	color:windowtext;}
span.EmailStyle18
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle19
	{mso-style-type:personal-reply;
	font-family:Arial;
	color:navy;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>John K. got the following on 2 =
different
bobcat linux nodes supposedly running the same =
release:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
color=3Dnavy face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial;color:navy'>Mar 14 13:23:58 =
g1r5 :
0:0:cluster2:INFO: Cluster_SendMsgSock: sendto failed, code 9 (Network =
is down)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
color=3Dnavy face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial;color:navy'>Mar 14 13:24:12 =
g5r3 : 0:0:cluster2:INFO:
Cluster_SendMsgSock: sendto failed, code 9 (Host is =
down)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>The code that generates this =
is:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;
errno =3D 0;<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;
stat =3D sendto(sock, msg, msgLen, 0, (struct sockaddr*)&amp;to, =
sizeof(to));<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;
if (stat !=3D msgLen) {<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
/* The send failed, so retry it.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
*/<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
if (stat &lt; 0) {<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
CLUSTER_INFO((&quot;%s: sendto failed, code %d =
(%s)\n&quot;,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
__FUNCTION__,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
errno,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
strerror(errno)));<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>So, (1) Why do the two nodes =
translate an
errno of 9 differently? &nbsp;(2) Why do they translate 9 to ENETDOWN or
EHOSTDOWN when 9 is EBADF?&nbsp; I can believe that perhaps I need to =
stash a
copy of errno into a local variable in case somewhere in the bowels of
CLUSTER_INFO() (which maps to an elog) we inadvertently step on errno, =
but I&#8217;d
kind of expect the printed int and string to at least be =
consistent.&nbsp; Sometimes
I can be pretty dense, however&#8230;&nbsp; </span></font><font size=3D2
color=3Dnavy face=3DWingdings><span =
style=3D'font-size:10.0pt;font-family:Wingdings;
color:navy'>L</span></font><font size=3D2 color=3Dnavy =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial;color:navy'><o:p></o:p></span=
></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>ChrisV<o:p></o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> Chris =
Vandever <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Friday, March 14, =
2008 1:40
PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">John
 Keiffer</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> dl-QA<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: Bobcat =
cluster help?</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>There&#8217;s a problem with the
networking on the ssc on one or both nodes:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Can each of them ping the =
other?<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>(I find it very interesting that =
one node
translates an errno of 9 to &#8220;Network is down&#8221; while the =
other is
&#8220;Host is down&#8221;, both of which are =
wrong.)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>ChrisV<o:p></o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">John Keiffer</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Friday, March 14, =
2008 1:27
PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Chris Vandever<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> dl-QA<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> Bobcat cluster =
help?</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Hello Chris,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>My bobcat cluster is really having problems today. =
This
morning when I came in (<st1:place w:st=3D"on"><st1:City =
w:st=3D"on">Pleasanton</st1:City></st1:place>
office), both filer had crashed and were down to the point I had to have =
them
power cycled.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>I have since upgraded both Bobcats with the latest =
build
from Larry (I&#8217;ll call it Sub12.v2). Currently, neither filer can =
do a
vsvr show&#8230;<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>G5r3 is up (though once the FC was in =
prom_init)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;
<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>13:18:15 g5r3 diag&gt; =
cluster show
cluster<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>Cluster Name:
g1r5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <st1:place =
w:st=3D"on"><st1:PlaceName
 w:st=3D"on">Cluster</st1:PlaceName> <st1:PlaceType =
w:st=3D"on">State</st1:PlaceType></st1:place>:&nbsp;&nbsp;
On<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>NAS
Gateways&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
IP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;
State&nbsp;&nbsp; PCC<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>----------------------------=
--------------------------<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>g1r5&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
10.2.1.21&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
UP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
NO<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>g5r3&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
10.2.1.18&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
UP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
YES<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>G1r5 is up, but it only starts about 11 or 12 onstor
processes.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>03/14/08 13:17:51 g1r5 =
diag&gt;
cluster show cluster<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>Cluster
Name:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <st1:place =
w:st=3D"on"><st1:PlaceName
 w:st=3D"on">Cluster</st1:PlaceName> <st1:PlaceType =
w:st=3D"on">State</st1:PlaceType></st1:place>:&nbsp;&nbsp;
Off<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>NAS
Gateways&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
IP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;
State&nbsp;&nbsp; PCC<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>----------------------------=
--------------------------<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>g1r5&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
10.2.1.21&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
N/A&nbsp;&nbsp;&nbsp;&nbsp; N/A<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>g5r3&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;10.2.1.18&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;
N/A&nbsp;&nbsp;&nbsp;&nbsp; N/A<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'><o:p>&nbsp;</o:p></span></fo=
nt></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'># =
onstor<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p; 12<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>root&nbsp;&nbsp;&nbsp;&nbsp;=

12246&nbsp; 0.0&nbsp; 0.2&nbsp; 2020&nbsp;&nbsp; 476 ??&nbsp;
Ss&nbsp;&nbsp;&nbsp;&nbsp; 1:14PM&nbsp;&nbsp;&nbsp; 0:00.06 =
/onstor/bin/sshd<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>root&nbsp;&nbsp;&nbsp;&nbsp;=

16200&nbsp; 0.0&nbsp; 0.3&nbsp;&nbsp; 500&nbsp;&nbsp; 776 ??&nbsp;
Ss&nbsp;&nbsp;&nbsp;&nbsp; 1:15PM&nbsp;&nbsp;&nbsp; 0:00.40 =
/onstor/bin/pm<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>root&nbsp;&nbsp;&nbsp;&nbsp;=

24192&nbsp; 0.0&nbsp; 0.6&nbsp;&nbsp; 436&nbsp; 1416 ??&nbsp;
S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1:15PM&nbsp;&nbsp;&nbsp; 0:01.18
/onstor/bin/elog<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>root&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;
2737&nbsp; 0.0&nbsp; 0.6&nbsp;&nbsp; 796&nbsp; 1588 ??&nbsp;
S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1:15PM&nbsp;&nbsp;&nbsp; 0:00.88
/onstor/bin/ncmd<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>root&nbsp;&nbsp;&nbsp;&nbsp;=
 25055&nbsp;
0.0&nbsp; 0.6&nbsp;&nbsp; 496&nbsp; 1564 ??&nbsp;
S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1:15PM&nbsp;&nbsp;&nbsp; 0:00.56
/onstor/bin/eventd<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>root&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;
7116&nbsp; 0.0&nbsp; 0.5&nbsp;&nbsp; 412&nbsp; 1308 ??&nbsp;
S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1:15PM&nbsp;&nbsp;&nbsp; 0:00.17
/onstor/bin/timekeeper<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>root&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;
3309&nbsp; 0.0&nbsp; 0.2&nbsp;&nbsp; 156&nbsp;&nbsp; 468 ??&nbsp;
S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1:15PM&nbsp;&nbsp;&nbsp; 0:00.48
/onstor/bin/chassisd<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>root&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;
3289&nbsp; 0.0&nbsp; 1.2&nbsp; 2888&nbsp; 2968 ??&nbsp;
S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1:15PM&nbsp;&nbsp;&nbsp; 0:03.17
/onstor/bin/sdm_cfgd<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>root&nbsp;&nbsp;&nbsp;&nbsp;=

13035&nbsp; 0.0&nbsp; 0.9&nbsp; 1684&nbsp; 2196 ??&nbsp;
S&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1:15PM&nbsp;&nbsp;&nbsp; 0:00.36
/onstor/bin/evm_cfgd<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>root&nbsp;&nbsp;&nbsp;&nbsp;=

15469&nbsp; 0.0&nbsp; 0.4&nbsp;&nbsp; 516&nbsp; 1152 ??&nbsp;
S&lt;&nbsp;&nbsp;&nbsp;&nbsp; 1:15PM&nbsp;&nbsp;&nbsp; 0:00.11
/onstor/bin/cluster_server<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>root&nbsp;&nbsp;&nbsp;&nbsp;=

14965&nbsp; 0.1&nbsp; 0.8&nbsp; 1204&nbsp; 1936 ??&nbsp;
S&lt;&nbsp;&nbsp;&nbsp;&nbsp; 1:15PM&nbsp;&nbsp;&nbsp; 0:00.45
/onstor/bin/cluster_contrl<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>root&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;
8075&nbsp; 0.0&nbsp; 0.6&nbsp; 1520&nbsp; 1496 ??&nbsp;
S&lt;&nbsp;&nbsp;&nbsp;&nbsp; 1:15PM&nbsp;&nbsp;&nbsp; 0:00.26
/onstor/bin/cluster_contrl<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>#<o:p></o:p></span></font></=
p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Here are the repeating elogs from =
g1r5:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>Mar 14 13:23:58 g1r5 : =
0:0:cluster2:ERROR:
cluster_getRecordIdByKey: no reply bck -1<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>Mar 14 13:23:58 g1r5 :
0:0:nfxsh:NOTICE: cmd[0]: -&gt; EMRS: tried (5 times) without success to =
get
the EMRS config from nfxsh : status[2]<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>Mar 14 13:23:58 g1r5 :
0:0:cluster2:INFO: Cluster_SendMsgSock: sendto failed, code 9 (Network =
is down)<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>Mar 14 13:23:58 g1r5 :
0:0:cluster2:ERROR: ClusterContrl_GetRecordWhole: ubik_call failed, =
recType
4(elogCfg), code -1, rc 30<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>Mar 14 13:23:58 g1r5 :
0:0:cluster2:INFO: cluster_lookup_sess: fail to open sess for app =
cluster2, rc
-20<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Here are the repeating elogs from =
g5r3:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>Mar 14 13:24:12 g5r3 :
0:0:cluster2:INFO: cluster_clientSendRmcRpc: Error sending rpc to =
cluster2,
flags 820a, name tape-driver, rc -20, =
retrying...<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>Mar 14 13:24:12 g5r3 :
0:0:cluster2:INFO: Cluster_SendMsgSock: sendto failed, code 9 (Host is =
down)<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>Mar 14 13:24:12 g5r3 :
0:0:cluster2:INFO: cluster_clientSendRmcRpc: Error sending rpc to =
cluster2,
flags 820a, name vsd, rc -20, retrying...<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>Mar 14 13:24:13 g5r3 :
0:0:cluster2:INFO: cluster_clientSendRmcRpc: Error sending rpc to =
cluster2,
flags 820a, name vtm, rc -20, retrying...<o:p></o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-left:.5in'><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>Mar 14 13:24:13 g5r3 :
0:0:cluster2:INFO: Cluster_SendMsgSock: sendto failed, code 9 (Host is =
down)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Thank you,</span></font><o:p></o:p></p>

<p class=3DMsoNormal><st1:PersonName w:st=3D"on"><font size=3D2 =
face=3DArial><span
 style=3D'font-size:10.0pt;font-family:Arial'>John =
Keiffer</span></font></st1:PersonName><o:p></o:p></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

</div>

</body>

</html>

------_=_NextPart_001_01C88615.8FB74246--
