X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C71FAC.5C25F479@onstor-exch02.onstor.net>; Thu, 14 Dec 2006 10:19:14 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C71FAC.5C25F479"
Content-class: urn:content-classes:message
Subject: BC Soak needs attention
Date: Thu, 14 Dec 2006 10:19:14 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E01C0AC15@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: BC Soak needs attention
thread-index: AccfrFv7cgaIoR0bSfuYf9StHJ2FPQ==
From: "Raj Kumar" <raj.kumar@onstor.com>
To: "dl-Clio" <dl-Clio@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C71FAC.5C25F479
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

Looks like we have issues with the BC soak (eng51 - 10.2.1.9)

1. spm is unresponsive. It's in the same state now.

eng51 diag> lun show disk -t free
Timed out waiting for response.
Failed to get response from SPM
% Command failure.

2. There seem to be problems with ncm. Ncm was restarted around 1:30 AM
and there is a core.

Dec 14 10:15:26 eng51 : 0:0:ncm:WARNING: ncmd : ncm_reopen_filer:
ncm_open_filer(eng49) succeeded=20
Dec 14 10:15:29 eng51 : 0:0:ncm:WARNING: ncmd : ncm_reopen_filer:
ncm_open_filer(eng49) succeeded=20
Dec 14 10:15:33 eng51 : 0:0:ncm:WARNING: ncmd : ncm_reopen_filer:
retries to filer(eng49) failed - aborting attemtpts
Dec 14 10:15:33 eng51 : 0:0:ncm:WARNING: ncmd : ncm_handle_net_msg_done:
ncm_repoen filer failed - -1
Dec 14 10:15:42 eng51 : 0:0:cluster2:ERROR: cluster_iClusterInfo: fail
to send ncm msg: -20=20
Dec 14 10:15:42 eng51 : 0:0:spm:ERROR: failed to get filer list from
cluster daemon

3. Auth agent crashed and there were some CIFS authentication errors due
to that. This was one time issue, not critical I believe.

Wed Dec 13 17:18:58 2006 (504911 usec) 35: Auth rpc (txrx->ssc) failed
due to some internal error (rc =3D -1); (req=3D0x100395c2c0, stack =
fr=3D1,
stack=3DAUTH_SSC_RPC_WAIT_STATE:CIFS_SESS_SETUP_WAIT_SSC_REPLY:CIFS_SESS_=
S
ETUP_WAIT_AUTH,
callers=3D0xffffffff831b4f84,0xffffffff831b587c,0xffffffff831b5694,0xffff=
f
fff831b5a60,0xffffffff831b51fc)
Wed Dec 13 17:18:58 2006 (559857 usec) 36: auth_rpc_open: 423: Auth-rpc
RMC session opened again for session :0x10036ef800, prev_session:
0x10036ef800

Can somebody have a look?

Thanks.

------_=_NextPart_001_01C71FAC.5C25F479
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7650.28">
<TITLE>BC Soak needs attention</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Hi,</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Looks =
like we have issues with the BC soak</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial"> (eng51</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT SIZE=3D2 FACE=3D"Arial">&#8211;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial"> 10.2.1.9)</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">1. =
spm is unresponsive.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial"></FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 =
FACE=3D"Arial">It&#8217;s</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial"> in the same state =
now.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">eng51 =
diag&gt; lun show disk -t free</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Timed =
out waiting for response.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Failed to get response from SPM</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">% =
Command failure.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">2. =
There seem to be problems with ncm. Ncm</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 =
FACE=3D"Arial">was restarted around 1:30 AM and there is a =
core.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Dec =
14 10:15:26 eng51 : 0:0:ncm:WARNING: ncmd : ncm_reopen_filer: =
ncm_open_filer(eng49) succeeded </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Dec =
14 10:15:29 eng51 : 0:0:ncm:WARNING: ncmd : ncm_reopen_filer: =
ncm_open_filer(eng49) succeeded </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Dec =
14 10:15:33 eng51 : 0:0:ncm:WARNING: ncmd : ncm_reopen_filer: retries to =
filer(eng49) failed - aborting attemtpts</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Dec =
14 10:15:33 eng51 : 0:0:ncm:WARNING: ncmd : ncm_handle_net_msg_done: =
ncm_repoen filer failed - -1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Dec =
14 10:15:42 eng51 : 0:0:cluster2:ERROR: cluster_iClusterInfo: fail to =
send ncm msg: -20 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Dec =
14 10:15:42 eng51 : 0:0:spm:ERROR: failed to get filer list from cluster =
daemon</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">3. =
Auth agent crashed and there</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 =
FACE=3D"Arial">were</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial"> some CIFS authentication =
errors due to that. This was one time issue, not critical I =
believe.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Wed =
Dec 13 17:18:58 2006 (504911 usec) 35: Auth rpc (txrx-&gt;ssc) failed =
due to some internal error (rc =3D -1); (req=3D0x100395c2c0, stack =
fr=3D1, =
stack=3DAUTH_SSC_RPC_WAIT_STATE:CIFS_SESS_SETUP_WAIT_SSC_REPLY:CIFS_SESS_=
SETUP_WAIT_AUTH, =
callers=3D0xffffffff831b4f84,0xffffffff831b587c,0xffffffff831b5694,0xffff=
ffff831b5a60,0xffffffff831b51fc)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Wed =
Dec 13 17:18:58 2006 (559857 usec) 36: auth_rpc_open: 423: Auth-rpc RMC =
session opened again for session :0x10036ef800, prev_session: =
0x10036ef800</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Can =
somebody have a look?</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Thanks.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

</BODY>
</HTML>
------_=_NextPart_001_01C71FAC.5C25F479--
