X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C8646A.ADEDE249@onstor-exch02.onstor.net>; Thu, 31 Jan 2008 17:37:52 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C8646A.ADEDE249"
Content-class: urn:content-classes:message
Subject: Don't Sync If You Run Cheetahs...
Date: Thu, 31 Jan 2008 17:37:52 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E03E9A59F@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Don't Sync If You Run Cheetahs...
Thread-Index: Achkaq3V6vSZUZo7TwKpLBpnDv9J4A==
From: "Chris Vandever" <chris.vandever@onstor.com>
To: "dl-Software" <dl-software@onstor.com>
Cc: "dl-QA" <dl-qa@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C8646A.ADEDE249
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Cheetahs no longer boot properly for dev builds.  I was running sub 5
and that worked fine.  I sync'd to top of tree around noon (I got
#27550), did a clean build, nfs mounts, and ran into problems.  So, we
broke something between then and now.  I don't know the status for sub
6.  The apps think the FP is down, although "system show chassis"
reports "interesting" information:

eng20 diag> system show chassis

slot    module     cpu         state
----------------------------------------------
   0    SSC        SSC         UP
                   SSC_MGMT    UP
   1    SSC        SSC         UP
                   SSC_MGMT    UP
   2    FCNIM      FC0         UP
                   FC1         UP
                   FC2         UP
                   FC3         UP
----------------------------------------------

The elogs show not all of the CPUs report cpu up events, so the apps
don't think the card is up:

Jan 31 15:37:22 eng20 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 0,
CPU 0, State Up
Jan 31 15:37:22 eng20 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 0,
CPU 1, State Up
Jan 31 15:37:31 eng20 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
CPU 0, State Up
Jan 31 15:38:22 eng20 : 0:0:vtm:INFO: vtm_sendCardStateMsg: Sending card
DOWN to eng20
Jan 31 15:38:23 eng20 : 0:0:vtm:INFO: Vtm_ProcCardStateMsg: card state
DOWN message from eng20
Jan 31 15:38:59 eng20 : 0:0:evm:ERROR: evm_sendCreateLvReq : NFP card
not up. Create LV (eng28-vol2) failed

Similarly on another filer, I don't get all of the requisite CPU events:

Jan 31 15:42:10 eng28 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 0,
CPU 0, State Up
Jan 31 15:42:10 eng28 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 0,
CPU 1, State Up
Jan 31 15:42:10 eng28 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 2,
CPU 0, State Up
Jan 31 15:42:10 eng28 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 2,
CPU 1, State Up
Jan 31 15:42:10 eng28 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 2,
CPU 2, State Up

Rendell is investigating the problem.

ChrisV


------_=_NextPart_001_01C8646A.ADEDE249
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7653.38">
<TITLE>Don't Sync If You Run Cheetahs...</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Cheetahs no longer boot properly for dev =
builds.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us">&nbsp;<FONT SIZE=3D2 FACE=3D"Arial"> I was running sub 5 =
and that worked fine.&nbsp; I sync</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&#8217;</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">d to top of tree around =
noon (I got #27550), did a clean build, nfs mounts, =
and</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT =
SIZE=3D2 FACE=3D"Arial">ran into problems.&nbsp; So, we broke something =
between then and</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT SIZE=3D2 FACE=3D"Arial">now</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">&nbsp; I =
don</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 FACE=3D"Arial">&#8217;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">t know the status for sub 6.</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp; T</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">he apps think the FP is =
down</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 FACE=3D"Arial">, although</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 =
FACE=3D"Arial">&#8220;</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">system show =
chassis</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">&#8221;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial"></FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT SIZE=3D2 FACE=3D"Arial">reports</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 =
FACE=3D"Arial">&#8220;</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">interesting</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&#8221;</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial"> =
information:</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">eng2</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">0</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida Console"> diag&gt; system =
show chassis</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">slot&nbsp;&nbsp;&nbsp; module&nbsp;&nbsp;&nbsp;&nbsp; =
cpu&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
state</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">----------------------------------------------</FONT></SPAN></P>=


<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; =
SSC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
SSC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; UP</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SSC_MGMT&nbsp;&nbsp;&nbsp; =
UP</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">&nbsp;&nbsp; 1&nbsp;&nbsp;&nbsp; =
SSC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
SSC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; UP</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SSC_MGMT&nbsp;&nbsp;&nbsp; =
UP</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">&nbsp;&nbsp; 2&nbsp;&nbsp;&nbsp; =
FCNIM&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
FC0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; UP</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
FC1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; UP</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
FC2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; UP</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
FC3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; UP</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">----------------------------------------------</FONT></SPAN><SPA=
N LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">The =
elogs show not all of the CPUs report cpu up events, so the apps =
don</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 FACE=3D"Arial">&#8217;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">t think the card is up:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">Jan 31 15:37:22 eng20 : 0:0:eventd:NOTICE: Process-EVENT CPU: =
Slot 0, CPU 0, State Up</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">Jan 31 15:37:22 eng20 : 0:0:eventd:NOTICE: Process-EVENT CPU: =
Slot 0, CPU 1, State Up</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">Jan 31 15:37:31 eng20 : 0:0:eventd:NOTICE: Process-EVENT CPU: =
Slot 1, CPU 0, State Up</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">Jan 31 15:38:22 eng20 : 0:0:vtm:INFO: vtm_sendCardStateMsg: =
Sending card DOWN to eng20</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">Jan 31 15:38:23 eng20 : 0:0:vtm:INFO: Vtm_ProcCardStateMsg: =
card state DOWN message from eng20</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">Jan 31 15:38:59 eng20 : 0:0:evm:ERROR: evm_sendCreateLvReq : =
NFP card not up. Create LV (eng28-vol2) failed</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Similarly on another filer, I don</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&#8217;</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">t get all of the requisite =
CPU events:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">Jan 31 15:42:10 eng28 : 0:0:eventd:NOTICE: Process-EVENT CPU: =
Slot 0, CPU 0, State Up</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">Jan 31 15:42:10 eng28 : 0:0:eventd:NOTICE: Process-EVENT CPU: =
Slot 0, CPU 1, State Up</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">Jan 31 15:42:10 eng28 : 0:0:eventd:NOTICE: Process-EVENT CPU: =
Slot 2, CPU 0, State Up</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">Jan 31 15:42:10 eng28 : 0:0:eventd:NOTICE: Process-EVENT CPU: =
Slot 2, CPU 1, State Up</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Lucida =
Console">Jan 31 15:42:10 eng28 : 0:0:eventd:NOTICE: Process-EVENT CPU: =
Slot 2, CPU 2, State Up</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Rendell is investigating the problem.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">ChrisV</FONT></SPAN></P>

</BODY>
</HTML>
------_=_NextPart_001_01C8646A.ADEDE249--
