X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C84E8C.BAB3DCD1@onstor-exch02.onstor.net>; Thu, 3 Jan 2008 21:46:11 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C84E8C.BAB3DCD1"
Content-class: urn:content-classes:message
Subject: RE: Sub 3 candidate not doing so well.
Date: Thu, 3 Jan 2008 21:46:11 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E056C927D@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E056C927A@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Sub 3 candidate not doing so well.
Thread-Index: AchOhbpmrDlJjATcRsm7DTVABafJcQAA/twQ
From: "Larry Scheer" <larry.scheer@onstor.com>
To: "Larry Scheer" <larry.scheer@onstor.com>,
	"dl-Cougar" <dl-Cougar@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C84E8C.BAB3DCD1
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Here are the stack traces:

Eventd:

#0  0x2b5fdb04 in kill () from /lib/libc.so.6
#1  0x2b5ff200 in abort () from /lib/libc.so.6
#2  0x2af043b8 in rmc_printf (ecode=3D3, format=3D0x2af05c98
"rmc_spin_unlock(): lock (%p) val(%d)") at rmc_util.c:200
#3  0x2aefb8a4 in rmc_spin_unlock (lck=3D0x4a92d8) at rmc.c:3771
#4  0x2aefddd0 in rmc_close (sess=3D0x4a92d8) at rmc_api.c:1092
#5  0x00402aa4 in handleMessageFailure (msg=3D0x4ad258) at event.c:370

Elog around time of eventd crash:
Jan  3 19:44:50 g11r10 : 0:0:evm:NOTICE: evm_removeVolEntry: deleted
volume[g12r10-raj-vs1-mirrorvol8].
Jan  3 19:45:31 g11r10 : 0:0:evm:ERROR: evm_updateVolState : failed to
get response from FP for volume[g12r10-raj-vs1-mirrorvol8]
Jan  3 19:45:31 g11r10 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
'g12r10-raj-vs1-mirrorvol8', Id 0x0000073f000000a0, Event 'Create',
State 'Down'
Jan  3 19:45:32 g11r10 : 0:0:ssh:NOTICE: 'root' logged in through remote
host(larrys.onstor.net)
Jan  3 19:46:04 g11r10 : 0:0:eventd:NOTICE: handleMessageFailure: Msg
failure (-19) for session 0x4a92d8 app  unused 0. Closing connection.
Jan  3 19:46:04 g11r10 : 0:0:eventd:INFO: close_session: Lost
connection. Close rmc session 0x4a92d8 for event session (nil) app
unused 0.
Jan  3 19:46:05 g11r10 : 0:0:pm:ERROR: pm_sig_handler:
/onstor/bin/eventd (pid 856) terminated with signal Aborted
Jan  3 19:46:06 g11r10 : 0:0:pm:INFO: /onstor/bin/eventd: finished
initialization.

Evm_cfgd:

#0  0x0042f9f8 in evm_delLunFromDiscList (lunEntry=3D0x4fd750) at
evm-db.c:1480
#1  0x00417254 in evm_verifyLunReqProc (evmTxn=3D0x2bb89188) at
evm-cfgd-rx.c:2748
#2  0x00446210 in evm_processTxnRunQueue (maxProcess=3D0) at
evm-util.c:1625
#3  0x00406c18 in main (argc=3D1, argv=3D0x7fdf1e34, envp=3D0x7fdf1e3c) =
at
evm-cfgd.c:945
(gdb) list=20
1475
1476    --*/
1477    void
1478    evm_delLunFromDiscList(evm_lunAttr_t *lunEntry)
1479    {
1480        TAILQ_REMOVE(&evm_lunDiscHead, lunEntry, dNext);
1481        return;
1482    }
1483
1484

print *lunEntry
$1 =3D {sid =3D 0, openHandle =3D 0, devType =3D 0, lunState =3D =
EVM_LUN_UNKNOWN,
devName =3D '\0' <repeats 127 times>, blkSize =3D 0,=20
  evmBlkCount =3D 0, fsBlkCount =3D 0, vendor =3D
"\000\000\000\000\000\000\000\000", model =3D '\0' <repeats 16 times>,
rLevel =3D 0,=20
  aIndex =3D 0, nAlloc =3D 0, allocUnit =3D {{volId =3D {whole_thing =3D =
0,
details =3D {u =3D {id =3D 0, parts =3D {clusterUnique =3D 0,=20
              reserved =3D 0}}, worldUnique =3D 0}}, startBlk =3D 0, =
numBlks =3D
0}}, volEntry =3D 0x0, next =3D {tqe_next =3D 0x0,=20
    tqe_prev =3D 0x0}, vNext =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, =
dNext =3D
{tqe_next =3D 0x0, tqe_prev =3D 0x0}, odCfg =3D 0x0}


The elog is full of stuff like:

		Jan  3 20:19:41 g11r10 : 0:0:evm:ERROR:
evm_getLunReqProc: failed to get label for LUN[3PARdata_0565_0016C0565]
		Jan  3 20:19:49 g11r10 : 0:0:spm:INFO: spm_sendNcmMsg:
attempt send before open completed
		Jan  3 20:19:49 g11r10 : 0:0:spm:INFO: spm_sendNcmMsg:
attempt send before open completed
		Jan  3 20:20:51 g11r10 : 0:0:evm:ERROR:
evm_getLunReqProc: failed to get label for LUN[3PARdata_0565_001C50565]
		Jan  3 20:20:51 g11r10 : 0:0:evm:ERROR:
evm_getLunReqProc: failed to get label for LUN[3PARdata_0565_000EE0565]


Here is the elog at the time the evm_cfg=20

Jan  3 19:48:18 g11r10 : 0:0:evm:INFO: evm_rcvRmcMsg : SDM session open
success
		Jan  3 19:48:10 g11r10 : 0:0:evm:ERROR:
evm_verifyLunReqProc: Cannot read label for LUN[3PARdata_0565_0015B0565]
		Jan  3 19:48:10 g11r10 : 0:0:evm:ERROR:
evm_verifyLunReqProc: Missing label on LUN[3PARdata_0565_0015B0565]!
		Jan  3 19:48:10 g11r10 : 0:0:evm:ERROR:
evm_verifyLunReqProc: removing vol[] LUN[3PARdata_0565_0015B0565] from
cluster[g11r10]
		Jan  3 19:48:10 g11r10 : 0:0:evm:ERROR:
evm_verifyLunReqProc: Cannot read label for LUN[3PARdata_0565_001120565]
		Jan  3 19:48:17 g11r10 : 0:0:evm:ERROR:
evm_verifyLunReqProc: removing vol[] LUN[3PARdata_0565_001030565] from
cluster[g11r10]
		Jan  3 19:48:17 g11r10 : 0:0:eventd:NOTICE:
Process-EVENT Volume: name 'g12r10-raj-vs1-sourcevol16', Id
0x0000073f00000098, Event 'Broken', State 'Down'
		Jan  3 19:48:17 g11r10 : 0:0:pm:ERROR: pm_sig_handler:
/onstor/bin/evm_cfgd (pid 987) terminated with signal Segmentation fault

After evm_cfgd restarts more of:
		Jan  3 19:48:58 g11r10 : 0:0:spm:INFO: spm_sendNcmMsg:
attempt send before open completed
		Jan  3 19:48:58 g11r10 : 0:0:spm:INFO: spm_sendNcmMsg:
attempt send before open completed
		Jan  3 19:50:06 g11r10 : 0:0:evm:ERROR:
evm_updateVolState : failed to get response from FP for
volume[g12r10-raj-vs1-sourcevol4]
		Jan  3 19:50:06 g11r10 : 0:0:evm:ERROR:
evm_updateVolState : failed to get response from FP for
volume[g12r10-raj-vs1-sourcevol14]
		Jan  3 19:50:06 g11r10 : 0:0:evm:ERROR:
evm_updateVolState : failed to get response from FP for
volume[g12r10-raj-vs1-sourcevol9]

James thinks spm didn't give LUN info to evm which triggered the evm
core.
_____________________________________________
From: Larry Scheer=20
Sent: Thursday, January 03, 2008 7:56 PM
To: dl-Cougar
Subject: Sub 3 candidate not doing so well.

After a number of fits and starts I managed to get today's build (from
~11:30 AM) onto g11r10 and g12r10.

I configured a new VS, and after some issues, created a new volume. When
I started some basic traffic tests I see an eventd and evm_cfgd core.

Just a quick note to say I don't think this build can be used as
submittal  3.

------_=_NextPart_001_01C84E8C.BAB3DCD1
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7653.38">
<TITLE>RE: Sub 3 candidate not doing so well.</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Here are the stack traces:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Eventd:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">#0&nbsp; 0x2b5fdb04 in kill () from =
/lib/libc.so.6</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">#1&nbsp; 0x2b5ff200 in abort () from =
/lib/libc.so.6</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">#2&nbsp; 0x2af043b8 in rmc_printf (ecode=3D3, =
format=3D0x2af05c98 &quot;rmc_spin_unlock(): lock (%p) val(%d)&quot;) at =
rmc_util.c:200</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">#3&nbsp; 0x2aefb8a4 in rmc_spin_unlock (lck=3D0x4a92d8) =
at rmc.c:3771</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">#4&nbsp; 0x2aefddd0 in rmc_close (sess=3D0x4a92d8) at =
rmc_api.c:1092</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">#5&nbsp; 0x00402aa4 in handleMessageFailure =
(msg=3D0x4ad258) at event.c:370</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Elog around time of eventd crash:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jan&nbsp; 3 19:44:50 g11r10 : 0:0:evm:NOTICE: =
evm_removeVolEntry: deleted =
volume[g12r10-raj-vs1-mirrorvol8].</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jan&nbsp; 3 19:45:31 g11r10 : 0:0:evm:ERROR: =
evm_updateVolState : failed to get response from FP for =
volume[g12r10-raj-vs1-mirrorvol8]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jan&nbsp; 3 19:45:31 g11r10 : 0:0:eventd:NOTICE: =
Process-EVENT Volume: name 'g12r10-raj-vs1-mirrorvol8', Id =
0x0000073f000000a0, Event 'Create', State 'Down'</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jan&nbsp; 3 19:45:32 g11r10 : 0:0:ssh:NOTICE: 'root' =
logged in through remote host(larrys.onstor.net)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jan&nbsp; 3 19:46:04 g11r10 : 0:0:eventd:NOTICE: =
handleMessageFailure: Msg failure (-19) for session 0x4a92d8 app&nbsp; =
unused 0. Closing connection.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jan&nbsp; 3 19:46:04 g11r10 : 0:0:eventd:INFO: =
close_session: Lost connection. Close rmc session 0x4a92d8 for event =
session (nil) app unused 0.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jan&nbsp; 3 19:46:05 g11r10 : 0:0:pm:ERROR: =
pm_sig_handler: /onstor/bin/eventd (pid 856) terminated with signal =
Aborted</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jan&nbsp; 3 19:46:06 g11r10 : 0:0:pm:INFO: =
/onstor/bin/eventd: finished initialization.</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Evm_cfgd:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">#0&nbsp; 0x0042f9f8 in evm_delLunFromDiscList =
(lunEntry=3D0x4fd750) at evm-db.c:1480</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">#1&nbsp; 0x00417254 in evm_verifyLunReqProc =
(evmTxn=3D0x2bb89188) at evm-cfgd-rx.c:2748</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">#2&nbsp; 0x00446210 in evm_processTxnRunQueue =
(maxProcess=3D0) at evm-util.c:1625</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">#3&nbsp; 0x00406c18 in main (argc=3D1, argv=3D0x7fdf1e34, =
envp=3D0x7fdf1e3c) at evm-cfgd.c:945</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">(gdb) list </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">1475</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">1476&nbsp;&nbsp;&nbsp; --*/</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">1477&nbsp;&nbsp;&nbsp; void</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">1478&nbsp;&nbsp;&nbsp; =
evm_delLunFromDiscList(evm_lunAttr_t *lunEntry)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">1479&nbsp;&nbsp;&nbsp; {</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">1480&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
TAILQ_REMOVE(&amp;evm_lunDiscHead, lunEntry, dNext);</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">1481&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
return;</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">1482&nbsp;&nbsp;&nbsp; }</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">1483</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">1484</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">print *lunEntry</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">$1 =3D {sid =3D 0, openHandle =3D 0, devType =3D 0, =
lunState =3D EVM_LUN_UNKNOWN, devName =3D '\0' &lt;repeats 127 =
times&gt;, blkSize =3D 0, </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp; evmBlkCount =3D 0, fsBlkCount =3D 0, vendor =3D =
&quot;\000\000\000\000\000\000\000\000&quot;, model =3D '\0' &lt;repeats =
16 times&gt;, rLevel =3D 0, </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp; aIndex =3D 0, nAlloc =3D 0, allocUnit =3D {{volId =
=3D {whole_thing =3D 0, details =3D {u =3D {id =3D 0, parts =3D =
{clusterUnique =3D 0, </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp; reserved =3D 0}}, worldUnique =3D 0}}, startBlk =3D =
0, numBlks =3D 0}}, volEntry =3D 0x0, next =3D {tqe_next =3D 0x0, =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp; tqe_prev =3D 0x0}, vNext =3D {tqe_next =
=3D 0x0, tqe_prev =3D 0x0}, dNext =3D {tqe_next =3D 0x0, tqe_prev =3D =
0x0}, odCfg =3D 0x0}</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">The elog is full of stuff like:</FONT></SPAN></P>
<UL><UL>
<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 20:19:41 g11r10 : 0:0:evm:ERROR: =
evm_getLunReqProc: failed to get label for =
LUN[3PARdata_0565_0016C0565]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 20:19:49 g11r10 : 0:0:spm:INFO: =
spm_sendNcmMsg: attempt send before open completed</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 20:19:49 g11r10 : 0:0:spm:INFO: =
spm_sendNcmMsg: attempt send before open completed</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 20:20:51 g11r10 : 0:0:evm:ERROR: =
evm_getLunReqProc: failed to get label for =
LUN[3PARdata_0565_001C50565]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 20:20:51 g11r10 : 0:0:evm:ERROR: =
evm_getLunReqProc: failed to get label for =
LUN[3PARdata_0565_000EE0565]</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>
<BR>
</UL></UL>
<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">H</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">ere =
is</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">the elog at the time the =
evm_cfg</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial"></FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> </SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jan&nbsp; 3 19:48:18 g11r10 : 0:0:evm:INFO: evm_rcvRmcMsg =
: SDM session open success</FONT></SPAN></P>
<UL><UL>
<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 19:48:10 g11r10 : 0:0:evm:ERROR: =
evm_verifyLunReqProc: Cannot read label for =
LUN[3PARdata_0565_0015B0565]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 19:48:10 g11r10 : 0:0:evm:ERROR: =
evm_verifyLunReqProc: Missing label on =
LUN[3PARdata_0565_0015B0565]!</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 19:48:10 g11r10 : 0:0:evm:ERROR: =
evm_verifyLunReqProc: removing vol[] LUN[3PARdata_0565_0015B0565] from =
cluster[g11r10]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 19:48:10 g11r10 : 0:0:evm:ERROR: =
evm_verifyLunReqProc: Cannot read label for =
LUN[3PARdata_0565_001120565]</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 19:48:17 g11r10 : 0:0:evm:ERROR: =
evm_verifyLunReqProc: removing vol[] LUN[3PARdata_0565_001030565] from =
cluster[g11r10]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 19:48:17 g11r10 : 0:0:eventd:NOTICE: =
Process-EVENT Volume: name 'g12r10-raj-vs1-sourcevol16', Id =
0x0000073f00000098, Event 'Broken', State 'Down'</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 19:48:17 g11r10 : 0:0:pm:ERROR: =
pm_sig_handler: /onstor/bin/evm_cfgd (pid 987) terminated with signal =
Segmentation fault</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>
</UL></UL>
<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">After evm_cfgd restarts more of:</FONT></SPAN></P>
<UL><UL>
<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 19:48:58 g11r10 : 0:0:spm:INFO: =
spm_sendNcmMsg: attempt send before open completed</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 19:48:58 g11r10 : 0:0:spm:INFO: =
spm_sendNcmMsg: attempt send before open completed</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 19:50:06 g11r10 : 0:0:evm:ERROR: =
evm_updateVolState : failed to get response from FP for =
volume[g12r10-raj-vs1-sourcevol4]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 19:50:06 g11r10 : 0:0:evm:ERROR: =
evm_updateVolState : failed to get response from FP for =
volume[g12r10-raj-vs1-sourcevol14]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Courier New">Jan&nbsp; 3 19:50:06 g11r10 : 0:0:evm:ERROR: =
evm_updateVolState : failed to get response from FP for =
volume[g12r10-raj-vs1-sourcevol9]</FONT></SPAN></P>
</UL></UL>
<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">James thinks spm didn</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">&#8217;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">t give LUN info to evm which triggered the evm =
core.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Larry Scheer<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, January 03, =
2008 7:56 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Cougar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Sub 3 candidate not doing so well.</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">After =
a number of fits and starts I managed to get today&#8217;s build (from =
~11:30 AM) onto g11r10 and g12r10.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I =
configured a new VS, and after some issues, created a new volume. When I =
started some basic traffic tests I see an eventd and evm_cfgd =
core.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Just =
a quick note to say I don&#8217;t think this build can be used as =
submittal&nbsp; 3.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

</BODY>
</HTML>
------_=_NextPart_001_01C84E8C.BAB3DCD1--
