X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C854BB.C9B40725@onstor-exch02.onstor.net>; Fri, 11 Jan 2008 18:38:09 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C854BB.C9B40725"
Content-class: urn:content-classes:message
Subject: RE: Submittal 4 build candidate not passing acceptance tests
Date: Fri, 11 Jan 2008 18:38:09 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E056C92B0@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E030E3ADC@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Submittal 4 build candidate not passing acceptance tests
Thread-Index: AchUtw5pnOwDnaGBSAuOi47kTAI5EQAABHYAAAASn2AAALM+sAAAVIzg
From: "Larry Scheer" <larry.scheer@onstor.com>
To: "Mike Lee" <mike.lee@onstor.com>
Cc: "dl-Cougar" <dl-Cougar@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C854BB.C9B40725
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

FP crash on g4r5 which just had storage added to it is:

Jan 11 16:17:34 g4r5 kernel: fp3: Panic : ispfc_device.c:1130: ASSERTION
failuree
 "!ispfc_fabric_map_is_allocated_id(cb, ma

I botched the dgb rcon session so I am attempting to get a trace now.

_____________________________________________
From: Mike Lee=20
Sent: Friday, January 11, 2008 5:36 PM
To: Larry Scheer
Cc: dl-Cougar
Subject: RE: Submittal 4 build candidate not passing acceptance tests

g11r10 has an spm core only; it is crashing due to seg fault in
spm_pollAddDevice() on a call to memcpy() where the source pointer is
null.

on g12r10:
sdm_cfgd crash is a manifestation of the RMC problem documented in
Defect 20580
eventd crash is a manifestation of the RMC problem documented in Defect
20580
evm_cfgd crash is crashing due to seg fault in evm_delLunFromDiscList()
in evm-db.c; looks like we're accessing a freed lunEntry pointer.

I'll take a look at g4r5 once I find out its IP address.

	 -----Original Message-----
	From: 	Larry Scheer =20
	Sent:	Friday, January 11, 2008 5:12 PM
	To:	Mike Lee
	Subject:	RE: Submittal 4 build candidate not passing
acceptance tests

	One FP crash was definitely the spinlock crash, but the other is
on the stand-alone system g4r5 which just had storage added to it and I
have yet to create a volime.

	G11r10, g12r10 and g4r5 are all getting the eventd crash.

	G12r10 has the evm_cfgd and sdm_cfgd crashes as well.

	_____________________________________________
	From: Mike Lee=20
	Sent: Friday, January 11, 2008 5:06 PM
	To: Larry Scheer
	Subject: RE: Submittal 4 build candidate not passing acceptance
tests

	Larry,=20
	The FP crash may be related to the spin lock change that Max
backed out today.
	What machines are the eventd / sdm_cfgd / evm_cfgd crashes on?
	Thanks.
	-Mike

		 -----Original Message-----
		From: 	Larry Scheer =20
		Sent:	Friday, January 11, 2008 5:04 PM
		To:	Vikas Saini; Raj Kumar
		Cc:	dl-Cougar
		Subject:	Submittal 4 build candidate not passing
acceptance tests

		I loaded 3 cougar systems (1 2 node cluster and  1
standalone)  with //depot/dev/...@27286 and I am seeing crashes in
eventd, sdm_cfgd, evm_cfgd and FP.



------_=_NextPart_001_01C854BB.C9B40725
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7653.38">
<TITLE>RE: Submittal 4 build candidate not passing acceptance =
tests</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">FP crash on g4r5 which just had storage added to it =
is:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jan 11 16:17:34 g4r5 kernel: fp3: Panic : =
ispfc_device.c:1130: ASSERTION failuree</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&quot;!ispfc_fabric_map_is_allocated_id(cb, =
ma</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">I botched the dgb rcon session so I am attempting to get =
a trace now.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Mike Lee<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Friday, January 11, 2008 =
5:36 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Larry Scheer<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Cougar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: Submittal 4 build candidate not passing acceptance =
tests</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">g11r10 has an spm core only; it is crashing due to seg =
fault in spm_pollAddDevice() on a call to memcpy() where the source =
pointer is null.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">on g12r10:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">sdm_cfgd crash is a manifestation of the RMC problem =
documented in Defect 20580</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">eventd crash is a manifestation of the RMC problem =
documented in Defect 20580</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">evm_cfgd crash is crashing due to seg fault in =
evm_delLunFromDiscList() in evm-db.c; looks like we're accessing a freed =
lunEntry pointer.<BR>
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">I'll take a look at g4r5 once I find out its IP =
address.</FONT></SPAN></P>
<UL>
<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT =
FACE=3D"Arial"></FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us">&nbsp;<FONT SIZE=3D1 =
FACE=3D"Tahoma">-----Original Message-----</FONT></SPAN></P>

<P ALIGN=3DLEFT><B><SPAN LANG=3D"en-us"><FONT SIZE=3D1 =
FACE=3D"Tahoma">From: &nbsp;</FONT></SPAN></B><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> =
<FONT SIZE=3D1 FACE=3D"Tahoma">Larry Scheer&nbsp; </FONT></SPAN></P>

<P ALIGN=3DLEFT><B><SPAN LANG=3D"en-us"><FONT SIZE=3D1 =
FACE=3D"Tahoma">Sent:&nbsp;&nbsp;</FONT></SPAN></B><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> =
<FONT SIZE=3D1 FACE=3D"Tahoma">Friday, January 11, 2008 5:12 =
PM</FONT></SPAN></P>

<P ALIGN=3DLEFT><B><SPAN LANG=3D"en-us"><FONT SIZE=3D1 =
FACE=3D"Tahoma">To:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></SPAN></B><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> =
<FONT SIZE=3D1 FACE=3D"Tahoma">Mike Lee</FONT></SPAN></P>

<P ALIGN=3DLEFT><B><SPAN LANG=3D"en-us"><FONT SIZE=3D1 =
FACE=3D"Tahoma">Subject:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT>=
</SPAN></B><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT SIZE=3D1 FACE=3D"Tahoma">RE: Submittal 4 build =
candidate not passing acceptance tests</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">One FP crash was definitely the spinlock crash, but the =
other is on the stand-alone system g4r5 which just had storage added to =
it and I have yet to create a volime.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">G11r10, g12r10 and g4r5 are all getting the eventd =
crash.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">G12r10 has the evm_cfgd and sdm_cfgd crashes as =
well.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Mike Lee<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Friday, January 11, 2008 =
5:06 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Larry Scheer<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: Submittal 4 build candidate not passing acceptance =
tests</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">Larry, </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">The FP crash may be related to the spin lock change that =
Max backed out today.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">What machines are the eventd / sdm_cfgd / evm_cfgd =
crashes on?</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">Thanks.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">-Mike</FONT></SPAN></P>
<UL>
<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT =
FACE=3D"Arial"></FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us">&nbsp;<FONT SIZE=3D1 =
FACE=3D"Tahoma">-----Original Message-----</FONT></SPAN></P>

<P ALIGN=3DLEFT><B><SPAN LANG=3D"en-us"><FONT SIZE=3D1 =
FACE=3D"Tahoma">From: &nbsp;</FONT></SPAN></B><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> =
<FONT SIZE=3D1 FACE=3D"Tahoma">Larry Scheer&nbsp; </FONT></SPAN></P>

<P ALIGN=3DLEFT><B><SPAN LANG=3D"en-us"><FONT SIZE=3D1 =
FACE=3D"Tahoma">Sent:&nbsp;&nbsp;</FONT></SPAN></B><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> =
<FONT SIZE=3D1 FACE=3D"Tahoma">Friday, January 11, 2008 5:04 =
PM</FONT></SPAN></P>

<P ALIGN=3DLEFT><B><SPAN LANG=3D"en-us"><FONT SIZE=3D1 =
FACE=3D"Tahoma">To:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></SPAN></B><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> =
<FONT SIZE=3D1 FACE=3D"Tahoma">Vikas Saini; Raj Kumar</FONT></SPAN></P>

<P ALIGN=3DLEFT><B><SPAN LANG=3D"en-us"><FONT SIZE=3D1 =
FACE=3D"Tahoma">Cc:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></SPAN></B><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> =
<FONT SIZE=3D1 FACE=3D"Tahoma">dl-Cougar</FONT></SPAN></P>

<P ALIGN=3DLEFT><B><SPAN LANG=3D"en-us"><FONT SIZE=3D1 =
FACE=3D"Tahoma">Subject:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT>=
</SPAN></B><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT SIZE=3D1 FACE=3D"Tahoma">Submittal 4 build =
candidate not passing acceptance tests</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I =
loaded 3 cougar systems (1 2 node cluster and&nbsp; 1 standalone)&nbsp; =
with //depot/dev/&#8230;@27286 and I am seeing crashes in eventd, =
sdm_cfgd, evm_cfgd and FP.</FONT></SPAN></P>
<BR>
</UL></UL>
</BODY>
</HTML>
------_=_NextPart_001_01C854BB.C9B40725--
