X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7778F.36148623@onstor-exch02.onstor.net>; Thu, 5 Apr 2007 07:32:17 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7778F.36148623"
Content-class: urn:content-classes:message
Subject: RE: PM down during first install
Date: Thu, 5 Apr 2007 07:32:15 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E031DF78F@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E031DF69B@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: PM down during first install
Thread-Index: Acd3I6wPoulbIFZmTzazk8oB7oaitwAaUSXQ
From: "Eric Crutchlow" <eric.crutchlow@onstor.com>
To: "Brian DeForest" <brian.deforest@onstor.com>,
	"Paul Hammer" <paul.hammer@onstor.com>,
	"Sandrine Boulanger" <sandrine.boulanger@onstor.com>,
	"Raj Kumar" <raj.kumar@onstor.com>,
	"Jay Michlin" <jay.michlin@onstor.com>,
	"Tim Gardner" <tim.gardner@onstor.com>,
	"Larry Scheer" <larry.scheer@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>,
	"Jerry Lopatin" <jerry.lopatin@onstor.com>,
	"Raghu Murthy" <raghu.murthy@onstor.com>
Cc: "Bill Duffy" <bill.duffy@onstor.com>,
	"Caeli Collins" <caeli.collins@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7778F.36148623
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Brian,

Thanks for the update. This is one of the most common issues we see at
installs and it really makes us look bad in front of a customer. I can't
tell you how much this causes customers to question our ability to work
in production. Two recent installs, Fandango and Genoptix were done in
the past few weeks and we are now hand-holding them daily to make sure
they understand the systems will work and we are on top of any issues
(Genoptix had the pm issue, both had issues with the GUI).=20

So far we have seen pm issues in the areas of initial config, right
after config, during the copy of data to a vol, and when Sandrine was
modifying ntp in a cluster. I don't know if they are all related, but I
am glad we have cores and you are addresses them.

Bill and I have reviewed our recent installs and have come up with a
better approach to the sequence of steps we take to perform them. It
basically requires several reboots to insure the box is 'clear' and in a
consistent, stable state. I will update everyone tomorrow on how it goes
today.

You have noted two new defects against Delorean. Will this also be
addressed in the 2.2.x.x version?

Regards,

Eric Crutchlow
Professional Services Manager
t: 408-376-3113
f: 408-963-2409
m: 408-596-1155

_____________________________________________
From: Brian DeForest=20
Sent: Wednesday, April 04, 2007 6:42 PM
To: Paul Hammer; Sandrine Boulanger; Raj Kumar; Jay Michlin; Tim
Gardner; Larry Scheer; Andy Sharp; Jerry Lopatin; Eric Crutchlow; Raghu
Murthy
Subject: PM down during first install

An update on the "PM down" problem Eric reported.

Sandrine has been successful in reproducing this in-house.   There are 2
defects filed against Delorean:

18354
18385

Both appear to be due to memory allocation failures.   Currently, BSD
has been instrumented to core dump an application when memory
allocations fail in an effort to troubleshoot which app(s) are running
out of memory.   This doesn't tell us the root cause, but it is a first
step toward isolating the problem, so don't get too excited.




------_=_NextPart_001_01C7778F.36148623
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7652.24">
<TITLE>RE: PM down during first install</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Brian,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Thanks for the update. This is one of the most common =
issues we see at installs and it really makes us look bad in front of a =
customer. I can</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">&#8217;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">t tell you how much this causes customers to question our =
ability to work in production. Two recent installs, Fandango and =
Genoptix</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial"></FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 =
FACE=3D"Arial">were</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT SIZE=3D2 FACE=3D"Arial">done in =
the</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT =
SIZE=3D2 FACE=3D"Arial">past few weeks and we are now hand-holding them =
daily to make sure they understand the systems will work and we are on =
top of any issues</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial"> (</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Genoptix had the pm issue, both had issues with the =
GUI)</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 FACE=3D"Arial">. </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">So =
far we have seen pm issues in the</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 =
FACE=3D"Arial">areas</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial"> of initial config, right =
after config, during the copy of data to a vol, and when Sandrine =
was</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT =
SIZE=3D2 FACE=3D"Arial">modifying ntp in a cluster. I =
don</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 FACE=3D"Arial">&#8217;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">t know if they are all related, but</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT SIZE=3D2 =
FACE=3D"Arial">I am glad we have cores and you are addresses =
them.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Bill =
and I have reviewed our recent installs and have come up with =
a</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT =
SIZE=3D2 FACE=3D"Arial">better approach to the sequence of steps we take =
to perform them. It basically requires several reboots to insure the box =
is</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT =
SIZE=3D2 FACE=3D"Arial">&#8216;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">clear</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">&#8217;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial"> and in a consistent, stable state. I will update =
everyone tomorrow on how it goes today.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">You =
have noted two new defects against Delorean. Will this also be addressed =
in the 2.2.x.x version?</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"></SPAN><A NAME=3D""><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Regards,</FONT></SPAN></A></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Eric =
Crutchlow</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Professional Services Manager</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">t: =
408-376-3113</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">f: =
408-963-2409</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">m: =
408-596-1155</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Wednesday, April 04, 2007 =
6:42 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Paul Hammer; Sandrine =
Boulanger; Raj Kumar; Jay Michlin; Tim Gardner; Larry Scheer; Andy =
Sharp; Jerry Lopatin; Eric Crutchlow; Raghu Murthy<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> PM down during first install</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">An =
update on the &#8220;PM down&#8221; problem Eric =
reported.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Sandrine has been successful in reproducing this =
in-house.&nbsp;&nbsp; There are 2 defects filed against =
Delorean:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">18354</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">18385</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Both =
appear to be due to memory allocation failures.&nbsp;&nbsp; Currently, =
BSD has been instrumented to core dump an application when memory =
allocations fail in an effort to troubleshoot which app(s) are running =
out of memory.&nbsp;&nbsp; This doesn&#8217;t tell us the root cause, =
but it is a first step toward isolating the problem, so don&#8217;t get =
too excited.</FONT></SPAN></P>
<BR>
<BR>

</BODY>
</HTML>
------_=_NextPart_001_01C7778F.36148623--
