X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C78B7F.1EB7BEEE@onstor-exch02.onstor.net>; Mon, 30 Apr 2007 16:27:29 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C78B7F.1EB7BEEE"
Content-class: urn:content-classes:message
Subject: FW: Focus on Install Problems
Date: Mon, 30 Apr 2007 16:27:29 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E02215782@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Focus on Install Problems
Thread-Index: AceJK0/tj1HjaUlMTMuKaxap6208bABTgpfDADTPXPAADIQ3kA==
From: "Larry Scheer" <larry.scheer@onstor.com>
To: "Andy Sharp" <andy.sharp@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C78B7F.1EB7BEEE
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Andy,

    I was talking with Jay in my 1on1 about the install situation. He
sent me this email and suggested I share it with you. It is just
information to let us know what management has been discussing. No
action is required from either of us.

=20

Larry

=20

________________________________

From: Jay Michlin=20
Sent: Monday, April 30, 2007 4:16 PM
To: Larry Scheer
Subject: FW: Focus on Install Problems

=20

FYI.

=20

________________________________

From: Paul Hammer=20
Sent: Sunday, April 29, 2007 9:38 AM
To: Jay Michlin; Caeli Collins; Jerry Lopatin
Cc: Brian DeForest; Tim Gardner
Subject: RE: Focus on Install Problems

Thanks Jay. A few comments and updates in-line.

________________________________

From: Jay Michlin
Sent: Fri 4/27/2007 5:22 PM
To: Caeli Collins; Jerry Lopatin; Paul Hammer
Cc: Brian DeForest; Tim Gardner
Subject: Focus on Install Problems

Caeli, Paul, Jerry,

As we discussed, here is a summary of work ongoing, planned and proposed
to address field install issues.

1. In Delorean

* We have committed to fix the Install "PM Down" problem. Sandrine has
been successful in reproducing it, and we have developed a fix, which is
currently in the Delorean code. We can't yet be certain this fixes every
instance encountered in the field, so we are still pursuing the problem.
But we are certain that the fix we've built so far attends to a serious
current problem and will harden the overlll system as well as the
install process.

** John reproduced the OCT PM failures a couple of weeks back in the HW
lab on new systems. Tried it on two different systems, both showed the
same failier. Think this is the one that CE/SE's are seeing.=20


* Andy's change to the flash layout will alleviate many apparently
random problems caused by running out of space in flash-based file
systems. It can also transparently support 1G flash parts, and if we
choose to ship those, we will further alleviate problems with flash
congestion. We are convinced that many install problems come from this
issue, and suspect that many of the NTF returns represent this too
(since we don't test returns for file system space allocation). Most
likely, many instances of "try rebooting a second time and it works"
come from flash file system exhaustion too, and those too will be fixed
by our new layout.

* Larry's improved upgrade program is generally more robust and will
eliminate some problems.

* QA is improving their test coverage of the install process to cover
workflows that they did not previously consider current.

** Going to add tests for the flash replacement workflow and for
upgrading the primary flash. Also we are going to annotate the
installation documents so that we all know why these steps are
necessary, hoping that this will cut down on shortcuts being taken
trying to save time.

2. After Delorean

* We are nominally discussing a release called Zonda which will be
focused on improving availability. We have always assumed that this
implies improvement of the CS and SE experience as well as software
hardening. The plan is to conduct a requirements gathering process in
which CS will be a key participant and SE will be involved to the degree
they are available.

* We have reviewed a document of "Support Requirements for Engineering",
http://wiki.onstor.net/wiki/Support_requirements_for_engineering_roadmap
It will serve as a starting point for improvements in the release after
Delorean, though we can't guarantee all the items it specifies without
additional discussion.

* The release after Delorean could become multiple releases if it is
urgent to deploy some CS and SE assistance early. As always, the
overhead of this will detract from what we can do overall, but the
option to consider it will be part of planning, and CS will participate.

* SW Development hopes to continue our just started program of sending
engineers to participate in field installs, and we have several
enthusiastic volunteers. It illuminates the process and problems for us
and has direct and specific impact on the software we produc.

3. Additionally

* Paul is editing a White Paper on install procedures to better educate
the field on what works and what to watch out for.

* Charissa, in a field install visit, learned that some problems occur
because even though the product "thinks" it is up and running, and it
accepts shell commands, in fact some daemons may not yet have completed
initialization. Issuing a shell command in this situation causes a crash
or other errors. We are sizing the project to make our nfxshell test
whether all the required daemons are running before accepting commands.
If the work is within reason, we will propose including it in Delorean.

* The current software upgrade process in Delorean (not requiring
shipping flash cards) is two steps. The first would upgrade the upgrade
program itself to the more robust version that comes with Delorean. If
we wanted, we could also modify this first upgrade to reformat the flash
to the improved layout. Even for customers who do not want to move to
Delorean, this would harden all future upgrades and installs. It is
available now (except that it would take an additional week to include
the flash reformat function if we want it), and it is relatively benign
and easy to test.

IN SUMMARY
We take this very seriously and have not worked on it before only
because it was invisible to us. In most cases we are not working on
"field install issues" as standalone projects because doing so would be
a huge blow to all the other work we are committed to. But we have
already launched a series of initiatives with immediate benefits and
more soon to come, and we are also launching both immediate (next
project) and long term initiatives to make attention to the field a
standard and ongoing part of the way we operate.

jay






------_=_NextPart_001_01C78B7F.1EB7BEEE
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<title>Focus on Install Problems</title>
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:blue;
	text-decoration:underline;}
p
	{mso-margin-top-alt:auto;
	margin-right:0in;
	mso-margin-bottom-alt:auto;
	margin-left:0in;
	font-size:12.0pt;
	font-family:"Times New Roman";}
span.EmailStyle18
	{mso-style-type:personal-reply;
	font-family:Arial;
	color:navy;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
	{page:Section1;}
-->
</style>

</head>

<body lang=3DEN-US link=3Dblue vlink=3Dblue>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Andy,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp; I was talking =
with Jay
in my 1on1 about the install situation. He sent me this email and =
suggested I
share it with you. It is just information to let us know what management =
has been
discussing. No action is required from either of =
us.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Larry<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> Jay =
Michlin <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, April 30, =
2007 4:16
PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Larry Scheer<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> FW: Focus on =
Install
Problems</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:blue'>FYI.</span></font><o:p></o:p></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;font-family:Tahoma'> Paul
Hammer <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Sunday, April 29, =
2007 9:38
AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Jay Michlin; Caeli =
Collins;
Jerry Lopatin<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Brian DeForest; Tim =
Gardner<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: Focus on =
Install
Problems</span></font><o:p></o:p></p>

<div id=3DidOWAReplyText70964>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>Thanks Jay. A few comments and updates =
in-line.<o:p></o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;font-family:Tahoma'> Jay
Michlin<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Fri 4/27/2007 5:22 =
PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Caeli Collins; Jerry =
Lopatin;
Paul Hammer<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Brian DeForest; Tim =
Gardner<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> Focus on Install =
Problems</span></font><o:p></o:p></p>

</div>

</div>

<div>

<p><font size=3D2 face=3D"Times New Roman"><span =
style=3D'font-size:10.0pt'>Caeli,
Paul, Jerry,<br>
<br>
As we discussed, here is a summary of work ongoing, planned and proposed =
to
address field install issues.<br>
<br>
1. In Delorean<br>
<br>
* We have committed to fix the Install &quot;PM Down&quot; problem. =
Sandrine
has been successful in reproducing it, and we have developed a fix, =
which is
currently in the Delorean code. We can't yet be certain this fixes every
instance encountered in the field, so we are still pursuing the problem. =
But we
are certain that the fix we've built so far attends to a serious current
problem and will harden the overlll system as well as the install =
process.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3D"Times New Roman"><span =
style=3D'font-size:
10.0pt'>** John reproduced the OCT PM failures a couple of weeks back in =
the HW
lab on new systems. Tried it on two different systems, both showed the =
same
failier. Think this is the one that CE/SE's are seeing. =
<o:p></o:p></span></font></p>

<p><font size=3D2 face=3D"Times New Roman"><span =
style=3D'font-size:10.0pt'><br>
* Andy's change to the flash layout will alleviate many apparently =
random problems
caused by running out of space in flash-based file systems. It can also
transparently support 1G flash parts, and if we choose to ship those, we =
will
further alleviate problems with flash congestion. We are convinced that =
many
install problems come from this issue, and suspect that many of the NTF =
returns
represent this too (since we don't test returns for file system space
allocation). Most likely, many instances of &quot;try rebooting a second =
time
and it works&quot; come from flash file system exhaustion too, and those =
too
will be fixed by our new layout.<br>
<br>
* Larry's improved upgrade program is generally more robust and will =
eliminate
some problems.<br>
<br>
* QA is improving their test coverage of the install process to cover =
workflows
that they did not previously consider =
current.<o:p></o:p></span></font></p>

<p style=3D'margin-bottom:12.0pt'><font size=3D2 face=3D"Times New =
Roman"><span
style=3D'font-size:10.0pt'>** Going to add tests for the flash =
replacement
workflow and for upgrading the primary flash. Also we are going to =
annotate the
installation documents so that we all know why these steps are =
necessary,
hoping that this will cut down on shortcuts being taken trying to save =
time.<br>
<br>
2. After Delorean<br>
<br>
* We are nominally discussing a release called Zonda which will be =
focused on
improving availability. We have always assumed that this implies =
improvement of
the CS and SE experience as well as software hardening. The plan is to =
conduct
a requirements gathering process in which CS will be a key participant =
and SE
will be involved to the degree they are available.<br>
<br>
* We have reviewed a document of &quot;Support Requirements for
Engineering&quot;, <a
href=3D"http://wiki.onstor.net/wiki/Support_requirements_for_engineering_=
roadmap">http://wiki.onstor.net/wiki/Support_requirements_for_engineering=
_roadmap</a><br>
It will serve as a starting point for improvements in the release after
Delorean, though we can't guarantee all the items it specifies without
additional discussion.<br>
<br>
* The release after Delorean could become multiple releases if it is =
urgent to
deploy some CS and SE assistance early. As always, the overhead of this =
will
detract from what we can do overall, but the option to consider it will =
be part
of planning, and CS will participate.<br>
<br>
* SW Development hopes to continue our just started program of sending
engineers to participate in field installs, and we have several =
enthusiastic
volunteers. It illuminates the process and problems for us and has =
direct and
specific impact on the software we produc.<br>
<br>
3. Additionally<br>
<br>
* Paul is editing a White Paper on install procedures to better educate =
the
field on what works and what to watch out for.<br>
<br>
* Charissa, in a field install visit, learned that some problems occur =
because
even though the product &quot;thinks&quot; it is up and running, and it =
accepts
shell commands, in fact some daemons may not yet have completed =
initialization.
Issuing a shell command in this situation causes a crash or other =
errors. We
are sizing the project to make our nfxshell test whether all the =
required
daemons are running before accepting commands. If the work is within =
reason, we
will propose including it in Delorean.<br>
<br>
* The current software upgrade process in Delorean (not requiring =
shipping
flash cards) is two steps. The first would upgrade the upgrade program =
itself
to the more robust version that comes with Delorean. If we wanted, we =
could
also modify this first upgrade to reformat the flash to the improved =
layout.
Even for customers who do not want to move to Delorean, this would =
harden all
future upgrades and installs. It is available now (except that it would =
take an
additional week to include the flash reformat function if we want it), =
and it is
relatively benign and easy to test.<br>
<br>
IN SUMMARY<br>
We take this very seriously and have not worked on it before only =
because it
was invisible to us. In most cases we are not working on &quot;field =
install
issues&quot; as standalone projects because doing so would be a huge =
blow to
all the other work we are committed to. But we have already launched a =
series
of initiatives with immediate benefits and more soon to come, and we are =
also
launching both immediate (next project) and long term initiatives to =
make
attention to the field a standard and ongoing part of the way we =
operate.<br>
<br>
jay<br>
<br>
<br>
<br>
<o:p></o:p></span></font></p>

</div>

</div>

</body>

</html>

------_=_NextPart_001_01C78B7F.1EB7BEEE--
