X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C781D8.5FBA803D@onstor-exch02.onstor.net>; Wed, 18 Apr 2007 09:41:12 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C781D8.5FBA803D"
Content-class: urn:content-classes:message
Subject: RE: Installation Issues
Date: Wed, 18 Apr 2007 09:41:12 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E0351B61B@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E034373A2@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Installation Issues
Thread-Index: AceAehf2xAdT6G14RwWj2IPJn0a4HAAAAAIwAABBwvAAADLMUAAAX89DAABgQDAAAWUmsAAcslBwAAnCBRAALnCgUA==
From: "Charissa Willard" <charissa.willard@onstor.com>
To: "Jay Michlin" <jay.michlin@onstor.com>
Cc: "dl-Software" <dl-software@onstor.com>,
	"Eric Crutchlow" <eric.crutchlow@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C781D8.5FBA803D
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Jay,

=20

We did not run into any major issues during yesterday's installation of
a 2 node cluster at France Telecom in SSF. Moreover, we may have
identified an initial configuration issue that could be causing most of
the problems the SEs are encountering in the field. The assumption was
being made that once PM was started, it is okay to go ahead and start
configuring virtual servers and volumes. It isn't enough to just make
sure PM started. Configuration should only continue after PM
successfully starts all daemons on the filer. During yesterday's
install, I was able to log in as root and monitor all processes as they
were coming up. Once the last process (currently sendmail) was started,
only then did we continue configuration. There are a few other issues we
need to look at, but I think we need to address this one and not let
configuration be done until all daemons are started.=20

=20

-Charissa

=20

=20

________________________________

From: Jay Michlin=20
Sent: Tuesday, April 17, 2007 11:36 AM
To: dl-Software
Cc: Eric Crutchlow
Subject: FW: Installation Issues

=20

Hello all,

=20

The email below is background reading about some of the problems our
Porfessional Services team is seeing during installs and upgrades at
customer sites. Most of that is invisible to us most of the time. And in
the same way, most of our challenges in Software Development are
invisible to field people most of the time.=20

=20

We'll all benefit by understanding better what life is like in other
parts of the company, so I offer the email below for your interest.
There are no assignments or action items in it. Just information worth
seeing.

=20

jay

=20

=20

________________________________

From: Eric Crutchlow=20
Sent: Tuesday, April 17, 2007 8:57 AM
To: Jay Michlin; Paul Hammer; Sandrine Boulanger; Ed Kwan
Cc: Caeli Collins
Subject: RE: Installation Issues

Jay,

=20

I appreciate your thoughts on the matter, and obviously I have not made
my case. Please allow me to do so now.

=20

Prior to Sudheesh's meeting, the field was encountering the problems we
have outline below (the installation issues list). From that meeting and
the subsequent meeting with your group, we have made strides to educate
and focus our efforts. The investment of time has produced the
installations issues list and help get several serious bugs addressed. I
think we would all agree that the time spent in these meetings were
productive and needed.

=20

But lets review were the fields' time has been spent. Two recent
examples will illustrate this well; BAE and Fandango.=20

=20

One year ago, we attempted to demo to BAE and the system failed. We lost
a sale (I have not been given the details, but if it would be helpful,
please let me know). A year later, BAE lets us do another demo. Scott
Moyer (our SE) went to BAE and attempted to install a two node cluster.
He hits the PM on initial config error and attempts to configure the
system anyway. He basically goes through four flash cards and now has no
way to continue the install. Bill Duffy and I are at a customer's site
in San Diego performing an install. Bill gets called away to help Scott.
Eventually, he cuts short his stay with me to immediately fly to
Philadelphia and help. The customer is wondering what is going on and is
getting ready to throw us out. Scott has wasted 8 hours. Bill now spends
two days to get everything working as well as make sure the customer
understands that the Bobcats are reliable and will work for them. Now
ONStor has invested 24 hours on a possible sale.

=20

Luckily, BAE is impressed with Bill's 'performance'. As fate would have
it, BAE's current system fail shortly thereafter and they decide to
purchase the systems ASAP. Bill helped with the migration and so far no
issues.=20

=20

A few weeks ago I installed Fandango with a two node cluster and no
issues. Last Thursday they contacted me and said that they changed their
network addresses and would like to update there Bobcat's. The reaction
to the fact that the only way they could do this was by resetting the
boxes was not well received. But I assured them that since they haven't
gone into production, this would not be a major problem and would only
take 1.5 hours. I first broke the cluster, upgraded to 2.2.2.2, and then
performed the sys config reset. I ran into the PM problem that Scott hit
at BAE. Rebooting didn't work and I executed a second sys config reset
which did. 1.5 turned into 4 hours. We left the systems that day setup
with no vservers except for the management ones. Later that night at
2am, there was a cpu crash (see issue #5). Now Fandango is waiting on us
to explain this before going live.=20

=20

And so with the two meetings we have had, we have created the list and
are addressing the issues. As outlined below, the future releases will
do a better job. But Delorean is a month away and ONStor must address
these issues in the current release. And we are. Again, these two short
meetings have helped us make strides.

=20

The GUI has always been an issue and one that I was working on with
Charissa prior to the meetings. When the field reported issues with PM
or the GUI, the response has been they were not reproducible. As such
the field became very frustrated and has not filed defects. This is
unacceptable. Charissa has been willing to work with me on this and is
making progress to resolve it. She is a great example of your suggestion
of ongoing engagements.=20

=20

But until the two meetings, this was the only ongoing engagement we had.

=20

We have not directly addressed the fields' view of the support
organization (CS, QA, Dev). We need to change that. The meetings were a
start and well worth the time.

=20

The field has also been told many times that items are WAD. My recent
experience with Bill (defect 18502) illustrates this point. WAD is being
touted too many times. This contributes to the 'mythology' of how our
systems work and serves no purpose. As I stated before, if something is
WAD, it must be documented and the documentation must be easily
accessible to a user. In my opinion, this is either in the SAG or GUI
documentation. Management needs to address this and communicate it to
the team. There should not be (cannot be) confusion on what WAD is for
any particular component of our systems. I do not underestimate the
challenge of this nor am I proposing the meetings I am requesting
directly address this.

=20

And so I put it to the team; by having two meeting to create the
installation issues list we have helped focus future development efforts
as outlined by Jay. We are also addressing the issues in the current
release (no resolutions as of yet, but as Jay has stated, everyone is
committed to fixing them). Prior to the two meetings, our method for
dealing with this was ongoing engagements. I believe this has not worked
and we need to change. I am only asking for one meeting for one-half
hour, every two weeks (my apologizes Jay if I mis-communicated this as a
weekly meeting). I will create an agenda and keep minutes to ensure the
meetings are organized and effective. Our goal will be to address the
issues and improve teamwork. More importantly, I propose that if we can
incorporate these effort into our other meetings and processes, we no
longer need this one.

=20

As I have illustrated with BAE and Fandango, we are already spending the
time and have lost/jeopardized sales because of it. Is meeting once
every two weeks not worth the investment?=20

Regards,=20

Eric Crutchlow=20
Professional Services Manager=20
t: 408-376-3113=20
f: 408-963-2409=20
m: 408-596-1155=20

________________________________

From: Eric Crutchlow
Sent: Mon 4/16/2007 3:53 PM
To: Paul Hammer; Sandrine Boulanger; Ed Kwan; Jay Michlin
Subject: Installation Issues

When: Occurs every 2 weeks on Thursday effective 4/19/2007 from 1:00 PM
to 1:30 PM (GMT-08:00) Pacific Time (US & Canada).

Where: TBD=20

*~*~*~*~*~*~*~*~*~*=20

I'm following up with the installation issues meeting we had a few weeks
ago to resolve the issues PS and the SE's are seeing in the field. The
meeting will be every two weeks and I will keep minutes with action
items. The goals are to resolve the current list (see below) and put
processes in place to deal with future installation/eval issues as part
of normal escalations. Once these two goals are reached, we can
determine if there is any further need for additional meetings.

The following is a list of issues we currently are addressing:=20

1) Prior to init config on a new nas, get errors. Reboot off of 2nd
flash and no problems.=20

2) Do init config 1st time. After reboot, the nas generates pm errors.
If you reboot, you get init config, but this time the nas remembers the
default route you entered the first time. 2nd reboot and everything is
fine. We also see this with config resets.

3) Upgrades from 1.3.3.x to 2.x seems to work fine, but when you try and
move the vs from the 1.3.3.x server to the 2.x, won't work. A cluster
show cluster on the 2.x nas reports the cluster name as 'na'.  You kill
ncmd on the 1.3.3.x nas  and now cluster name is right and you can move
vs.

4) System copy all. Do a compare and get errors. If you reboot the nas,
perform another copy all, compare works. We wonder if flash has space
issues.

5) New install of a two node cluster. One of the nodes will reboot with
a cpu crash. Just happened at an install Bill did yesterday.

6) An upgrade takes 15 - 20 minutes. Compare another 10 - 15. This takes
a lot of time, especially if you get an error and have to start over.
You can waste 1 - 1.5 hours this way.


------_=_NextPart_001_01C781D8.5FBA803D
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<title>Installation Issues</title>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"country-region"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"City"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"place"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
font-face {
	FONT-FAMILY: Tahoma
}

 /* Font Definitions */
 @font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:purple;
	text-decoration:underline;}
p
	{mso-margin-top-alt:auto;
	margin-right:0in;
	mso-margin-bottom-alt:auto;
	margin-left:0in;
	font-size:12.0pt;
	font-family:"Times New Roman";}
span.emailstyle18
	{font-family:Arial;
	color:windowtext;
	font-weight:normal;
	font-style:normal;
	text-decoration:none none;}
span.EmailStyle19
	{mso-style-type:personal;
	font-family:Arial;
	color:windowtext;
	font-weight:normal;
	font-style:normal;
	text-decoration:none none;}
span.EmailStyle20
	{mso-style-type:personal;
	font-family:Arial;
	color:windowtext;
	font-weight:normal;
	font-style:normal;
	text-decoration:none none;}
span.EmailStyle21
	{mso-style-type:personal-reply;
	font-family:"Courier New";
	color:windowtext;
	font-weight:normal;
	font-style:normal;
	text-decoration:none none;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>Jay,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>We did not run into any major issues during yesterday&#8217;s
installation of a 2 node cluster at France Telecom in SSF. Moreover, we =
may
have identified an initial configuration issue that could be causing =
most of
the problems the SEs are encountering in the field. The assumption was =
being
made that once PM was started, it is okay to go ahead and start =
configuring virtual
servers and volumes. It isn&#8217;t enough to just make sure PM started. =
Configuration
should only continue after PM successfully starts all daemons on the =
filer.
During yesterday&#8217;s install, I was able to log in as root and =
monitor all
processes as they were coming up. Once the last process (currently =
sendmail) was
started, only then did we continue configuration. There are a few other =
issues
we need to look at, but I think we need to address this one and not let
configuration be done until all daemons are started. =
<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>-Charissa<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt;
font-family:"Courier New"'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt;
font-family:"Courier New"'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">Jay Michlin</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, April 17, =
2007
11:36 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> dl-Software<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Eric Crutchlow<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> FW: Installation =
Issues</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:blue'>Hello =
all,</span></font><o:p></o:p></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:blue'>The email below is background =
reading
about some of the problems our Porfessional Services team is seeing =
during
installs and upgrades at customer sites. Most of that is invisible to us =
most
of the time. And in the same way, most of our challenges in Software
Development are invisible to field people most of the time. =
</span></font><o:p></o:p></p>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:blue'>We'll all benefit by understanding =
better
what life is like in other parts of the company, so I offer the email =
below for
your interest. There are no assignments or action items in it. Just =
information
worth seeing.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:blue'>jay</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

</div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;font-family:Tahoma'> Eric
Crutchlow <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, April 17, =
2007 8:57
AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Jay
 Michlin</st1:PersonName>; Paul Hammer; <st1:PersonName =
w:st=3D"on">Sandrine
 Boulanger</st1:PersonName>; <st1:PersonName w:st=3D"on">Ed =
Kwan</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Caeli
 Collins</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: Installation =
Issues</span></font><o:p></o:p></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Jay,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>I appreciate your thoughts on the matter, and =
obviously I
have not made my case. Please allow me to do so =
now.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Prior to Sudheesh&#8217;s meeting, the field was
encountering the problems we have outline below (the installation issues =
list).
From that meeting and the subsequent meeting with your group, we have =
made
strides to educate and focus our efforts. The investment of time has =
produced
the installations issues list and help get several serious bugs =
addressed. I
think we would all agree that the time spent in these meetings were =
productive
and needed.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>But lets review were the fields&#8217; time has been =
spent.
Two recent examples will illustrate this well; BAE and Fandango. =
<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>One year ago, we attempted to demo to BAE and the =
system
failed. We lost a sale (I have not been given the details, but if it =
would be
helpful, please let me know). A year later, BAE lets us do another demo. =
<st1:PersonName
w:st=3D"on">Scott Moyer</st1:PersonName> (our SE) went to BAE and =
attempted to
install a two node cluster. He hits the PM on initial config error and =
attempts
to configure the system anyway. He basically goes through four flash =
cards and
now has no way to continue the install. <st1:PersonName w:st=3D"on">Bill =
Duffy</st1:PersonName>
and I are at a customer&#8217;s site in <st1:place w:st=3D"on"><st1:City =
w:st=3D"on">San
  Diego</st1:City></st1:place> performing an install. Bill gets called =
away to
help Scott. Eventually, he cuts short his stay with me to immediately =
fly to <st1:place
w:st=3D"on"><st1:City w:st=3D"on">Philadelphia</st1:City></st1:place> =
and help. The
customer is wondering what is going on and is getting ready to throw us =
out.
Scott has wasted 8 hours. Bill now spends two days to get everything =
working as
well as make sure the customer understands that the Bobcats are reliable =
and
will work for them. Now ONStor has invested 24 hours on a possible =
sale.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Luckily, BAE is impressed with Bill&#8217;s
&#8216;performance&#8217;. As fate would have it, BAE&#8217;s current =
system
fail shortly thereafter and they decide to purchase the systems ASAP. =
Bill
helped with the migration and so far no issues. =
<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>A few weeks ago I installed Fandango with a two node =
cluster
and no issues. Last Thursday they contacted me and said that they =
changed their
network addresses and would like to update there Bobcat&#8217;s. The =
reaction
to the fact that the only way they could do this was by resetting the =
boxes was
not well received. But I assured them that since they haven&#8217;t gone =
into
production, this would not be a major problem and would only take 1.5 =
hours. I
first broke the cluster, upgraded to 2.2.2.2, and then performed the sys =
config
reset. I ran into the PM problem that Scott hit at BAE. Rebooting =
didn&#8217;t
work and I executed a second sys config reset which did. 1.5 turned into =
4
hours. We left the systems that day setup with no vservers except for =
the
management ones. Later that night at 2am, there was a cpu crash (see =
issue #5).
Now Fandango is waiting on us to explain this before going live. =
<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>And so with the two meetings we have had, we have =
created
the list and are addressing the issues. As outlined below, the future =
releases
will do a better job. But Delorean is a month away and ONStor must =
address
these issues in the current release. And we are. Again, these two short
meetings have helped us make strides.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>The GUI has always been an issue and one that I was =
working
on with Charissa prior to the meetings. When the field reported issues =
with PM
or the GUI, the response has been they were not reproducible. As such =
the field
became very frustrated and has not filed defects. This is unacceptable.
Charissa has been willing to work with me on this and is making progress =
to
resolve it. She is a great example of your suggestion of ongoing =
engagements. <o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>But until the two meetings, this was the only ongoing
engagement we had.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>We have not directly addressed the fields&#8217; view =
of the
support organization (CS, QA, Dev). We need to change that. The meetings =
were a
start and well worth the time.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>The field has also been told many times that items =
are WAD.
My recent experience with Bill (defect 18502) illustrates this point. =
WAD is
being touted too many times. This contributes to the =
&#8216;mythology&#8217; of
how our systems work and serves no purpose. As I stated before, if =
something is
WAD, it must be documented and the documentation must be easily =
accessible to a
user. In my opinion, this is either in the SAG or GUI documentation. =
Management
needs to address this and communicate it to the team. There should not =
be
(cannot be) confusion on what WAD is for any particular component of our
systems. I do not underestimate the challenge of this nor am I proposing =
the
meetings I am requesting directly address =
this.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>And so I put it to the team; by having two meeting to =
create
the installation issues list we have helped focus future development =
efforts as
outlined by Jay. We are also addressing the issues in the current =
release (no
resolutions as of yet, but as Jay has stated, everyone is committed to =
fixing
them). Prior to the two meetings, our method for dealing with this was =
ongoing
engagements. I believe this has not worked and we need to change. I am =
only
asking for one meeting for one-half hour, every two weeks (my apologizes =
Jay if
I mis-communicated this as a weekly meeting). I will create an agenda =
and keep
minutes to ensure the meetings are organized and effective. Our goal =
will be to
address the issues and improve teamwork. More importantly, I propose =
that if we
can incorporate these effort into our other meetings and processes, we =
no
longer need this one.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>As I have illustrated with BAE and Fandango, we are =
already
spending the time and have lost/jeopardized sales because of it. Is =
meeting
once every two weeks not worth the investment? =
<o:p></o:p></span></font></p>

<div>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>Regards,</span></font>
<o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>Eric
Crutchlow</span></font> <br>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>Professional
Services Manager</span></font> <br>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>t:
408-376-3113</span></font> <br>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>f:
408-963-2409</span></font> <br>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>m:
408-596-1155</span></font> <o:p></o:p></p>

</div>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

</div>

<div>

<div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;font-family:Tahoma'> Eric
Crutchlow<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Mon 4/16/2007 3:53 =
PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Paul Hammer; =
<st1:PersonName
w:st=3D"on">Sandrine Boulanger</st1:PersonName>; <st1:PersonName =
w:st=3D"on">Ed
 Kwan</st1:PersonName>; <st1:PersonName w:st=3D"on">Jay =
Michlin</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> Installation =
Issues</span></font><o:p></o:p></p>

</div>

<div>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>When:
Occurs every 2 weeks on Thursday effective 4/19/2007 from 1:00 PM to =
1:30 PM
(GMT-08:00) Pacific Time (US &amp; <st1:place =
w:st=3D"on"><st1:country-region
 =
w:st=3D"on">Canada</st1:country-region></st1:place>).</span></font><o:p><=
/o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>Where:
TBD</span></font> <o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>*~*~*~*~*~*~*~*~*~*</span></=
font>
<o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>I'm
following up with the installation issues meeting we had a few weeks ago =
to
resolve the issues PS and the SE's are seeing in the field. The meeting =
will be
every two weeks and I will keep minutes with action items. The goals are =
to
resolve the current list (see below) and put processes in place to deal =
with
future installation/eval issues as part of normal escalations. Once =
these two
goals are reached, we can determine if there is any further need for =
additional
meetings.</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>The
following is a list of issues we currently are addressing:</span></font> =
<o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>1)
Prior to init config on a new nas, get errors. Reboot off of 2nd flash =
and no
problems.</span></font> <o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>2)
Do init config 1st time. After reboot, the nas generates pm errors. If =
you
reboot, you get init config, but this time the nas remembers the default =
route
you entered the first time. 2nd reboot and everything is fine. We also =
see this
with config resets.</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>3)
Upgrades from 1.3.3.x to 2.x seems to work fine, but when you try and =
move the vs
from the 1.3.3.x server to the 2.x, won&#8217;t work. A cluster show =
cluster on
the 2.x nas reports the cluster name as &#8216;na&#8217;.&nbsp; You kill =
ncmd
on the 1.3.3.x nas&nbsp; and now cluster name is right and you can move =
vs.</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>4)
System copy all. Do a compare and get errors. If you reboot the nas, =
perform
another copy all, compare works. We wonder if flash has space =
issues.</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>5)
New install of a two node cluster. One of the nodes will reboot with a =
cpu
crash. Just happened at an install Bill did =
yesterday.</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>6)
An upgrade takes 15</span></font> <font size=3D2 face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma'>&#8211;</span></font><font =
size=3D2
face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial'> 20 =
minutes.
Compare another 10</span></font> <font size=3D2 face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma'>&#8211;</span></font><font =
size=3D2
face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial'> 15. =
This takes a
lot of time, especially if you get an error and have to start over. You =
can
waste 1</span></font> <font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma'>&#8211;</span></font><font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'> 1.5 hours this =
way.</span></font><o:p></o:p></p>

</div>

</div>

</div>

</body>

</html>

------_=_NextPart_001_01C781D8.5FBA803D--
