X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C781DC.B4623F95@onstor-exch02.onstor.net>; Wed, 18 Apr 2007 10:12:12 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C781DC.B4623F95"
Content-class: urn:content-classes:message
Subject: RE: Installation Issues
Date: Wed, 18 Apr 2007 10:08:35 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E023B38C6@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Installation Issues
Thread-Index: AceAehf2xAdT6G14RwWj2IPJn0a4HAAAAAIwAABBwvAAADLMUAAAX89DAABgQDAAAWUmsAAcslBwAAnCBRAALnCgUAAAPp6wAADJhpw=
References: <BB375AF679D4A34E9CA8DFA650E2B04E0351B629@onstor-exch02.onstor.net>
From: "Paul Hammer" <paul.hammer@onstor.com>
To: "Jay Michlin" <jay.michlin@onstor.com>,
	"Charissa Willard" <charissa.willard@onstor.com>
Cc: "dl-Software" <dl-software@onstor.com>,
	"Eric Crutchlow" <eric.crutchlow@onstor.com>,
	"Jerry Lopatin" <jerry.lopatin@onstor.com>,
	"Caeli Collins" <caeli.collins@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C781DC.B4623F95
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Very good observation. We have discussed previously adding a message to =
state when the box is fully enabled and ready to be configured (i.e all =
the daemons are up and the volumes are all mounted). There are numerous =
things that fail to work when the box is not fully enabled, believe =
there are several defects on this one. One thing we discussed was =
disabling nfxshl until everything is up to prevent some configuration =
failures.

________________________________

From: Jay Michlin
Sent: Wed 4/18/2007 9:49 AM
To: Charissa Willard
Cc: dl-Software; Eric Crutchlow; Jerry Lopatin; Paul Hammer; Caeli =
Collins
Subject: RE: Installation Issues


Charissa,
=20
This is great. Thanks for the update and for sharing it with the group. =
We should still continue working on installation issues already =
underway, including the P1 "PM Down" issue in Delorean. And we should =
also figure out how to use what you learned yesterday as part of our =
Zonda planning.
=20
jay
=20

________________________________

From: Charissa Willard=20
Sent: Wednesday, April 18, 2007 9:41 AM
To: Jay Michlin
Cc: dl-Software; Eric Crutchlow
Subject: RE: Installation Issues



Jay,

=20

We did not run into any major issues during yesterday's installation of =
a 2 node cluster at France Telecom in SSF. Moreover, we may have =
identified an initial configuration issue that could be causing most of =
the problems the SEs are encountering in the field. The assumption was =
being made that once PM was started, it is okay to go ahead and start =
configuring virtual servers and volumes. It isn't enough to just make =
sure PM started. Configuration should only continue after PM =
successfully starts all daemons on the filer. During yesterday's =
install, I was able to log in as root and monitor all processes as they =
were coming up. Once the last process (currently sendmail) was started, =
only then did we continue configuration. There are a few other issues we =
need to look at, but I think we need to address this one and not let =
configuration be done until all daemons are started.=20

=20

-Charissa

=20

=20

________________________________

From: Jay Michlin=20
Sent: Tuesday, April 17, 2007 11:36 AM
To: dl-Software
Cc: Eric Crutchlow
Subject: FW: Installation Issues

=20

Hello all,

=20

The email below is background reading about some of the problems our =
Porfessional Services team is seeing during installs and upgrades at =
customer sites. Most of that is invisible to us most of the time. And in =
the same way, most of our challenges in Software Development are =
invisible to field people most of the time.=20

=20

We'll all benefit by understanding better what life is like in other =
parts of the company, so I offer the email below for your interest. =
There are no assignments or action items in it. Just information worth =
seeing.

=20

jay

=20

=20

________________________________

From: Eric Crutchlow=20
Sent: Tuesday, April 17, 2007 8:57 AM
To: Jay Michlin; Paul Hammer; Sandrine Boulanger; Ed Kwan
Cc: Caeli Collins
Subject: RE: Installation Issues

Jay,

=20

I appreciate your thoughts on the matter, and obviously I have not made =
my case. Please allow me to do so now.

=20

Prior to Sudheesh's meeting, the field was encountering the problems we =
have outline below (the installation issues list). From that meeting and =
the subsequent meeting with your group, we have made strides to educate =
and focus our efforts. The investment of time has produced the =
installations issues list and help get several serious bugs addressed. I =
think we would all agree that the time spent in these meetings were =
productive and needed.

=20

But lets review were the fields' time has been spent. Two recent =
examples will illustrate this well; BAE and Fandango.=20

=20

One year ago, we attempted to demo to BAE and the system failed. We lost =
a sale (I have not been given the details, but if it would be helpful, =
please let me know). A year later, BAE lets us do another demo. Scott =
Moyer (our SE) went to BAE and attempted to install a two node cluster. =
He hits the PM on initial config error and attempts to configure the =
system anyway. He basically goes through four flash cards and now has no =
way to continue the install. Bill Duffy and I are at a customer's site =
in San Diego performing an install. Bill gets called away to help Scott. =
Eventually, he cuts short his stay with me to immediately fly to =
Philadelphia and help. The customer is wondering what is going on and is =
getting ready to throw us out. Scott has wasted 8 hours. Bill now spends =
two days to get everything working as well as make sure the customer =
understands that the Bobcats are reliable and will work for them. Now =
ONStor has invested 24 hours on a possible sale.

=20

Luckily, BAE is impressed with Bill's 'performance'. As fate would have =
it, BAE's current system fail shortly thereafter and they decide to =
purchase the systems ASAP. Bill helped with the migration and so far no =
issues.=20

=20

A few weeks ago I installed Fandango with a two node cluster and no =
issues. Last Thursday they contacted me and said that they changed their =
network addresses and would like to update there Bobcat's. The reaction =
to the fact that the only way they could do this was by resetting the =
boxes was not well received. But I assured them that since they haven't =
gone into production, this would not be a major problem and would only =
take 1.5 hours. I first broke the cluster, upgraded to 2.2.2.2, and then =
performed the sys config reset. I ran into the PM problem that Scott hit =
at BAE. Rebooting didn't work and I executed a second sys config reset =
which did. 1.5 turned into 4 hours. We left the systems that day setup =
with no vservers except for the management ones. Later that night at =
2am, there was a cpu crash (see issue #5). Now Fandango is waiting on us =
to explain this before going live.=20

=20

And so with the two meetings we have had, we have created the list and =
are addressing the issues. As outlined below, the future releases will =
do a better job. But Delorean is a month away and ONStor must address =
these issues in the current release. And we are. Again, these two short =
meetings have helped us make strides.

=20

The GUI has always been an issue and one that I was working on with =
Charissa prior to the meetings. When the field reported issues with PM =
or the GUI, the response has been they were not reproducible. As such =
the field became very frustrated and has not filed defects. This is =
unacceptable. Charissa has been willing to work with me on this and is =
making progress to resolve it. She is a great example of your suggestion =
of ongoing engagements.=20

=20

But until the two meetings, this was the only ongoing engagement we had.

=20

We have not directly addressed the fields' view of the support =
organization (CS, QA, Dev). We need to change that. The meetings were a =
start and well worth the time.

=20

The field has also been told many times that items are WAD. My recent =
experience with Bill (defect 18502) illustrates this point. WAD is being =
touted too many times. This contributes to the 'mythology' of how our =
systems work and serves no purpose. As I stated before, if something is =
WAD, it must be documented and the documentation must be easily =
accessible to a user. In my opinion, this is either in the SAG or GUI =
documentation. Management needs to address this and communicate it to =
the team. There should not be (cannot be) confusion on what WAD is for =
any particular component of our systems. I do not underestimate the =
challenge of this nor am I proposing the meetings I am requesting =
directly address this.

=20

And so I put it to the team; by having two meeting to create the =
installation issues list we have helped focus future development efforts =
as outlined by Jay. We are also addressing the issues in the current =
release (no resolutions as of yet, but as Jay has stated, everyone is =
committed to fixing them). Prior to the two meetings, our method for =
dealing with this was ongoing engagements. I believe this has not worked =
and we need to change. I am only asking for one meeting for one-half =
hour, every two weeks (my apologizes Jay if I mis-communicated this as a =
weekly meeting). I will create an agenda and keep minutes to ensure the =
meetings are organized and effective. Our goal will be to address the =
issues and improve teamwork. More importantly, I propose that if we can =
incorporate these effort into our other meetings and processes, we no =
longer need this one.

=20

As I have illustrated with BAE and Fandango, we are already spending the =
time and have lost/jeopardized sales because of it. Is meeting once =
every two weeks not worth the investment?=20

Regards,=20

Eric Crutchlow=20
Professional Services Manager=20
t: 408-376-3113=20
f: 408-963-2409=20
m: 408-596-1155=20

________________________________

From: Eric Crutchlow
Sent: Mon 4/16/2007 3:53 PM
To: Paul Hammer; Sandrine Boulanger; Ed Kwan; Jay Michlin
Subject: Installation Issues

When: Occurs every 2 weeks on Thursday effective 4/19/2007 from 1:00 PM =
to 1:30 PM (GMT-08:00) Pacific Time (US & Canada).

Where: TBD=20

*~*~*~*~*~*~*~*~*~*=20

I'm following up with the installation issues meeting we had a few weeks =
ago to resolve the issues PS and the SE's are seeing in the field. The =
meeting will be every two weeks and I will keep minutes with action =
items. The goals are to resolve the current list (see below) and put =
processes in place to deal with future installation/eval issues as part =
of normal escalations. Once these two goals are reached, we can =
determine if there is any further need for additional meetings.

The following is a list of issues we currently are addressing:=20

1) Prior to init config on a new nas, get errors. Reboot off of 2nd =
flash and no problems.=20

2) Do init config 1st time. After reboot, the nas generates pm errors. =
If you reboot, you get init config, but this time the nas remembers the =
default route you entered the first time. 2nd reboot and everything is =
fine. We also see this with config resets.

3) Upgrades from 1.3.3.x to 2.x seems to work fine, but when you try and =
move the vs from the 1.3.3.x server to the 2.x, won't work. A cluster =
show cluster on the 2.x nas reports the cluster name as 'na'.  You kill =
ncmd on the 1.3.3.x nas  and now cluster name is right and you can move =
vs.

4) System copy all. Do a compare and get errors. If you reboot the nas, =
perform another copy all, compare works. We wonder if flash has space =
issues.

5) New install of a two node cluster. One of the nodes will reboot with =
a cpu crash. Just happened at an install Bill did yesterday.

6) An upgrade takes 15 - 20 minutes. Compare another 10 - 15. This takes =
a lot of time, especially if you get an error and have to start over. =
You can waste 1 - 1.5 hours this way.


------_=_NextPart_001_01C781DC.B4623F95
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<HTML dir=3Dltr><HEAD><TITLE>Installation Issues</TITLE>=0A=
<META http-equiv=3DContent-Type content=3D"text/html; charset=3Dunicode">=0A=
<META content=3D"MSHTML 6.00.2900.3059" name=3DGENERATOR>=0A=
<STYLE>font-face {=0A=
	font-family: Tahoma;=0A=
}=0A=
=0A=
font-face {=0A=
	FONT-FAMILY: Tahoma=0A=
}=0A=
P.MsoNormal {=0A=
	FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"=0A=
}=0A=
LI.MsoNormal {=0A=
	FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"=0A=
}=0A=
DIV.MsoNormal {=0A=
	FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"=0A=
}=0A=
A:link {=0A=
	COLOR: blue; TEXT-DECORATION: underline=0A=
}=0A=
SPAN.MsoHyperlink {=0A=
	COLOR: blue; TEXT-DECORATION: underline=0A=
}=0A=
A:visited {=0A=
	COLOR: purple; TEXT-DECORATION: underline=0A=
}=0A=
SPAN.MsoHyperlinkFollowed {=0A=
	COLOR: purple; TEXT-DECORATION: underline=0A=
}=0A=
P {=0A=
	FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY: =
"Times New Roman";}=0A=
SPAN.emailstyle18 {=0A=
	FONT-WEIGHT: normal; COLOR: windowtext; FONT-STYLE: normal; =
FONT-FAMILY: Arial; TEXT-DECORATION: none=0A=
}=0A=
SPAN.EmailStyle19 {=0A=
	FONT-WEIGHT: normal; COLOR: windowtext; FONT-STYLE: normal; =
FONT-FAMILY: Arial; TEXT-DECORATION: none;}=0A=
SPAN.EmailStyle20 {=0A=
	FONT-WEIGHT: normal; COLOR: windowtext; FONT-STYLE: normal; =
FONT-FAMILY: Arial; TEXT-DECORATION: none;}=0A=
SPAN.EmailStyle21 {=0A=
	FONT-WEIGHT: normal; COLOR: windowtext; FONT-STYLE: normal; =
FONT-FAMILY: "Courier New"; TEXT-DECORATION: none;}=0A=
DIV.Section1 {=0A=
	page: Section1=0A=
}=0A=
</STYLE>=0A=
</HEAD>=0A=
<BODY lang=3DEN-US vLink=3Dpurple link=3Dblue>=0A=
<DIV id=3DidOWAReplyText66896 dir=3Dltr>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 size=3D2>Very good =
observation. We have discussed previously adding a message to state when =
the box is fully enabled and ready to be configured (i.e all the daemons =
are up and the volumes are all mounted). There are numerous things that =
fail to work when the box is not fully enabled, believe there are =
several defects on this one. One thing we discussed was disabling nfxshl =
until everything is up to prevent some configuration =
failures.</FONT></DIV></DIV>=0A=
<DIV dir=3Dltr><BR>=0A=
<HR tabIndex=3D-1>=0A=
<FONT face=3DTahoma size=3D2><B>From:</B> Jay Michlin<BR><B>Sent:</B> =
Wed 4/18/2007 9:49 AM<BR><B>To:</B> Charissa Willard<BR><B>Cc:</B> =
dl-Software; Eric Crutchlow; Jerry Lopatin; Paul Hammer; Caeli =
Collins<BR><B>Subject:</B> RE: Installation Issues<BR></FONT><BR></DIV>=0A=
<DIV>=0A=
<DIV dir=3Dltr align=3Dleft><FONT face=3DArial color=3D#0000ff =
size=3D2><SPAN class=3D554034616-18042007>Charissa,</SPAN></FONT></DIV>=0A=
<DIV dir=3Dltr align=3Dleft><FONT face=3DArial color=3D#0000ff =
size=3D2><SPAN class=3D554034616-18042007></SPAN></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr align=3Dleft><FONT face=3DArial color=3D#0000ff =
size=3D2><SPAN class=3D554034616-18042007>This is great. Thanks for the =
update and for sharing it with the group. We should still continue =
working on installation issues already underway, including the P1 "PM =
Down" issue in Delorean. And we should also figure out how to use what =
you learned yesterday as part of our Zonda planning.</SPAN></FONT></DIV>=0A=
<DIV dir=3Dltr align=3Dleft><FONT face=3DArial color=3D#0000ff =
size=3D2><SPAN class=3D554034616-18042007></SPAN></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr align=3Dleft><FONT face=3DArial color=3D#0000ff =
size=3D2><SPAN class=3D554034616-18042007>jay</SPAN></FONT></DIV>=0A=
<DIV dir=3Dltr align=3Dleft><FONT face=3DArial color=3D#0000ff =
size=3D2><SPAN class=3D554034616-18042007></SPAN></FONT>&nbsp;</DIV><BR>=0A=
<DIV class=3DOutlookMessageHeader lang=3Den-us dir=3Dltr align=3Dleft>=0A=
<HR tabIndex=3D-1>=0A=
<FONT face=3DTahoma size=3D2><B>From:</B> Charissa Willard =
<BR><B>Sent:</B> Wednesday, April 18, 2007 9:41 AM<BR><B>To:</B> Jay =
Michlin<BR><B>Cc:</B> dl-Software; Eric Crutchlow<BR><B>Subject:</B> RE: =
Installation Issues<BR></FONT><BR></DIV>=0A=
<DIV></DIV>=0A=
<DIV class=3DSection1>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt">Jay,</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt">We did not run into any major issues during =
yesterday&#8217;s installation of a 2 node cluster at France Telecom in =
SSF. Moreover, we may have identified an initial configuration issue =
that could be causing most of the problems the SEs are encountering in =
the field. The assumption was being made that once PM was started, it is =
okay to go ahead and start configuring virtual servers and volumes. It =
isn&#8217;t enough to just make sure PM started. Configuration should =
only continue after PM successfully starts all daemons on the filer. =
During yesterday&#8217;s install, I was able to log in as root and =
monitor all processes as they were coming up. Once the last process =
(currently sendmail) was started, only then did we continue =
configuration. There are a few other issues we need to look at, but I =
think we need to address this one and not let configuration be done =
until all daemons are started. </SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt">-Charissa</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3D"Courier New" size=3D1><SPAN =
style=3D"FONT-SIZE: 9pt; FONT-FAMILY: 'Courier =
New'"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3D"Courier New" size=3D1><SPAN =
style=3D"FONT-SIZE: 9pt; FONT-FAMILY: 'Courier =
New'"></SPAN></FONT>&nbsp;</P>=0A=
<DIV>=0A=
<DIV class=3DMsoNormal style=3D"TEXT-ALIGN: center" align=3Dcenter><FONT =
face=3D"Times New Roman" size=3D3><SPAN style=3D"FONT-SIZE: 12pt">=0A=
<HR tabIndex=3D-1 align=3Dcenter width=3D"100%" SIZE=3D2>=0A=
</SPAN></FONT></DIV>=0A=
<P class=3DMsoNormal><B><FONT face=3DTahoma size=3D2><SPAN =
style=3D"FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: =
Tahoma">From:</SPAN></FONT></B><FONT face=3DTahoma size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"> Jay Michlin <BR><B><SPAN =
style=3D"FONT-WEIGHT: bold">Sent:</SPAN></B> Tuesday, April 17, 2007 =
11:36 AM<BR><B><SPAN style=3D"FONT-WEIGHT: bold">To:</SPAN></B> =
dl-Software<BR><B><SPAN style=3D"FONT-WEIGHT: bold">Cc:</SPAN></B> Eric =
Crutchlow<BR><B><SPAN style=3D"FONT-WEIGHT: bold">Subject:</SPAN></B> =
FW: Installation Issues</SPAN></FONT></P></DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial color=3Dblue size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">Hello =
all,</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial color=3Dblue size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">The email =
below is background reading about some of the problems our Porfessional =
Services team is seeing during installs and upgrades at customer sites. =
Most of that is invisible to us most of the time. And in the same way, =
most of our challenges in Software Development are invisible to field =
people most of the time. </SPAN></FONT></P>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3DArial color=3Dblue size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">We'll all =
benefit by understanding better what life is like in other parts of the =
company, so I offer the email below for your interest. There are no =
assignments or action items in it. Just information worth =
seeing.</SPAN></FONT></P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3DArial color=3Dblue size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: =
Arial">jay</SPAN></FONT></P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P></DIV>=0A=
<DIV class=3DMsoNormal style=3D"TEXT-ALIGN: center" align=3Dcenter><FONT =
face=3D"Times New Roman" size=3D3><SPAN style=3D"FONT-SIZE: 12pt">=0A=
<HR tabIndex=3D-1 align=3Dcenter width=3D"100%" SIZE=3D2>=0A=
</SPAN></FONT></DIV>=0A=
<P class=3DMsoNormal style=3D"MARGIN-BOTTOM: 12pt"><B><FONT =
face=3DTahoma size=3D2><SPAN style=3D"FONT-WEIGHT: bold; FONT-SIZE: =
10pt; FONT-FAMILY: Tahoma">From:</SPAN></FONT></B><FONT face=3DTahoma =
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"> Eric =
Crutchlow <BR><B><SPAN style=3D"FONT-WEIGHT: bold">Sent:</SPAN></B> =
Tuesday, April 17, 2007 8:57 AM<BR><B><SPAN style=3D"FONT-WEIGHT: =
bold">To:</SPAN></B> Jay Michlin; Paul Hammer; Sandrine Boulanger; Ed =
Kwan<BR><B><SPAN style=3D"FONT-WEIGHT: bold">Cc:</SPAN></B> Caeli =
Collins<BR><B><SPAN style=3D"FONT-WEIGHT: bold">Subject:</SPAN></B> RE: =
Installation Issues</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">Jay,</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">I appreciate your thoughts =
on the matter, and obviously I have not made my case. Please allow me to =
do so now.</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">Prior to Sudheesh&#8217;s =
meeting, the field was encountering the problems we have outline below =
(the installation issues list). From that meeting and the subsequent =
meeting with your group, we have made strides to educate and focus our =
efforts. The investment of time has produced the installations issues =
list and help get several serious bugs addressed. I think we would all =
agree that the time spent in these meetings were productive and =
needed.</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">But lets review were the =
fields&#8217; time has been spent. Two recent examples will illustrate =
this well; BAE and Fandango. </SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">One year ago, we attempted =
to demo to BAE and the system failed. We lost a sale (I have not been =
given the details, but if it would be helpful, please let me know). A =
year later, BAE lets us do another demo. Scott Moyer (our SE) went to =
BAE and attempted to install a two node cluster. He hits the PM on =
initial config error and attempts to configure the system anyway. He =
basically goes through four flash cards and now has no way to continue =
the install. Bill Duffy and I are at a customer&#8217;s site in San =
Diego performing an install. Bill gets called away to help Scott. =
Eventually, he cuts short his stay with me to immediately fly to =
Philadelphia and help. The customer is wondering what is going on and is =
getting ready to throw us out. Scott has wasted 8 hours. Bill now spends =
two days to get everything working as well as make sure the customer =
understands that the Bobcats are reliable and will work for them. Now =
ONStor has invested 24 hours on a possible sale.</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">Luckily, BAE is impressed =
with Bill&#8217;s &#8216;performance&#8217;. As fate would have it, =
BAE&#8217;s current system fail shortly thereafter and they decide to =
purchase the systems ASAP. Bill helped with the migration and so far no =
issues. </SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">A few weeks ago I =
installed Fandango with a two node cluster and no issues. Last Thursday =
they contacted me and said that they changed their network addresses and =
would like to update there Bobcat&#8217;s. The reaction to the fact that =
the only way they could do this was by resetting the boxes was not well =
received. But I assured them that since they haven&#8217;t gone into =
production, this would not be a major problem and would only take 1.5 =
hours. I first broke the cluster, upgraded to 2.2.2.2, and then =
performed the sys config reset. I ran into the PM problem that Scott hit =
at BAE. Rebooting didn&#8217;t work and I executed a second sys config =
reset which did. 1.5 turned into 4 hours. We left the systems that day =
setup with no vservers except for the management ones. Later that night =
at 2am, there was a cpu crash (see issue #5). Now Fandango is waiting on =
us to explain this before going live. </SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">And so with the two =
meetings we have had, we have created the list and are addressing the =
issues. As outlined below, the future releases will do a better job. But =
Delorean is a month away and ONStor must address these issues in the =
current release. And we are. Again, these two short meetings have helped =
us make strides.</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">The GUI has always been an =
issue and one that I was working on with Charissa prior to the meetings. =
When the field reported issues with PM or the GUI, the response has been =
they were not reproducible. As such the field became very frustrated and =
has not filed defects. This is unacceptable. Charissa has been willing =
to work with me on this and is making progress to resolve it. She is a =
great example of your suggestion of ongoing engagements. =
</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">But until the two =
meetings, this was the only ongoing engagement we had.</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">We have not directly =
addressed the fields&#8217; view of the support organization (CS, QA, =
Dev). We need to change that. The meetings were a start and well worth =
the time.</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">The field has also been =
told many times that items are WAD. My recent experience with Bill =
(defect 18502) illustrates this point. WAD is being touted too many =
times. This contributes to the &#8216;mythology&#8217; of how our =
systems work and serves no purpose. As I stated before, if something is =
WAD, it must be documented and the documentation must be easily =
accessible to a user. In my opinion, this is either in the SAG or GUI =
documentation. Management needs to address this and communicate it to =
the team. There should not be (cannot be) confusion on what WAD is for =
any particular component of our systems. I do not underestimate the =
challenge of this nor am I proposing the meetings I am requesting =
directly address this.</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">And so I put it to the =
team; by having two meeting to create the installation issues list we =
have helped focus future development efforts as outlined by Jay. We are =
also addressing the issues in the current release (no resolutions as of =
yet, but as Jay has stated, everyone is committed to fixing them). Prior =
to the two meetings, our method for dealing with this was ongoing =
engagements. I believe this has not worked and we need to change. I am =
only asking for one meeting for one-half hour, every two weeks (my =
apologizes Jay if I mis-communicated this as a weekly meeting). I will =
create an agenda and keep minutes to ensure the meetings are organized =
and effective. Our goal will be to address the issues and improve =
teamwork. More importantly, I propose that if we can incorporate these =
effort into our other meetings and processes, we no longer need this =
one.</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"></SPAN></FONT>&nbsp;</P>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">As I have illustrated with =
BAE and Fandango, we are already spending the time and have =
lost/jeopardized sales because of it. Is meeting once every two weeks =
not worth the investment? </SPAN></FONT></P>=0A=
<DIV>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">Regards,</SPAN></FONT> </P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">Eric Crutchlow</SPAN></FONT> <BR><FONT face=3DArial =
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">Professional Services Manager</SPAN></FONT> <BR><FONT =
face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">t: 408-376-3113</SPAN></FONT> <BR><FONT face=3DArial =
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">f: =
408-963-2409</SPAN></FONT> <BR><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">m: =
408-596-1155</SPAN></FONT> </P></DIV>=0A=
<DIV>=0A=
<DIV class=3DMsoNormal style=3D"TEXT-ALIGN: center" align=3Dcenter><FONT =
face=3D"Times New Roman" size=3D3><SPAN style=3D"FONT-SIZE: 12pt">=0A=
<HR tabIndex=3D-1 align=3Dcenter width=3D"100%" SIZE=3D2>=0A=
</SPAN></FONT></DIV></DIV>=0A=
<DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal style=3D"MARGIN-BOTTOM: 12pt"><B><FONT =
face=3DTahoma size=3D2><SPAN style=3D"FONT-WEIGHT: bold; FONT-SIZE: =
10pt; FONT-FAMILY: Tahoma">From:</SPAN></FONT></B><FONT face=3DTahoma =
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"> Eric =
Crutchlow<BR><B><SPAN style=3D"FONT-WEIGHT: bold">Sent:</SPAN></B> Mon =
4/16/2007 3:53 PM<BR><B><SPAN style=3D"FONT-WEIGHT: bold">To:</SPAN></B> =
Paul Hammer; Sandrine Boulanger; Ed Kwan; Jay Michlin<BR><B><SPAN =
style=3D"FONT-WEIGHT: bold">Subject:</SPAN></B> Installation =
Issues</SPAN></FONT></P></DIV>=0A=
<DIV>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">When: Occurs every 2 weeks on Thursday effective =
4/19/2007 from 1:00 PM to 1:30 PM (GMT-08:00) Pacific Time (US &amp; =
Canada).</SPAN></FONT></P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">Where: TBD</SPAN></FONT> </P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">*~*~*~*~*~*~*~*~*~*</SPAN></FONT> </P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">I'm following up with the installation issues =
meeting we had a few weeks ago to resolve the issues PS and the SE's are =
seeing in the field. The meeting will be every two weeks and I will keep =
minutes with action items. The goals are to resolve the current list =
(see below) and put processes in place to deal with future =
installation/eval issues as part of normal escalations. Once these two =
goals are reached, we can determine if there is any further need for =
additional meetings.</SPAN></FONT></P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">The following is a list of issues we currently are =
addressing:</SPAN></FONT> </P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">1) Prior to init config on a new nas, get errors. =
Reboot off of 2nd flash and no problems.</SPAN></FONT> </P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">2) Do init config 1st time. After reboot, the nas =
generates pm errors. If you reboot, you get init config, but this time =
the nas remembers the default route you entered the first time. 2nd =
reboot and everything is fine. We also see this with config =
resets.</SPAN></FONT></P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">3) Upgrades from 1.3.3.x to 2.x seems to work fine, =
but when you try and move the vs from the 1.3.3.x server to the 2.x, =
won&#8217;t work. A cluster show cluster on the 2.x nas reports the =
cluster name as &#8216;na&#8217;.&nbsp; You kill ncmd on the 1.3.3.x =
nas&nbsp; and now cluster name is right and you can move =
vs.</SPAN></FONT></P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">4) System copy all. Do a compare and get errors. If =
you reboot the nas, perform another copy all, compare works. We wonder =
if flash has space issues.</SPAN></FONT></P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">5) New install of a two node cluster. One of the =
nodes will reboot with a cpu crash. Just happened at an install Bill did =
yesterday.</SPAN></FONT></P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">6) An upgrade takes 15</SPAN></FONT> <FONT =
face=3DTahoma size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Tahoma">&#8211;</SPAN></FONT><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"> 20 minutes. Compare =
another 10</SPAN></FONT> <FONT face=3DTahoma size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Tahoma">&#8211;</SPAN></FONT><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"> 15. This takes a lot of =
time, especially if you get an error and have to start over. You can =
waste 1</SPAN></FONT> <FONT face=3DTahoma size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Tahoma">&#8211;</SPAN></FONT><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"> 1.5 hours this =
way.</SPAN></FONT></P></DIV></DIV></DIV></DIV></BODY></HTML>
------_=_NextPart_001_01C781DC.B4623F95--
