X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7A8B9.E843F257@onstor-exch02.onstor.net>; Wed, 6 Jun 2007 21:11:22 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7A8B9.E843F257"
Content-class: urn:content-classes:message
Subject: FW: AT&T release
Date: Wed, 6 Jun 2007 21:11:22 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E040F06A0@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: AT&T release
Thread-Index: Aceok6E5jBGJSj54RHyYOoADWHNLIQAEsa0QAATJpQA=
From: "Brian DeForest" <brian.deforest@onstor.com>
To: "Jay Michlin" <jay.michlin@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>,
	"Larry Scheer" <larry.scheer@onstor.com>,
	"Tim Gardner" <tim.gardner@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7A8B9.E843F257
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

FYI - ATT Delorean installation detailed report.   It was first reported
that the installation was a total disaster, but subsequently the problem
turned out to be a bad flash.  You can read the details below.   I
subsequently heard that the customer was *not* present during the
install, so this worked out well in the end.  =20

=20

=20

________________________________

From: Doug Cook=20
Sent: Wednesday, June 06, 2007 4:37 PM
To: Caeli Collins; Eric Barrett; Sandrine Boulanger; Sudheesh Nair
Cc: Marc Tanguay; Jerry Lopatin; Tom Gallivan; Chris Vandever; Carlos
Mora; Bill Duffy; Paul Hammer
Subject: Re: AT&T release

=20

That is correct.  As Eric mentioned, had we retried this procedure with
a different flash we would have encountered the expected results
Sandrine mentions.  But, we just didn't know that 3.0 has the
intelligence to copy the necessary information from the 2.x flash.

I will be sending the bad 3.0 flash back to corporate as soon as
possible.  Is there anyone in particular to whom I should send it?

DC

________________________________

From: "Caeli Collins" <caeli.collins@onstor.com>
Date: Wed, 6 Jun 2007 16:32:46 -0700
To: "Doug Cook" <doug.cook@onstor.com>, "Eric Barrett"
<eric.barrett@onstor.com>, "Sandrine Boulanger"
<sandrine.boulanger@onstor.com>, "Sudheesh Nair"
<sudheesh.nair@onstor.com>
Cc: "Marc Tanguay" <marc.tanguay@onstor.com>, "Jerry Lopatin"
<jerry.lopatin@onstor.com>, "Tom Gallivan" <tom.gallivan@onstor.com>,
"Chris Vandever" <chris.vandever@onstor.com>, "Carlos Mora"
<carlos.mora@onstor.com>, "Bill Duffy" <bill.duffy@onstor.com>, "Paul
Hammer" <paul.hammer@onstor.com>
Subject: RE: AT&T release

Was that the first flash card?  The one we believe had a format issue?

Caeli=20

=20

________________________________

From: Doug Cook=20
Sent: Wednesday, June 06, 2007 4:31 PM
To: Eric Barrett; Sandrine Boulanger; Sudheesh Nair
Cc: Marc Tanguay; Jerry Lopatin; Caeli Collins; Tom Gallivan; Chris
Vandever; Carlos Mora; Bill Duffy; Paul Hammer
Subject: Re: AT&T release

I'd like to also add that all on the call bantered around the thought
that maybe the 3.0 flash had the intelligence to automagically pull the
configuration from the 2.x flash.  Hey, we tried it and followed the
release notes' steps:  we swapped the flash and rebooted the system.
After about 10 minutes and the system wasn't back running, we gave up on
that particular flash and rebooted the system back to 2.2.1.0.

________________________________

From: "Eric Barrett" <eric.barrett@onstor.com>
Date: Wed, 6 Jun 2007 16:19:32 -0700
To: "Eric Barrett" <eric.barrett@onstor.com>, "Sandrine Boulanger"
<sandrine.boulanger@onstor.com>, "Doug Cook" <doug.cook@onstor.com>,
"Sudheesh Nair" <sudheesh.nair@onstor.com>
Cc: "Marc Tanguay" <marc.tanguay@onstor.com>, "Jerry Lopatin"
<jerry.lopatin@onstor.com>, "Caeli Collins" <caeli.collins@onstor.com>,
"Tom Gallivan" <tom.gallivan@onstor.com>, "Chris Vandever"
<chris.vandever@onstor.com>, "Carlos Mora" <carlos.mora@onstor.com>,
"Bill Duffy" <bill.duffy@onstor.com>, "Paul Hammer"
<paul.hammer@onstor.com>
Subject: RE: AT&T release

Oh, let me also add that, as implemented, the solution of automatically
copying the configuration from the old flash was a good choice, and I
can only describe it as elegant.  If only we'd known about it!

________________________________

From: Eric Barrett=20
Sent: Wednesday, June 06, 2007 4:05 PM
To: Sandrine Boulanger; Doug Cook; Sudheesh Nair
Cc: Marc Tanguay; Jerry Lopatin; Caeli Collins; Tom Gallivan; Chris
Vandever; Carlos Mora; Bill Duffy; Paul Hammer
Subject: RE: AT&T release

We think there were two problems.

1) The first Delorean compact flash that we tried seemed to be invalid.
It appeared to have been formatted incorrectly, or not formatted at all.

2) All participants in the call were unaware that the Delorean compact
flash would copy the configuration from the old flash on first boot.
While the release notes had the correct procedure, it was not
communicated to anybody in support (including Dennis, who wrote the
release notes) or even members of Engineering who were on the call, that
this would take place.  Therefore, after replacing the first badly
formatted flash card, we didn't think to try rebooting onto the new card
before copying the configuration.  As Caeli said, we're "creatures of
habit," and when we realized there was no explicit config copy in the
steps we assumed the steps were wrong and sought to implement one!

#1 was addressed at the time of the call by using an alternate card.

#2 will be addressed shortly by an update to the release notes saying
explicitly that the configuration copy takes place.

________________________________

From: Sandrine Boulanger=20
Sent: Wednesday, June 06, 2007 3:47 PM
To: Doug Cook; Sudheesh Nair
Cc: Marc Tanguay; Jerry Lopatin; Caeli Collins; Tom Gallivan; Chris
Vandever; Carlos Mora; Eric Barrett; Bill Duffy; Paul Hammer
Subject: RE: AT&T release

I've attached the release notes with the upgrade procedure we followed
to upgrade QA clusters from 2.2. and below to 3.0 multiple times and
Mightydog 2 times.
The "system config copy" step is intentionally removed from the
procedure because it is NOT needed.
We encountered the problem the first time we tested: given the flash
layout change, system config copy was no longer working. A fix was
implemented: when building the flash, setup a flag that will trigger the
automatic copy of the config files once the 3.0 flash boots. That's the
way it has been tested. That's how the master falsh was built.
After making the master flash yesterday, a copy of the master was tested
and we verified that the automatic copy was happening.
=20
I don't know what was the problem with the first flash that was tried,
and we should ship it back here so we can look at it and try to find
what happened.
=20
About steps 4 and 5, 3.0 flashes should both have the same layout, so
system config copy should have worked there. I'm curious to see the
output of "mount" on each one to check the partitions.
=20

________________________________

From: Doug Cook=20
Sent: Wednesday, June 06, 2007 2:53 PM
To: Sudheesh Nair
Cc: Marc Tanguay; Audra Browning; Jerry Lopatin; Caeli Collins; Tom
Gallivan; Chris Vandever; Carlos Mora; Eric Barrett; Bill Duffy; Paul
Hammer; Sandrine Boulanger
Subject: Re: AT&T release

Hey All,

Well, as a team we pulled it off.  The at&t systems were all upgraded to
3.0 today, and the customers are happy.  Gene, the developer who will
most use the new features, had two very specific compliments on our
systems.

1.	"These  systems are  extremely easy to add storage to"    =20
2.	The  "delete  feature" is exactly what he was looking for, and
he's excited to  receive  it and see it work.=20


The upgrade process was not without issues, though.  Had it not been for
Chris, Carlos, and Eric, I would have had to spend lots of time working
my way through getting the systems upgraded.  Many thanks to you three.
Issues we encountered were with a) incorrect release notes, and b) no
method to transfer the cluster configuration to the new flash.

Upgrading the systems, we thought, should have been as simple as the
following steps:

1.	Replace  the 2.x  secondary flash with a new 3.0 flash    =20
2.	Copy  the system  configuration information from the primary
flash to the new 3.0  flash  using the "system config copy" command    =20
3.	Reboot  onto the 3.0  flash    =20
4.	Swap  the remaining  2.x flash with a second 3.0 flash    =20
5.	Copy  the system  configuration information to the new secondary
3.0  flash =20


However, we found that after swapping the secondary flashes and
attempting to copy the configuration information to the 3.0 flash, there
was no way to do this.  The "system config copy" command depends on the
target flash being partitioned a certain way - a way that in 3.0 is no
longer in use.  Carlos and Eric quickly put together a script that
allowed us to hack our way through getting the configuration properly
transferred to the new flash.  Once this was done, we were able to
continue with step 3 above.  About four hours after we started, all
three systems were upgraded and running.

Thanks again to everyone who helped make this possible.  If you have
questions or want to drill deeper into today's events, call or email me.

Doug

________________________________

From: "Sudheesh Nair" <sudheesh.nair@onstor.com>
Date: Mon, 4 Jun 2007 09:32:20 -0700
To: "Doug Cook" <doug.cook@onstor.com>
Cc: "Marc Tanguay" <marc.tanguay@onstor.com>, "Audra Browning"
<audra.browning@onstor.com>, "Jerry Lopatin" <jerry.lopatin@onstor.com>,
"Caeli Collins" <caeli.collins@onstor.com>, "Tom Gallivan"
<tom.gallivan@onstor.com>
Subject: AT&T release

Doug

We are ready for AT&T upgrade.

1. Eng will send the flash overnight today
2. Chris from eng will be helping you from HQ over Webex
3. You will have to be onsite running the upgrade and webex.
4. Pls open a case just to have Caeli's team plugged in.

Caeli, this will be a good oppt to have someone in your team view. We
can invite your team in the webex.

Doug, "smooth" should be the theme of the day

Good luck!!


Regards,
Sudheesh
sudheesh.nair@onstor.com
408-802-2856
--------------------------
Sent from my BlackBerry Wireless Handheld=20


------_=_NextPart_001_01C7A8B9.E843F257
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<title>Re: AT&amp;T release</title>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
	{font-family:Verdana;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:navy;
	text-decoration:underline;}
p
	{mso-margin-top-alt:auto;
	margin-right:0in;
	mso-margin-bottom-alt:auto;
	margin-left:0in;
	font-size:12.0pt;
	font-family:"Times New Roman";}
span.EmailStyle18
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle19
	{mso-style-type:personal-reply;
	font-family:Arial;
	color:navy;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
	{page:Section1;}
 /* List Definitions */
 @list l0
	{mso-list-id:184681783;
	mso-list-template-ids:-1874295746;}
@list l0:level1
	{mso-level-tab-stop:.5in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l0:level2
	{mso-level-tab-stop:1.0in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l0:level3
	{mso-level-tab-stop:1.5in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l0:level4
	{mso-level-tab-stop:2.0in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l0:level5
	{mso-level-tab-stop:2.5in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l0:level6
	{mso-level-tab-stop:3.0in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l0:level7
	{mso-level-tab-stop:3.5in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l0:level8
	{mso-level-tab-stop:4.0in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l0:level9
	{mso-level-tab-stop:4.5in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l1
	{mso-list-id:693968847;
	mso-list-template-ids:446205394;}
@list l2
	{mso-list-id:732703115;
	mso-list-template-ids:-362115282;}
@list l3
	{mso-list-id:1225723182;
	mso-list-template-ids:-1191133500;}
@list l3:level1
	{mso-level-tab-stop:.5in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l3:level2
	{mso-level-tab-stop:1.0in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l3:level3
	{mso-level-tab-stop:1.5in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l3:level4
	{mso-level-tab-stop:2.0in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l3:level5
	{mso-level-tab-stop:2.5in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l3:level6
	{mso-level-tab-stop:3.0in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l3:level7
	{mso-level-tab-stop:3.5in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l3:level8
	{mso-level-tab-stop:4.0in;
	mso-level-number-position:left;
	text-indent:-.25in;}
@list l3:level9
	{mso-level-tab-stop:4.5in;
	mso-level-number-position:left;
	text-indent:-.25in;}
ol
	{margin-bottom:0in;}
ul
	{margin-bottom:0in;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-US link=3Dblue vlink=3Dnavy>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>FYI &#8211; ATT Delorean =
installation
detailed report.&nbsp;&nbsp; It was first reported that the installation =
was a
total disaster, but subsequently the problem turned out to be a bad
flash.&nbsp; You can read the details below.&nbsp;&nbsp; I subsequently =
heard
that the customer was *<b><span =
style=3D'font-weight:bold'>not</span></b>*
present during the install, so this worked out well in the =
end.&nbsp;&nbsp; <o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">Doug Cook</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Wednesday, June 06, =
2007
4:37 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Caeli
 Collins</st1:PersonName>; Eric Barrett; <st1:PersonName =
w:st=3D"on">Sandrine
 Boulanger</st1:PersonName>; <st1:PersonName w:st=3D"on">Sudheesh =
Nair</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Marc Tanguay; Jerry =
Lopatin; <st1:PersonName
w:st=3D"on">Tom Gallivan</st1:PersonName>; Chris Vandever; =
<st1:PersonName w:st=3D"on">Carlos
 Mora</st1:PersonName>; Bill Duffy; <st1:PersonName w:st=3D"on">Paul =
Hammer</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> Re: AT&amp;T =
release</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><font size=3D1 =
face=3DVerdana><span
style=3D'font-size:9.0pt;font-family:Verdana'>That is correct. &nbsp;As =
Eric
mentioned, had we retried this procedure with a different flash we would =
have
encountered the expected results Sandrine mentions. &nbsp;But, we just
didn&#8217;t know that 3.0 has the intelligence to copy the necessary =
information
from the 2.x flash.<br>
<br>
I will be sending the bad 3.0 flash back to corporate as soon as =
possible.
&nbsp;Is there anyone in particular to whom I should send it?<br>
<br>
DC<o:p></o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D1
face=3DVerdana><span style=3D'font-size:9.0pt;font-family:Verdana'>

<hr size=3D3 width=3D"95%" align=3Dcenter>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D1 =
face=3DVerdana><span
style=3D'font-size:9.0pt;font-family:Verdana;font-weight:bold'>From: =
</span></font></b><font
size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;font-family:Verdana'>&quot;<st1:PersonName
w:st=3D"on">Caeli Collins</st1:PersonName>&quot; =
&lt;caeli.collins@onstor.com&gt;<br>
<b><span style=3D'font-weight:bold'>Date: </span></b>Wed, 6 Jun 2007 =
16:32:46
-0700<br>
<b><span style=3D'font-weight:bold'>To: </span></b>&quot;<st1:PersonName =
w:st=3D"on">Doug
 Cook</st1:PersonName>&quot; &lt;doug.cook@onstor.com&gt;, &quot;Eric
Barrett&quot; &lt;eric.barrett@onstor.com&gt;, &quot;<st1:PersonName =
w:st=3D"on">Sandrine
 Boulanger</st1:PersonName>&quot; &lt;sandrine.boulanger@onstor.com&gt;, =
&quot;<st1:PersonName
w:st=3D"on">Sudheesh Nair</st1:PersonName>&quot; =
&lt;sudheesh.nair@onstor.com&gt;<br>
<b><span style=3D'font-weight:bold'>Cc: </span></b>&quot;Marc =
Tanguay&quot;
&lt;marc.tanguay@onstor.com&gt;, &quot;Jerry Lopatin&quot;
&lt;jerry.lopatin@onstor.com&gt;, &quot;<st1:PersonName w:st=3D"on">Tom =
Gallivan</st1:PersonName>&quot;
&lt;tom.gallivan@onstor.com&gt;, &quot;Chris Vandever&quot;
&lt;chris.vandever@onstor.com&gt;, &quot;<st1:PersonName =
w:st=3D"on">Carlos Mora</st1:PersonName>&quot;
&lt;carlos.mora@onstor.com&gt;, &quot;Bill Duffy&quot;
&lt;bill.duffy@onstor.com&gt;, &quot;<st1:PersonName w:st=3D"on">Paul =
Hammer</st1:PersonName>&quot;
&lt;paul.hammer@onstor.com&gt;<br>
<b><span style=3D'font-weight:bold'>Subject: </span></b>RE: AT&amp;T =
release<br>
<br>
</span></font><font size=3D1 color=3Dblue face=3DArial><span =
style=3D'font-size:9.0pt;
font-family:Arial;color:blue'>Was that the first flash card? &nbsp;The =
one we
believe had a format issue?<br>
</span></font><font size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;
font-family:Verdana'><br>
</span></font><font size=3D1 face=3DArial><span =
style=3D'font-size:9.0pt;font-family:
Arial'>Caeli</span></font><font size=3D1 face=3DVerdana><span =
style=3D'font-size:
9.0pt;font-family:Verdana'> <br>
<br>
&nbsp;<o:p></o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D1
face=3DVerdana><span style=3D'font-size:9.0pt;font-family:Verdana'>

<hr size=3D3 width=3D"100%" align=3Dcenter>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D1 =
face=3DTahoma><span
style=3D'font-size:9.0pt;font-family:Tahoma;font-weight:bold'>From:</span=
></font></b><font
size=3D1 face=3DTahoma><span =
style=3D'font-size:9.0pt;font-family:Tahoma'> <st1:PersonName
w:st=3D"on">Doug Cook</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Wednesday, June 06, =
2007
4:31 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Eric Barrett; =
<st1:PersonName
w:st=3D"on">Sandrine Boulanger</st1:PersonName>; <st1:PersonName =
w:st=3D"on">Sudheesh
 Nair</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Marc Tanguay; Jerry =
Lopatin; <st1:PersonName
w:st=3D"on">Caeli Collins</st1:PersonName>; <st1:PersonName =
w:st=3D"on">Tom
 Gallivan</st1:PersonName>; Chris Vandever; <st1:PersonName =
w:st=3D"on">Carlos Mora</st1:PersonName>;
Bill Duffy; <st1:PersonName w:st=3D"on">Paul Hammer</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> Re: AT&amp;T =
release<br>
</span></font><font size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;
font-family:Verdana'><br>
I&#8217;d like to also add that all on the call bantered around the =
thought
that maybe the 3.0 flash had the intelligence to automagically pull the
configuration from the 2.x flash. &nbsp;Hey, we tried it and followed =
the
release notes&#8217; steps: &nbsp;we swapped the flash and rebooted the =
system.
&nbsp;After about 10 minutes and the system wasn&#8217;t back running, =
we gave
up on that particular flash and rebooted the system back to =
2.2.1.0.<o:p></o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D1
face=3DVerdana><span style=3D'font-size:9.0pt;font-family:Verdana'>

<hr size=3D3 width=3D"95%" align=3Dcenter>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D1 =
face=3DVerdana><span
style=3D'font-size:9.0pt;font-family:Verdana;font-weight:bold'>From: =
</span></font></b><font
size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;font-family:Verdana'>&quot;Eric
Barrett&quot; &lt;eric.barrett@onstor.com&gt;<br>
<b><span style=3D'font-weight:bold'>Date: </span></b>Wed, 6 Jun 2007 =
16:19:32
-0700<br>
<b><span style=3D'font-weight:bold'>To: </span></b>&quot;Eric =
Barrett&quot;
&lt;eric.barrett@onstor.com&gt;, &quot;<st1:PersonName =
w:st=3D"on">Sandrine
 Boulanger</st1:PersonName>&quot; &lt;sandrine.boulanger@onstor.com&gt;, =
&quot;<st1:PersonName
w:st=3D"on">Doug Cook</st1:PersonName>&quot; =
&lt;doug.cook@onstor.com&gt;, &quot;<st1:PersonName
w:st=3D"on">Sudheesh Nair</st1:PersonName>&quot; =
&lt;sudheesh.nair@onstor.com&gt;<br>
<b><span style=3D'font-weight:bold'>Cc: </span></b>&quot;Marc =
Tanguay&quot;
&lt;marc.tanguay@onstor.com&gt;, &quot;Jerry Lopatin&quot;
&lt;jerry.lopatin@onstor.com&gt;, &quot;<st1:PersonName =
w:st=3D"on">Caeli Collins</st1:PersonName>&quot;
&lt;caeli.collins@onstor.com&gt;, &quot;<st1:PersonName w:st=3D"on">Tom =
Gallivan</st1:PersonName>&quot;
&lt;tom.gallivan@onstor.com&gt;, &quot;Chris Vandever&quot;
&lt;chris.vandever@onstor.com&gt;, &quot;<st1:PersonName =
w:st=3D"on">Carlos Mora</st1:PersonName>&quot;
&lt;carlos.mora@onstor.com&gt;, &quot;Bill Duffy&quot; =
&lt;bill.duffy@onstor.com&gt;,
&quot;<st1:PersonName w:st=3D"on">Paul Hammer</st1:PersonName>&quot;
&lt;paul.hammer@onstor.com&gt;<br>
<b><span style=3D'font-weight:bold'>Subject: </span></b>RE: AT&amp;T =
release<br>
<br>
</span></font><font size=3D1 face=3DArial><span =
style=3D'font-size:9.0pt;font-family:
Arial'>Oh, let me also add that, as implemented, the solution of =
automatically
copying the configuration from the old flash was a good choice, and I =
can only
describe it as elegant. &nbsp;If only we'd known about =
it!</span></font><font
size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;font-family:Verdana'><o:p></o:p></span></font></=
p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D1
face=3DVerdana><span style=3D'font-size:9.0pt;font-family:Verdana'>

<hr size=3D3 width=3D"100%" align=3Dcenter>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D1 =
face=3DTahoma><span
style=3D'font-size:9.0pt;font-family:Tahoma;font-weight:bold'>From:</span=
></font></b><font
size=3D1 face=3DTahoma><span =
style=3D'font-size:9.0pt;font-family:Tahoma'> Eric
Barrett <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Wednesday, June 06, =
2007
4:05 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Sandrine
 Boulanger</st1:PersonName>; <st1:PersonName w:st=3D"on">Doug =
Cook</st1:PersonName>;
<st1:PersonName w:st=3D"on">Sudheesh Nair</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Marc Tanguay; Jerry =
Lopatin; <st1:PersonName
w:st=3D"on">Caeli Collins</st1:PersonName>; <st1:PersonName =
w:st=3D"on">Tom
 Gallivan</st1:PersonName>; Chris Vandever; <st1:PersonName =
w:st=3D"on">Carlos
 Mora</st1:PersonName>; Bill Duffy; <st1:PersonName w:st=3D"on">Paul =
Hammer</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: AT&amp;T =
release<br>
</span></font><font size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;
font-family:Verdana'><br>
</span></font><font size=3D1 face=3DArial><span =
style=3D'font-size:9.0pt;font-family:
Arial'>We think there were two problems.<br>
</span></font><font size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;
font-family:Verdana'><br>
</span></font><font size=3D1 face=3DArial><span =
style=3D'font-size:9.0pt;font-family:
Arial'>1) The first Delorean compact flash that we tried seemed to be =
invalid.
&nbsp;It appeared to have been formatted incorrectly, or not formatted =
at all.<br>
</span></font><font size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;
font-family:Verdana'><br>
</span></font><font size=3D1 face=3DArial><span =
style=3D'font-size:9.0pt;font-family:
Arial'>2) All participants in the call were unaware that the Delorean =
compact
flash would <b><span style=3D'font-weight:bold'>copy the configuration =
from the
old flash on first boot</span></b>. &nbsp;While the release notes had =
the
correct procedure, it was not communicated to anybody in support =
(including
Dennis, who wrote the release notes) or even members of Engineering who =
were on
the call, that this would take place. &nbsp;Therefore, after replacing =
the
first badly formatted flash card, we didn't think to try rebooting onto =
the new
card before copying the configuration. &nbsp;As Caeli said, we're =
&quot;creatures
of habit,&quot; and when we realized there was no explicit config copy =
in the
steps we assumed the steps were wrong and sought to implement one!<br>
</span></font><font size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;
font-family:Verdana'><br>
</span></font><font size=3D1 face=3DArial><span =
style=3D'font-size:9.0pt;font-family:
Arial'>#1 was addressed at the time of the call by using an alternate =
card.<br>
</span></font><font size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;
font-family:Verdana'><br>
</span></font><font size=3D1 face=3DArial><span =
style=3D'font-size:9.0pt;font-family:
Arial'>#2 will be addressed shortly by an update to the release notes =
saying
explicitly that the configuration copy takes place.</span></font><font =
size=3D1
face=3DVerdana><span =
style=3D'font-size:9.0pt;font-family:Verdana'><o:p></o:p></span></font></=
p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D1
face=3DVerdana><span style=3D'font-size:9.0pt;font-family:Verdana'>

<hr size=3D3 width=3D"100%" align=3Dcenter>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D1 face=3DTahoma><span =
style=3D'font-size:9.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D1
face=3DTahoma><span style=3D'font-size:9.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">Sandrine Boulanger</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Wednesday, June 06, =
2007
3:47 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Doug
 Cook</st1:PersonName>; <st1:PersonName w:st=3D"on">Sudheesh =
Nair</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Marc Tanguay; Jerry =
Lopatin; <st1:PersonName
w:st=3D"on">Caeli Collins</st1:PersonName>; <st1:PersonName =
w:st=3D"on">Tom
 Gallivan</st1:PersonName>; Chris Vandever; <st1:PersonName =
w:st=3D"on">Carlos
 Mora</st1:PersonName>; Eric Barrett; Bill Duffy; <st1:PersonName =
w:st=3D"on">Paul
 Hammer</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: AT&amp;T =
release<br>
</span></font><font size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;
font-family:Verdana'><br>
</span></font><font size=3D2 color=3Dnavy face=3DVerdana><span =
style=3D'font-size:10.5pt;
font-family:Verdana;color:navy'>I&#8217;ve attached the release notes =
with the
upgrade procedure we followed to upgrade QA clusters from 2.2. and below =
to 3.0
multiple times and Mightydog 2 times.<br>
The &#8220;system config copy&#8221; step is intentionally removed from =
the
procedure because it is NOT needed.<br>
We encountered the problem the first time we tested: given the flash =
layout
change, system config copy was no longer working. A fix was implemented: =
when
building the flash, setup a flag that will trigger the automatic copy of =
the
config files once the 3.0 flash boots. That&#8217;s the way it has been =
tested.
That&#8217;s how the master falsh was built.<br>
After making the master flash yesterday, a copy of the master was tested =
and we
verified that the automatic copy was happening.<br>
&nbsp;<br>
I don&#8217;t know what was the problem with the first flash that was =
tried,
and we should ship it back here so we can look at it and try to find =
what
happened.<br>
&nbsp;<br>
About steps 4 and 5, 3.0 flashes should both have the same layout, so =
system
config copy should have worked there. I&#8217;m curious to see the =
output of
&#8220;mount&#8221; on each one to check the partitions.<br>
&nbsp;</span></font><o:p></o:p></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D4
face=3D"Times New Roman"><span style=3D'font-size:13.5pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter>

</span></font></div>

<p><b><font size=3D2 face=3DVerdana><span =
style=3D'font-size:10.5pt;font-family:Verdana;
font-weight:bold'>From:</span></font></b><font size=3D2 =
face=3DVerdana><span
style=3D'font-size:10.5pt;font-family:Verdana'> <st1:PersonName =
w:st=3D"on">Doug
 Cook</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Wednesday, June 06, =
2007
2:53 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Sudheesh
 Nair</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Marc Tanguay; Audra =
Browning;
Jerry Lopatin; <st1:PersonName w:st=3D"on">Caeli =
Collins</st1:PersonName>; <st1:PersonName
w:st=3D"on">Tom Gallivan</st1:PersonName>; Chris Vandever; =
<st1:PersonName w:st=3D"on">Carlos
 Mora</st1:PersonName>; Eric Barrett; Bill Duffy; <st1:PersonName =
w:st=3D"on">Paul
 Hammer</st1:PersonName>; <st1:PersonName w:st=3D"on">Sandrine =
Boulanger</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> Re: AT&amp;T =
release<br>
</span></font><font size=3D4><span style=3D'font-size:13.5pt'><br>
</span></font><font size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;
font-family:Verdana'>Hey All,<br>
<br>
Well, as a team we pulled it off. &nbsp;The at&amp;t systems were all =
upgraded
to 3.0 today, and the customers are happy. &nbsp;Gene, the developer who =
will
most use the new features, had two very specific compliments on our =
systems.</span></font><o:p></o:p></p>

<ol start=3D1 type=3D1>
 <li class=3DMsoNormal =
style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
     mso-list:l3 level1 lfo3'><font size=3D1 face=3DVerdana><span =
style=3D'font-size:
     9.0pt;font-family:Verdana'>&#8220;These &nbsp;systems are =
&nbsp;extremely
     easy to add storage to&#8221; &nbsp;&nbsp;&nbsp; =
</span></font><o:p></o:p></li>
 <li class=3DMsoNormal =
style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
     mso-list:l3 level1 lfo3'><font size=3D1 face=3DVerdana><span =
style=3D'font-size:
     9.0pt;font-family:Verdana'>The &nbsp;&#8220;delete =
&nbsp;feature&#8221; is
     exactly what he was looking for, and he&#8217;s excited to =
&nbsp;receive
     &nbsp;it and see it work. </span></font><o:p></o:p></li>
</ol>

<p class=3DMsoNormal><font size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;
font-family:Verdana'><br>
The upgrade process was not without issues, though. &nbsp;Had it not =
been for
Chris, Carlos, and Eric, I would have had to spend lots of time working =
my way
through getting the systems upgraded. &nbsp;Many thanks to you three.
&nbsp;Issues we encountered were with a) incorrect release notes, and b) =
no
method to transfer the cluster configuration to the new flash.<br>
<br>
Upgrading the systems, we thought, should have been as simple as the =
following
steps:</span></font><o:p></o:p></p>

<ol start=3D1 type=3D1>
 <li class=3DMsoNormal =
style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
     mso-list:l0 level1 lfo6'><font size=3D1 face=3DVerdana><span =
style=3D'font-size:
     9.0pt;font-family:Verdana'>Replace &nbsp;the 2.x &nbsp;secondary =
flash
     with a new 3.0 flash &nbsp;&nbsp;&nbsp; =
</span></font><o:p></o:p></li>
 <li class=3DMsoNormal =
style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
     mso-list:l0 level1 lfo6'><font size=3D1 face=3DVerdana><span =
style=3D'font-size:
     9.0pt;font-family:Verdana'>Copy &nbsp;the system =
&nbsp;configuration
     information from the primary flash to the new 3.0 &nbsp;flash =
&nbsp;using
     the &#8220;system config copy&#8221; command &nbsp;&nbsp;&nbsp; =
</span></font><o:p></o:p></li>
 <li class=3DMsoNormal =
style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
     mso-list:l0 level1 lfo6'><font size=3D1 face=3DVerdana><span =
style=3D'font-size:
     9.0pt;font-family:Verdana'>Reboot &nbsp;onto the 3.0 &nbsp;flash
     &nbsp;&nbsp;&nbsp; </span></font><o:p></o:p></li>
 <li class=3DMsoNormal =
style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
     mso-list:l0 level1 lfo6'><font size=3D1 face=3DVerdana><span =
style=3D'font-size:
     9.0pt;font-family:Verdana'>Swap &nbsp;the remaining &nbsp;2.x flash =
with a
     second 3.0 flash &nbsp;&nbsp;&nbsp; </span></font><o:p></o:p></li>
 <li class=3DMsoNormal =
style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
     mso-list:l0 level1 lfo6'><font size=3D1 face=3DVerdana><span =
style=3D'font-size:
     9.0pt;font-family:Verdana'>Copy &nbsp;the system =
&nbsp;configuration
     information to the new secondary 3.0 &nbsp;flash =
&nbsp;</span></font><o:p></o:p></li>
</ol>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><font size=3D1 =
face=3DVerdana><span
style=3D'font-size:9.0pt;font-family:Verdana'><br>
However, we found that after swapping the secondary flashes and =
attempting to
copy the configuration information to the 3.0 flash, there was no way to =
do
this. &nbsp;The &#8220;system config copy&#8221; command depends on the =
target
flash being partitioned a certain way &#8211; a way that in 3.0 is no =
longer in
use. &nbsp;Carlos and Eric quickly put together a script that allowed us =
to
hack our way through getting the configuration properly transferred to =
the new
flash. &nbsp;Once this was done, we were able to continue with step 3 =
above.
&nbsp;About four hours after we started, all three systems were upgraded =
and
running.<br>
<br>
Thanks again to everyone who helped make this possible. &nbsp;If you =
have
questions or want to drill deeper into today&#8217;s events, call or =
email me.<br>
<br>
Doug</span></font><o:p></o:p></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D1
face=3DVerdana><span style=3D'font-size:9.0pt;font-family:Verdana'>

<hr size=3D3 width=3D"95%" align=3Dcenter>

</span></font></div>

<p style=3D'margin-bottom:12.0pt'><b><font size=3D1 face=3DVerdana><span
style=3D'font-size:9.0pt;font-family:Verdana;font-weight:bold'>From: =
</span></font></b><font
size=3D1 face=3DVerdana><span =
style=3D'font-size:9.0pt;font-family:Verdana'>&quot;<st1:PersonName
w:st=3D"on">Sudheesh Nair</st1:PersonName>&quot; =
&lt;sudheesh.nair@onstor.com&gt;<br>
<b><span style=3D'font-weight:bold'>Date: </span></b>Mon, 4 Jun 2007 =
09:32:20
-0700<br>
<b><span style=3D'font-weight:bold'>To: </span></b>&quot;<st1:PersonName =
w:st=3D"on">Doug
 Cook</st1:PersonName>&quot; &lt;doug.cook@onstor.com&gt;<br>
<b><span style=3D'font-weight:bold'>Cc: </span></b>&quot;Marc =
Tanguay&quot;
&lt;marc.tanguay@onstor.com&gt;, &quot;Audra Browning&quot;
&lt;audra.browning@onstor.com&gt;, &quot;Jerry Lopatin&quot;
&lt;jerry.lopatin@onstor.com&gt;, &quot;<st1:PersonName =
w:st=3D"on">Caeli Collins</st1:PersonName>&quot;
&lt;caeli.collins@onstor.com&gt;, &quot;<st1:PersonName w:st=3D"on">Tom =
Gallivan</st1:PersonName>&quot;
&lt;tom.gallivan@onstor.com&gt;<br>
<b><span style=3D'font-weight:bold'>Subject: </span></b>AT&amp;T =
release<br>
<br>
Doug<br>
<br>
We are ready for AT&amp;T upgrade.<br>
<br>
1. Eng will send the flash overnight today<br>
2. Chris from eng will be helping you from HQ over Webex<br>
3. You will have to be onsite running the upgrade and webex.<br>
4. Pls open a case just to have Caeli's team plugged in.<br>
<br>
Caeli, this will be a good oppt to have someone in your team view. We =
can
invite your team in the webex.<br>
<br>
Doug, &quot;smooth&quot; should be the theme of the day<br>
<br>
Good luck!!<br>
<br>
<br>
Regards,<br>
Sudheesh<br>
sudheesh.nair@onstor.com<br>
408-802-2856<br>
--------------------------<br>
Sent from my BlackBerry Wireless Handheld </span></font><o:p></o:p></p>

</div>

</body>

</html>

------_=_NextPart_001_01C7A8B9.E843F257--
