X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C85EF0.0FD8C15A@onstor-exch02.onstor.net>; Thu, 24 Jan 2008 18:17:33 -0700
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
Subject: RE: "system upgrade" vs. script for Cougar
Date: Thu, 24 Jan 2008 18:17:31 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E07E5FA65@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E05C7411E@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: "system upgrade" vs. script for Cougar
Thread-Index: AcheGCModWzkBXy1QRSuLILN5GEsWgACT0zAACH5qCAADa2Y4AADDwlgAADr3bA=
References: <BB375AF679D4A34E9CA8DFA650E2B04E07E5FA2B@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E05C7411E@onstor-exch02.onstor.net>
From: "Dennis Arellano" <dennis.arellano@onstor.com>
To: "Sandrine Boulanger" <sandrine.boulanger@onstor.com>,
	"Rich LaReau" <rich.lareau@onstor.com>,
	"Larry Scheer" <larry.scheer@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>,
	"Eric Barrett" <eric.barrett@onstor.com>
Cc: "Vikas Saini" <vikas.saini@onstor.com>,
	"Sudheesh Nair" <sudheesh.nair@onstor.com>,
	"Tim Gardner" <tim.gardner@onstor.com>,
	"Andrew LeFebvre" <andrew.lefebvre@onstor.com>,
	"Caeli Collins" <caeli.collins@onstor.com>,
	"Paul Hammer" <paul.hammer@onstor.com>,
	"Brian Stark" <brian.stark@onstor.com>

Sounds like we are close to closure.=20

Rich, when you have all the details worked out, we need to sit down and
write up the new Upgrade Procedure.

Thanks, Dennis

-----Original Message-----
From: Sandrine Boulanger=20
Sent: Thursday, January 24, 2008 4:53 PM
To: Rich LaReau; Larry Scheer; Andy Sharp; Eric Barrett
Cc: Vikas Saini; Sudheesh Nair; Tim Gardner; Dennis Arellano; Andrew
LeFebvre; Caeli Collins; Paul Hammer; Brian Stark
Subject: RE: "system upgrade" vs. script for Cougar

There a bug that ins some cases, after a copy init, a system upgrade -s
does not work with the error "not compatible with this system" or
similar. I ran into it this morning. I had to manually mount / from
secondary flash and copy /version from primary to secondary to be able
to upgrade. This needs to be fixed if we want to use this procedure. A
defect was filed months ago.

-----Original Message-----
From: Rich LaReau=20
Sent: Thursday, January 24, 2008 4:49 PM
To: Larry Scheer; Andy Sharp; Sandrine Boulanger; Eric Barrett
Cc: Vikas Saini; Sudheesh Nair; Tim Gardner; Dennis Arellano; Andrew
LeFebvre; Caeli Collins; Paul Hammer; Brian Stark
Subject: RE: "system upgrade" vs. script for Cougar



Hi all,

After further discussion with Larry, we think that we have a simple and
robust solution with the current implementation of the "system upgrade"
command.  In particular, we should consider making our primary supported
upgrade process the following:

< get logs, insure backups are good... etc.>

Then run the two commands:

> system copy init=20
> system upgrade ...

This essentially replicates what the script is doing.  The first command
will make a clean CF with empty partitions.  The second will make a
clean installation.  (I've tested this in 3.2, and there's still a
wasted slow step of "checking to see what's there" before the install,
but Larry says we can clean that up easily. I'll file the ECR.)

We could consider standardizing on this method for 3.2 patch releases,
and make any subsequent changes in time for Cougar, if necessary.

Would this meet everyone's requirements?

Rich


-----Original Message-----
From: Rich LaReau=20
Sent: Thursday, January 24, 2008 9:04 AM
To: Larry Scheer; Andy Sharp
Cc: Vikas Saini; Sudheesh Nair; Tim Gardner; Dennis Arellano; Andrew
LeFebvre; Caeli Collins; Paul Hammer; Brian Stark
Subject: RE: "system upgrade" vs. script for Cougar


Thanks Andy and Larry, for the comments.  I guess it's hard for me to
say one way or the other whether the current "system upgrade" process is
good enough for everyone's concerns.  I believe that customers have been
using the scripts exclusively since they were available, so we have no
data otherwise.  The admittedly old perceptions of the "system upgrade"
commands have not been replaced by good experience in the field. =20

We could try to use the non-script process for patch releases for 3.2
and see how well they go.  Although any feedback we get might be too
late to incorporate back into Cougar, if needed.=20

Anybody else have other suggestions?

Rich


-----Original Message-----
From: Larry Scheer
Sent: Wednesday, January 23, 2008 5:42 PM
To: Andy Sharp; Rich LaReau
Cc: Vikas Saini; Sudheesh Nair; Tim Gardner; Dennis Arellano; Andrew
LeFebvre; Caeli Collins; Paul Hammer; Brian Stark
Subject: RE: "system upgrade" vs. script for Cougar

Thanks Andy for forwarding this message to me; shame on you Rich, for
not sending me a copy of your original message. ;-)

I placed my comments in line below.

Larry

-----Original Message-----
From: Andy Sharp
Sent: Wednesday, January 23, 2008 3:32 PM
To: Rich LaReau
Cc: Vikas Saini; Sudheesh Nair; Tim Gardner; Dennis Arellano; Andrew
LeFebvre; Larry Scheer; Caeli Collins; Paul Hammer; Brian Stark
Subject: Re: "system upgrade" vs. script for Cougar

On Wed, 23 Jan 2008 15:02:01 -0800 "Rich LaReau"
<rich.lareau@onstor.com> wrote:

>=20
> Hi team,
>=20
> Yesterday after the core team meeting I reviewed the issues that=20
> Support had with the "system upgrade" procedure.  The concerns and=20
> requirements appear to come down to a few issues.  I wanted to get=20
> them out for review, and then ask you what our next steps need to be.
>=20
> The issues which are of CS concern are:
>=20
> 1)   Failed upgrades.  Before we used the scripts, it was common that
> we would get upgrades which outright failed, or worse, appeared to=20
> work but contained some corruption.  If we are to rely on "system=20
> upgrade" then these issues need to be resolved.  By comparison, the=20
> scripts completely blank the secondary flash and then copy everything=20
> in fresh.  This avoids problems with copying cruft (old core files in=20
> odd places, repair attempts, lost+found files, etc.)

I think this is largely an old perception.  I personally know of all
these problems you mentioned being fixed at least by the 3.0 release if
not sooner. At this point in time, and consider that this is the person
who originally wrote the install script saying this, if I it was my
customer, and I was upgrading from a 3.x to a 3.x, I would use system
upgrade.  Or flash_install with auto-upgrade.

Note that there is a workflow that uses system upgrade that does what
you mention in terms of wiping the CF card and installing fresh. Doing a
_system copy init_ followed by a _system upgrade_.  Code was
specifically added to make this possible.

[LCS] I completely agree with Andy. Your perceptions are very old. The
problems with system upgrade corrupting installations were fixed in
Release 2.1. The ability to erase a flash and install (as Andy mentions)
was introduced in release 3.0 when the layout of the flash changed.

> 2)  Time of upgrade.  This problem was primarily the result of needing

> to download the entire tarball twice from the ftp server.  The scripts

> are able to work with only one download.
>=20

[LCS] The double download problem was fixed in 3.0. The cw_install
script was created because the 2.X system upgrade could not reliably
upgrade the system to 3.0 due to the SSC running out of memory. A side
benefit of the cw_install script is it ran in less time. But it was
designed to get customers from 2.X and earlier to 3.0 in a single
unmonitored step. I am working on a faster upgrade but I can't promise
anything for system W (BSD based releases.)

> 3) Ease of use.  We do a lot of upgrades, so the process needs to be
> simple-- both in the upgrade itself and the recovery process in case=20
> something fails.

As I understand it, many SEs make a flash with flash_install.sh before
they head to the customer site, and set it to auto-upgrade when it is
booted. That means that you can just insert the flash card in the
secondary slot, and do a reboot.  When it boots, it copies the
configuration info from the other flash card.  For down time, this is
definitely the fastest way to go, but obviously there are a lot of
caveats to that as well, not the least of which is logistical, which
prevents us (I'm told) from making this our default upgrade procedure.

[LCS] I'm with you on this one Rich!

> Some people had suggestions about how we might implement a good system

> upgrade process:

Excellent.  Ideas always most welcome.  Keep 'em coming.  Often.

> A)  The system upgrade command should look in the downloaded tarball=20
> for a script to execute.  If it exists, the script is run; if not, it=20
> uses the old method.  That means if we do other flash card layouts, or

> even more drastic changes, the script can manage all of that, and it=20
> will be tied to the release bundle rather than having to be inserted=20
> earlier in the development cycle.

This is the way it actually works now.  Believe me when I say that this
has been long thought of and desired.  Because of the current design of
executing a script from the upgrade tarball itself, many changes can be
made in one release, but not all.  For instance, being able to use HTTP
as well as FTP, well, that's going into the next release, but it won't
help when upgrading to that release.  But eventually it will help.

[LCS] This is the way system upgrade has been working since 2.1. A
change was just checked into the dev branch to allow the System W (BSD
based) release to use HTTP. But remember, hen you run system upgrade it
is the older nfxsh without the new capability that is doing the
download.

> B)  Standardize to the WebUI and replace the need for an FTP server=20
> all together.  We could specify that the tar ball and md5 file are in=20
> the same directory pick them both up and check them before install=20
> commences, therefore ruling out a corrupt tar ball as a problem for=20
> the installation. If we also had a new upgrade script we could have=20
> the process look for this before commencing the installation as well,=20
> replace the current install script and then start the process again.
> So the process would be, go to the WebUI, open the system upgrade=20
> page, point it the directory containing the scripts and click on the=20
> version to install, then sit back get a coffee/tea and watch the=20
> status bar go straight up to 90% before waiting another 30 minutes for

> the process to complete (sorry this isn't MS is it!)

Sounds good to me!  Let us know when you've got something to test!  And
yes, that is on my list of faves but earliest would be post Cougar
release, and no, it's not on any schedule.  Keep in mind that
implementing this is no small project, and I'm not even including the
WebUI development part.  Ease of use is not always easy to do ~:^)

Cheers,

a
