X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C86378.74CB50D5@onstor-exch02.onstor.net>; Wed, 30 Jan 2008 12:43:58 -0700
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
Subject: RE: "system upgrade" vs. script for Cougar
Date: Wed, 30 Jan 2008 12:43:58 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E056C9323@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E0805818E@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: "system upgrade" vs. script for Cougar
Thread-Index: Ache8fxYV4sy+k48R/6Gwp/9aDzS6wAAPmUAACmuf3AA8kZs0AAEnP1QAABTq2AAAE7V8A==
From: "Larry Scheer" <larry.scheer@onstor.com>
To: "Rich LaReau" <rich.lareau@onstor.com>,
	"Tim Gardner" <tim.gardner@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>
Cc: "Eric Barrett" <eric.barrett@onstor.com>

No, I am talking about actual install times. As Tim said in his email I
spent a lot of time making improvements to verify_install. I am not
promising anything, but on cougars I am seeing upgrade times of less
than 5 minutes, but system upgrade isn't finished and the overall time
may increase. Bobcat BSD systems are still in the neighborhood of 15
minutes.

-----Original Message-----
From: Rich LaReau=20
Sent: Wednesday, January 30, 2008 11:36 AM
To: Tim Gardner; Larry Scheer; Andy Sharp
Cc: Eric Barrett
Subject: RE: "system upgrade" vs. script for Cougar

=20
I'm not sure how that increases upgrade time, as the verify step seems
itself to take a very long time.  I guess Larry was saying that
installing everything from scratch will take 15-20 minutes instead of
the usual 5 min.  But if the verify step takes 10 minutes on its own,
then it sounds like a wash.  I haven't timed them that closely, do we
know what those numbers are?

Part of the request was to cover removing any "cruft" leftover that we
don't want to keep around.  A clean install would always insure that.
Is that covered by the verify step?

Rich


-----Original Message-----
From: Tim Gardner=20
Sent: Wednesday, January 30, 2008 11:23 AM
To: Rich LaReau; Larry Scheer; Andy Sharp
Subject: RE: "system upgrade" vs. script for Cougar

We need to talk about #1.
We have spend considerable time perfecting verify install so that it can
detect the files that need to be upgraded and only upgrade those files.
Always doing a clean install renders all this functionality useless and
will increase the upgrade time.

> -----Original Message-----
> From: Rich LaReau
> Sent: Wednesday, January 30, 2008 9:14 AM
> To: Larry Scheer; Tim Gardner; Andy Sharp
> Cc: Andrew LeFebvre
> Subject: FW: "system upgrade" vs. script for Cougar
>=20
>=20
> Hi guys,
>=20
> At yesterday's Cougar meeting Tim said that he (?) is working on a=20
> design doc for "system upgrade".  I just wanted to ask if there was=20
> anything left for me to do here.  I filed an ECR which Larry was going

> to review.  Here was the request:
>=20
>=20
> TED 21999
> Would like to get a few changes made to "system upgrade" behavior=20
> (unclear if these should be separate ECR's)
>=20
> 1)  Combine "system copy init" to default "system upgrade".  We want=20
> default behavior to be to initialize secondary flash and then run the=20
> upgrade, all in one step.  (Include a flag which will disable the
> initialization)
>=20
> 2)  Make secondary flash upgrade default.  Include flag which upgrades

> primary flash.  (Obviously, the "init" step above will never run under

> this option.)
>=20
> 3)  "system upgrade" should be able to recognize an initialized flash,

> and skip the step where it scans for which files it needs to download.

> The flash is empty, everything needs to be downloaded, so we should=20
> skip this step to save some time.
>=20
>=20
> Was there anything else?
> Rich
>=20
>=20
>=20
> -----Original Message-----
> From: Rich LaReau
> Sent: Friday, January 25, 2008 1:34 PM
> To: Eric Barrett; Andy Sharp; Dennis Arellano
> Cc: Sandrine Boulanger; Larry Scheer; Vikas Saini; Sudheesh Nair; Tim=20
> Gardner; Andrew LeFebvre; Caeli Collins; Paul Hammer; Brian Stark
> Subject: RE: "system upgrade" vs. script for Cougar
>=20
>=20
> I filed an ECR to capture the requests for the system upgrade.  Please

> have a look and let me know if this will meet our needs, and if it=20
> needs more information, etc.
>=20
> TED00021999
>=20
> Thanks for all the help and valuable input!
> Rich
>=20
>=20
> -----Original Message-----
> From: Eric Barrett
> Sent: Thursday, January 24, 2008 5:39 PM
> To: Andy Sharp; Dennis Arellano
> Cc: Sandrine Boulanger; Rich LaReau; Larry Scheer; Vikas Saini;=20
> Sudheesh Nair; Tim Gardner; Andrew LeFebvre; Caeli Collins; Paul=20
> Hammer; Brian Stark
> Subject: RE: "system upgrade" vs. script for Cougar
>=20
> That kind of exception makes a lot of sense for internal use, but not=20
> much for customers.  Why not also make that an 'exception' flag, akin=20
> to what Tim suggested about making the secondary the default target,=20
> and the primary usable via -p?
>=20
>=20
> -----Original Message-----
> From: Andy Sharp
> Sent: Thursday, January 24, 2008 5:31 PM
> To: Dennis Arellano
> Cc: Sandrine Boulanger; Rich LaReau; Larry Scheer; Eric Barrett; Vikas

> Saini; Sudheesh Nair; Tim Gardner; Andrew LeFebvre; Caeli Collins;=20
> Paul Hammer; Brian Stark
> Subject: Re: "system upgrade" vs. script for Cougar
>=20
> I like all of what I'm hearing, except for the bug part, of course.
> Nice bit of teamwork.
>=20
> As for Tim's question, I can picture instances where there are=20
> mods/customizations, bashrc files, etc., that I wouldn't want wiped=20
> out just to do an upgrade, so I think keeping it a two stepper is OK.

> Keep in mind that on Linux, the 'init' step takes 30s or less, so the=20
> impact is lower and the impatience factor is also lower.  In the near
future.
> ~:^)
>=20
> Cheers,
>=20
> a
>=20
> On Thu, 24 Jan 2008 17:17:31 -0800 "Dennis Arellano"
> <dennis.arellano@onstor.com> wrote:
>=20
> > Sounds like we are close to closure.
> >
> > Rich, when you have all the details worked out, we need to sit down=20
> > and write up the new Upgrade Procedure.
> >
> > Thanks, Dennis
> >
> > -----Original Message-----
> > From: Sandrine Boulanger
> > Sent: Thursday, January 24, 2008 4:53 PM
> > To: Rich LaReau; Larry Scheer; Andy Sharp; Eric Barrett
> > Cc: Vikas Saini; Sudheesh Nair; Tim Gardner; Dennis Arellano; Andrew

> > LeFebvre; Caeli Collins; Paul Hammer; Brian Stark
> > Subject: RE: "system upgrade" vs. script for Cougar
> >
> > There a bug that ins some cases, after a copy init, a system upgrade

> > -s does not work with the error "not compatible with this system" or

> > similar. I ran into it this morning. I had to manually mount / from=20
> > secondary flash and copy /version from primary to secondary to be=20
> > able to upgrade. This needs to be fixed if we want to use this=20
> > procedure. A defect was filed months ago.
> >
> > -----Original Message-----
> > From: Rich LaReau
> > Sent: Thursday, January 24, 2008 4:49 PM
> > To: Larry Scheer; Andy Sharp; Sandrine Boulanger; Eric Barrett
> > Cc: Vikas Saini; Sudheesh Nair; Tim Gardner; Dennis Arellano; Andrew

> > LeFebvre; Caeli Collins; Paul Hammer; Brian Stark
> > Subject: RE: "system upgrade" vs. script for Cougar
> >
> >
> >
> > Hi all,
> >
> > After further discussion with Larry, we think that we have a simple=20
> > and robust solution with the current implementation of the "system=20
> > upgrade" command.  In particular, we should consider making our=20
> > primary supported upgrade process the following:
> >
> > < get logs, insure backups are good... etc.>
> >
> > Then run the two commands:
> >
> > > system copy init
> > > system upgrade ...
> >
> > This essentially replicates what the script is doing.  The first=20
> > command will make a clean CF with empty partitions.  The second will

> > make a clean installation.  (I've tested this in 3.2, and there's=20
> > still a wasted slow step of "checking to see what's there" before=20
> > the install, but Larry says we can clean that up easily. I'll file=20
> > the
> > ECR.)
> >
> > We could consider standardizing on this method for 3.2 patch=20
> > releases, and make any subsequent changes in time for Cougar, if
necessary.
> >
> > Would this meet everyone's requirements?
> >
> > Rich
> >
> >
> > -----Original Message-----
> > From: Rich LaReau
> > Sent: Thursday, January 24, 2008 9:04 AM
> > To: Larry Scheer; Andy Sharp
> > Cc: Vikas Saini; Sudheesh Nair; Tim Gardner; Dennis Arellano; Andrew

> > LeFebvre; Caeli Collins; Paul Hammer; Brian Stark
> > Subject: RE: "system upgrade" vs. script for Cougar
> >
> >
> > Thanks Andy and Larry, for the comments.  I guess it's hard for me=20
> > to say one way or the other whether the current "system upgrade"=20
> > process is good enough for everyone's concerns.  I believe that=20
> > customers have been using the scripts exclusively since they were=20
> > available, so we have no data otherwise.  The admittedly old=20
> > perceptions of the "system upgrade" commands have not been replaced=20
> > by good experience in the field.
> >
> > We could try to use the non-script process for patch releases for=20
> > 3.2 and see how well they go.  Although any feedback we get might be

> > too late to incorporate back into Cougar, if needed.
> >
> > Anybody else have other suggestions?
> >
> > Rich
> >
> >
> > -----Original Message-----
> > From: Larry Scheer
> > Sent: Wednesday, January 23, 2008 5:42 PM
> > To: Andy Sharp; Rich LaReau
> > Cc: Vikas Saini; Sudheesh Nair; Tim Gardner; Dennis Arellano; Andrew

> > LeFebvre; Caeli Collins; Paul Hammer; Brian Stark
> > Subject: RE: "system upgrade" vs. script for Cougar
> >
> > Thanks Andy for forwarding this message to me; shame on you Rich,=20
> > for not sending me a copy of your original message. ;-)
> >
> > I placed my comments in line below.
> >
> > Larry
> >
> > -----Original Message-----
> > From: Andy Sharp
> > Sent: Wednesday, January 23, 2008 3:32 PM
> > To: Rich LaReau
> > Cc: Vikas Saini; Sudheesh Nair; Tim Gardner; Dennis Arellano; Andrew

> > LeFebvre; Larry Scheer; Caeli Collins; Paul Hammer; Brian Stark
> > Subject: Re: "system upgrade" vs. script for Cougar
> >
> > On Wed, 23 Jan 2008 15:02:01 -0800 "Rich LaReau"
> > <rich.lareau@onstor.com> wrote:
> >
> > >
> > > Hi team,
> > >
> > > Yesterday after the core team meeting I reviewed the issues that=20
> > > Support had with the "system upgrade" procedure.  The concerns and

> > > requirements appear to come down to a few issues.  I wanted to get

> > > them out for review, and then ask you what our next steps need to=20
> > > be.
> > >
> > > The issues which are of CS concern are:
> > >
> > > 1)   Failed upgrades.  Before we used the scripts, it was common
> > > that we would get upgrades which outright failed, or worse,=20
> > > appeared to work but contained some corruption.  If we are to rely

> > > on "system upgrade" then these issues need to be resolved.  By=20
> > > comparison, the scripts completely blank the secondary flash and=20
> > > then copy everything in fresh.  This avoids problems with copying=20
> > > cruft (old core files in odd places, repair attempts, lost+found=20
> > > files, etc.)
> >
> > I think this is largely an old perception.  I personally know of all

> > these problems you mentioned being fixed at least by the 3.0 release

> > if not sooner. At this point in time, and consider that this is the=20
> > person who originally wrote the install script saying this, if I it=20
> > was my customer, and I was upgrading from a 3.x to a 3.x, I would=20
> > use system upgrade.  Or flash_install with auto-upgrade.
> >
> > Note that there is a workflow that uses system upgrade that does=20
> > what you mention in terms of wiping the CF card and installing
fresh.
> > Doing a _system copy init_ followed by a _system upgrade_.  Code was

> > specifically added to make this possible.
> >
> > [LCS] I completely agree with Andy. Your perceptions are very old.=20
> > The problems with system upgrade corrupting installations were fixed

> > in Release 2.1. The ability to erase a flash and install (as Andy
> > mentions) was introduced in release 3.0 when the layout of the flash

> > changed.
> >
> > > 2)  Time of upgrade.  This problem was primarily the result of=20
> > > needing
> >
> > > to download the entire tarball twice from the ftp server.  The=20
> > > scripts
> >
> > > are able to work with only one download.
> > >
> >
> > [LCS] The double download problem was fixed in 3.0. The cw_install=20
> > script was created because the 2.X system upgrade could not reliably

> > upgrade the system to 3.0 due to the SSC running out of memory. A=20
> > side benefit of the cw_install script is it ran in less time. But it

> > was designed to get customers from 2.X and earlier to 3.0 in a=20
> > single unmonitored step. I am working on a faster upgrade but I=20
> > can't promise anything for system W (BSD based releases.)
> >
> > > 3) Ease of use.  We do a lot of upgrades, so the process needs to=20
> > > be
> > > simple-- both in the upgrade itself and the recovery process in=20
> > > case something fails.
> >
> > As I understand it, many SEs make a flash with flash_install.sh=20
> > before they head to the customer site, and set it to auto-upgrade=20
> > when it is booted. That means that you can just insert the flash=20
> > card in the secondary slot, and do a reboot.  When it boots, it=20
> > copies the configuration info from the other flash card.  For down=20
> > time, this is definitely the fastest way to go, but obviously there=20
> > are a lot of caveats to that as well, not the least of which is=20
> > logistical, which prevents us (I'm told) from making this our
default upgrade procedure.
> >
> > [LCS] I'm with you on this one Rich!
> >
> > > Some people had suggestions about how we might implement a good=20
> > > system
> >
> > > upgrade process:
> >
> > Excellent.  Ideas always most welcome.  Keep 'em coming.  Often.
> >
> > > A)  The system upgrade command should look in the downloaded=20
> > > tarball for a script to execute.  If it exists, the script is run;

> > > if not, it uses the old method.  That means if we do other flash=20
> > > card layouts, or
> >
> > > even more drastic changes, the script can manage all of that, and=20
> > > it will be tied to the release bundle rather than having to be=20
> > > inserted earlier in the development cycle.
> >
> > This is the way it actually works now.  Believe me when I say that=20
> > this has been long thought of and desired.  Because of the current=20
> > design of executing a script from the upgrade tarball itself, many=20
> > changes can be made in one release, but not all.  For instance,=20
> > being able to use HTTP as well as FTP, well, that's going into the=20
> > next release, but it won't help when upgrading to that release.  But

> > eventually it will help.
> >
> > [LCS] This is the way system upgrade has been working since 2.1. A=20
> > change was just checked into the dev branch to allow the System W=20
> > (BSD
> > based) release to use HTTP. But remember, hen you run system upgrade

> > it is the older nfxsh without the new capability that is doing the=20
> > download.
> >
> > > B)  Standardize to the WebUI and replace the need for an FTP=20
> > > server all together.  We could specify that the tar ball and md5=20
> > > file are in the same directory pick them both up and check them=20
> > > before install commences, therefore ruling out a corrupt tar ball=20
> > > as a problem for the installation. If we also had a new upgrade=20
> > > script we could have the process look for this before commencing=20
> > > the installation as well, replace the current install script and=20
> > > then start the process again. So the process would be, go to the=20
> > > WebUI, open the system upgrade page, point it the directory=20
> > > containing the scripts and click on the version to install, then=20
> > > sit back get a coffee/tea and watch the status bar go straight up=20
> > > to 90% before waiting another 30 minutes for
> >
> > > the process to complete (sorry this isn't MS is it!)
> >
> > Sounds good to me!  Let us know when you've got something to test!
> > And yes, that is on my list of faves but earliest would be post=20
> > Cougar release, and no, it's not on any schedule.  Keep in mind that

> > implementing this is no small project, and I'm not even including=20
> > the WebUI development part.  Ease of use is not always easy to do=20
> > ~:^)
> >
> > Cheers,
> >
> > a
