X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7AF09.E4A9D169@onstor-exch02.onstor.net>; Thu, 14 Jun 2007 20:59:02 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7AF09.E4A9D169"
References: <BB375AF679D4A34E9CA8DFA650E2B04E042A4F68@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E03E564FF@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E02F3D29B@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E03E56503@onstor-exch02.onstor.net>
Content-class: urn:content-classes:message
Subject: RE: New Perforce server online...almost
Date: Thu, 14 Jun 2007 20:59:02 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E02F3D29F@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: New Perforce server online...almost
Thread-Index: Aceu5xnmcLGj2XaqQA+ae1rCv62r0AAGqsYZAABOImEAAHAukAAA+ZG9
From: "Ken Renshaw" <ken.renshaw@onstor.com>
To: "Paul Hammer" <paul.hammer@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>,
	"John Rogers" <john.rogers@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7AF09.E4A9D169
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Is that me sabotaging the cougar team Paul, or some developer sick of a =
slow Perforce server crashing it out of his own spite?=20

I guess we'll never know which it really was B)

-Ken

P.S. Actually, it was me killing a process, which *should* have been =
fine according to Perforce, but we hit a race bug whereby an incoming =
checkin ( Larry's merge ), the branch it's headed for ( cougar ), and =
the loss of in-memory state collide, and the result is a corruption the =
database files. In the land of Perforce if you're database is gone =
you're hosed and then it's time to find out if your IT is *really* =
backing your data up and if mightydog really *is* a good place to store =
your backup images...luckily for us it was.

P.P.S. This also meets the criterion for the full Disaster Recovery Plan =
we are enacting at ONStor prior to our filing an IPO. We should actually =
send the old Perforce server up to Pleasanton and make it a full =
remote/backup depot. This would not only facilitate business continuity =
but make the proof to the auditors that we are redundant and ready to =
go.


-----Original Message-----
From: Paul Hammer
Sent: Thu 6/14/2007 9:22 PM
To: Ken Renshaw; Andy Sharp; John Rogers
Subject: RE: New Perforce server online...almost
=20
Ah-HA! I suspect Sabotage!

________________________________

From: Ken Renshaw
Sent: Thu 6/14/2007 9:09 PM
To: Paul Hammer; Andy Sharp; John Rogers
Subject: RE: New Perforce server online...almost



You're welcome, I was planning on revamping the Perforce server =
anyway....just not *now*.

Thanks guys.

-Ken


-----Original Message-----
From: Paul Hammer
Sent: Thu 6/14/2007 9:00 PM
To: Ken Renshaw; Andy Sharp; John Rogers
Subject: RE: New Perforce server online...almost

Very nice, I appreciate the help/effort that each of you provided to get =
this critical resource going again..

John R, do I need to buy you a replacement machine for the one you =
donated to this critical effort? Just let me know.

-Paul

________________________________

From: Ken Renshaw
Sent: Thu 6/14/2007 5:49 PM
To: dl-Engineering
Subject: New Perforce server online...almost



The new server hardware is up and running the latest version of the =
Perforce server software, and I've run some tests and things look okay ( =
and much faster to boot ). Currently all branches are locked so I can =
complete a baseline backup and perform a test build. The outage occurred =
on Tuesday, and since I rolled back to Monday's backup files I still =
have to add the changes that came in the last couple days by hand. This =
will bring the top-of-tree for the active branches up to date with all =
current work checked in. I will be working on this recovery operation =
this evening and as I finish rebuilding branches I will open them up to =
you.



The branches I need to fix up are:



support

FB-DELOREAN

ZONDA

R2_2_0_rel

R2_2_2_x_rel

cougar

FB-NOSAMBA



Thanks for your patience throughout all of this. The new server software =
has many new features, is much faster and smarter WRT integration =
histories, and is on a nice and fast 64-bit linux kernel with 64-bit =
Perforce serve binaries and double the RAM. Thanks to John R. for =
providing it, and to Andy for helping John bring up the Debian linux we =
used for the OS install.



I'll send out an email later when things are open for business.



-Ken






------_=_NextPart_001_01C7AF09.E4A9D169
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Diso-8859-1">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7652.24">
<TITLE>RE: New Perforce server online...almost</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/plain format -->

<P><FONT SIZE=3D2>Is that me sabotaging the cougar team Paul, or some =
developer sick of a slow Perforce server crashing it out of his own =
spite?<BR>
<BR>
I guess we'll never know which it really was B)<BR>
<BR>
-Ken<BR>
<BR>
P.S. Actually, it was me killing a process, which *should* have been =
fine according to Perforce, but we hit a race bug whereby an incoming =
checkin ( Larry's merge ), the branch it's headed for ( cougar ), and =
the loss of in-memory state collide, and the result is a corruption the =
database files. In the land of Perforce if you're database is gone =
you're hosed and then it's time to find out if your IT is *really* =
backing your data up and if mightydog really *is* a good place to store =
your backup images...luckily for us it was.<BR>
<BR>
P.P.S. This also meets the criterion for the full Disaster Recovery Plan =
we are enacting at ONStor prior to our filing an IPO. We should actually =
send the old Perforce server up to Pleasanton and make it a full =
remote/backup depot. This would not only facilitate business continuity =
but make the proof to the auditors that we are redundant and ready to =
go.<BR>
<BR>
<BR>
-----Original Message-----<BR>
From: Paul Hammer<BR>
Sent: Thu 6/14/2007 9:22 PM<BR>
To: Ken Renshaw; Andy Sharp; John Rogers<BR>
Subject: RE: New Perforce server online...almost<BR>
<BR>
Ah-HA! I suspect Sabotage!<BR>
<BR>
________________________________<BR>
<BR>
From: Ken Renshaw<BR>
Sent: Thu 6/14/2007 9:09 PM<BR>
To: Paul Hammer; Andy Sharp; John Rogers<BR>
Subject: RE: New Perforce server online...almost<BR>
<BR>
<BR>
<BR>
You're welcome, I was planning on revamping the Perforce server =
anyway....just not *now*.<BR>
<BR>
Thanks guys.<BR>
<BR>
-Ken<BR>
<BR>
<BR>
-----Original Message-----<BR>
From: Paul Hammer<BR>
Sent: Thu 6/14/2007 9:00 PM<BR>
To: Ken Renshaw; Andy Sharp; John Rogers<BR>
Subject: RE: New Perforce server online...almost<BR>
<BR>
Very nice, I appreciate the help/effort that each of you provided to get =
this critical resource going again..<BR>
<BR>
John R, do I need to buy you a replacement machine for the one you =
donated to this critical effort? Just let me know.<BR>
<BR>
-Paul<BR>
<BR>
________________________________<BR>
<BR>
From: Ken Renshaw<BR>
Sent: Thu 6/14/2007 5:49 PM<BR>
To: dl-Engineering<BR>
Subject: New Perforce server online...almost<BR>
<BR>
<BR>
<BR>
The new server hardware is up and running the latest version of the =
Perforce server software, and I've run some tests and things look okay ( =
and much faster to boot ). Currently all branches are locked so I can =
complete a baseline backup and perform a test build. The outage occurred =
on Tuesday, and since I rolled back to Monday's backup files I still =
have to add the changes that came in the last couple days by hand. This =
will bring the top-of-tree for the active branches up to date with all =
current work checked in. I will be working on this recovery operation =
this evening and as I finish rebuilding branches I will open them up to =
you.<BR>
<BR>
<BR>
<BR>
The branches I need to fix up are:<BR>
<BR>
<BR>
<BR>
support<BR>
<BR>
FB-DELOREAN<BR>
<BR>
ZONDA<BR>
<BR>
R2_2_0_rel<BR>
<BR>
R2_2_2_x_rel<BR>
<BR>
cougar<BR>
<BR>
FB-NOSAMBA<BR>
<BR>
<BR>
<BR>
Thanks for your patience throughout all of this. The new server software =
has many new features, is much faster and smarter WRT integration =
histories, and is on a nice and fast 64-bit linux kernel with 64-bit =
Perforce serve binaries and double the RAM. Thanks to John R. for =
providing it, and to Andy for helping John bring up the Debian linux we =
used for the OS install.<BR>
<BR>
<BR>
<BR>
I'll send out an email later when things are open for business.<BR>
<BR>
<BR>
<BR>
-Ken<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
</FONT>
</P>

</BODY>
</HTML>
------_=_NextPart_001_01C7AF09.E4A9D169--
