X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7AF10.0C6E1C5F@onstor-exch02.onstor.net>; Thu, 14 Jun 2007 21:43:06 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7AF10.0C6E1C5F"
References: <BB375AF679D4A34E9CA8DFA650E2B04E042A4F68@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E03E564FF@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E02F3D29B@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E03E56503@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E02F3D29F@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E03E56506@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E02F3D2A1@onstor-exch02.onstor.net>
Content-class: urn:content-classes:message
Subject: RE: New Perforce server online...almost
Date: Thu, 14 Jun 2007 21:41:20 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E03E56507@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: New Perforce server online...almost
Thread-Index: Aceu5xnmcLGj2XaqQA+ae1rCv62r0AAGqsYZAABOImEAAHAukAAA+ZG9AADITagAAMOjUQAAPkbT
From: "Paul Hammer" <paul.hammer@onstor.com>
To: "Ken Renshaw" <ken.renshaw@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>,
	"John Rogers" <john.rogers@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7AF10.0C6E1C5F
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Got it Ken. John R, can we have John V get the old system up to P'town, =
rack it and make it available on the network? If so then we can work the =
rest with Kevin/Brian B.

________________________________

From: Ken Renshaw
Sent: Thu 6/14/2007 10:34 PM
To: Paul Hammer; Andy Sharp; John Rogers
Subject: RE: New Perforce server online...almost



Thanks Paul. All I need is the old server we decommissioned today, and =
some IT/elab help getting it racked up and integrated into the P-town =
lab. Once we have that, I'll handle the rest, and Brian Baker will be =
much relieved....he was going to give me a vmware 'machine' on a windows =
box in Pleasanton....shared with a bunch of other linux vm's. Yuck.

I'm glad you're behind the idea, I've been waiting for a moment to sneak =
it into the conversation.

Thanks,

-Ken


-----Original Message-----
From: Paul Hammer
Sent: Thu 6/14/2007 10:12 PM
To: Ken Renshaw; Andy Sharp; John Rogers
Subject: RE: New Perforce server online...almost

All kidding aside, I like and support the idea of an offsite IP =
insurance policy, lets do that!
Do you need any assistance from me to drive that forward. Great idea =
Ken.

________________________________

From: Ken Renshaw
Sent: Thu 6/14/2007 9:59 PM
To: Paul Hammer; Andy Sharp; John Rogers
Subject: RE: New Perforce server online...almost



Is that me sabotaging the cougar team Paul, or some developer sick of a =
slow Perforce server crashing it out of his own spite?

I guess we'll never know which it really was B)

-Ken

P.S. Actually, it was me killing a process, which *should* have been =
fine according to Perforce, but we hit a race bug whereby an incoming =
checkin ( Larry's merge ), the branch it's headed for ( cougar ), and =
the loss of in-memory state collide, and the result is a corruption the =
database files. In the land of Perforce if you're database is gone =
you're hosed and then it's time to find out if your IT is *really* =
backing your data up and if mightydog really *is* a good place to store =
your backup images...luckily for us it was.

P.P.S. This also meets the criterion for the full Disaster Recovery Plan =
we are enacting at ONStor prior to our filing an IPO. We should actually =
send the old Perforce server up to Pleasanton and make it a full =
remote/backup depot. This would not only facilitate business continuity =
but make the proof to the auditors that we are redundant and ready to =
go.


-----Original Message-----
From: Paul Hammer
Sent: Thu 6/14/2007 9:22 PM
To: Ken Renshaw; Andy Sharp; John Rogers
Subject: RE: New Perforce server online...almost

Ah-HA! I suspect Sabotage!

________________________________

From: Ken Renshaw
Sent: Thu 6/14/2007 9:09 PM
To: Paul Hammer; Andy Sharp; John Rogers
Subject: RE: New Perforce server online...almost



You're welcome, I was planning on revamping the Perforce server =
anyway....just not *now*.

Thanks guys.

-Ken


-----Original Message-----
From: Paul Hammer
Sent: Thu 6/14/2007 9:00 PM
To: Ken Renshaw; Andy Sharp; John Rogers
Subject: RE: New Perforce server online...almost

Very nice, I appreciate the help/effort that each of you provided to get =
this critical resource going again..

John R, do I need to buy you a replacement machine for the one you =
donated to this critical effort? Just let me know.

-Paul

________________________________

From: Ken Renshaw
Sent: Thu 6/14/2007 5:49 PM
To: dl-Engineering
Subject: New Perforce server online...almost



The new server hardware is up and running the latest version of the =
Perforce server software, and I've run some tests and things look okay ( =
and much faster to boot ). Currently all branches are locked so I can =
complete a baseline backup and perform a test build. The outage occurred =
on Tuesday, and since I rolled back to Monday's backup files I still =
have to add the changes that came in the last couple days by hand. This =
will bring the top-of-tree for the active branches up to date with all =
current work checked in. I will be working on this recovery operation =
this evening and as I finish rebuilding branches I will open them up to =
you.



The branches I need to fix up are:



support

FB-DELOREAN

ZONDA

R2_2_0_rel

R2_2_2_x_rel

cougar

FB-NOSAMBA



Thanks for your patience throughout all of this. The new server software =
has many new features, is much faster and smarter WRT integration =
histories, and is on a nice and fast 64-bit linux kernel with 64-bit =
Perforce serve binaries and double the RAM. Thanks to John R. for =
providing it, and to Andy for helping John bring up the Debian linux we =
used for the OS install.



I'll send out an email later when things are open for business.



-Ken











------_=_NextPart_001_01C7AF10.0C6E1C5F
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<HTML dir=3Dltr><HEAD><TITLE>RE: New Perforce server =
online...almost</TITLE>=0A=
<META http-equiv=3DContent-Type content=3D"text/html; charset=3Dunicode">=0A=
<META content=3D"MSHTML 6.00.2900.3059" name=3DGENERATOR></HEAD>=0A=
<BODY>=0A=
<DIV id=3DidOWAReplyText9420 dir=3Dltr>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 size=3D2>Got it Ken. =
John R, can we have John V get&nbsp;the old system&nbsp;up to P'town, =
rack it and make it available on the network? If so then we can work the =
rest with Kevin/Brian B.</FONT></DIV></DIV>=0A=
<DIV dir=3Dltr><BR>=0A=
<HR tabIndex=3D-1>=0A=
<FONT face=3DTahoma size=3D2><B>From:</B> Ken Renshaw<BR><B>Sent:</B> =
Thu 6/14/2007 10:34 PM<BR><B>To:</B> Paul Hammer; Andy Sharp; John =
Rogers<BR><B>Subject:</B> RE: New Perforce server =
online...almost<BR></FONT><BR></DIV>=0A=
<DIV>=0A=
<P><FONT size=3D2>Thanks Paul. All I need is the old server we =
decommissioned today, and some IT/elab help getting it racked up and =
integrated into the P-town lab. Once we have that, I'll handle the rest, =
and Brian Baker will be much relieved....he was going to give me a =
vmware 'machine' on a windows box in Pleasanton....shared with a bunch =
of other linux vm's. Yuck.<BR><BR>I'm glad you're behind the idea, I've =
been waiting for a moment to sneak it into the =
conversation.<BR><BR>Thanks,<BR><BR>-Ken<BR><BR><BR>-----Original =
Message-----<BR>From: Paul Hammer<BR>Sent: Thu 6/14/2007 10:12 PM<BR>To: =
Ken Renshaw; Andy Sharp; John Rogers<BR>Subject: RE: New Perforce server =
online...almost<BR><BR>All kidding aside, I like and support the idea of =
an offsite IP insurance policy, lets do that!<BR>Do you need any =
assistance from me to drive that forward. Great idea =
Ken.<BR><BR>________________________________<BR><BR>From: Ken =
Renshaw<BR>Sent: Thu 6/14/2007 9:59 PM<BR>To: Paul Hammer; Andy Sharp; =
John Rogers<BR>Subject: RE: New Perforce server =
online...almost<BR><BR><BR><BR>Is that me sabotaging the cougar team =
Paul, or some developer sick of a slow Perforce server crashing it out =
of his own spite?<BR><BR>I guess we'll never know which it really was =
B)<BR><BR>-Ken<BR><BR>P.S. Actually, it was me killing a process, which =
*should* have been fine according to Perforce, but we hit a race bug =
whereby an incoming checkin ( Larry's merge ), the branch it's headed =
for ( cougar ), and the loss of in-memory state collide, and the result =
is a corruption the database files. In the land of Perforce if you're =
database is gone you're hosed and then it's time to find out if your IT =
is *really* backing your data up and if mightydog really *is* a good =
place to store your backup images...luckily for us it was.<BR><BR>P.P.S. =
This also meets the criterion for the full Disaster Recovery Plan we are =
enacting at ONStor prior to our filing an IPO. We should actually send =
the old Perforce server up to Pleasanton and make it a full =
remote/backup depot. This would not only facilitate business continuity =
but make the proof to the auditors that we are redundant and ready to =
go.<BR><BR><BR>-----Original Message-----<BR>From: Paul Hammer<BR>Sent: =
Thu 6/14/2007 9:22 PM<BR>To: Ken Renshaw; Andy Sharp; John =
Rogers<BR>Subject: RE: New Perforce server online...almost<BR><BR>Ah-HA! =
I suspect Sabotage!<BR><BR>________________________________<BR><BR>From: =
Ken Renshaw<BR>Sent: Thu 6/14/2007 9:09 PM<BR>To: Paul Hammer; Andy =
Sharp; John Rogers<BR>Subject: RE: New Perforce server =
online...almost<BR><BR><BR><BR>You're welcome, I was planning on =
revamping the Perforce server anyway....just not *now*.<BR><BR>Thanks =
guys.<BR><BR>-Ken<BR><BR><BR>-----Original Message-----<BR>From: Paul =
Hammer<BR>Sent: Thu 6/14/2007 9:00 PM<BR>To: Ken Renshaw; Andy Sharp; =
John Rogers<BR>Subject: RE: New Perforce server =
online...almost<BR><BR>Very nice, I appreciate the help/effort that each =
of you provided to get this critical resource going again..<BR><BR>John =
R, do I need to buy you a replacement machine for the one you donated to =
this critical effort? Just let me =
know.<BR><BR>-Paul<BR><BR>________________________________<BR><BR>From: =
Ken Renshaw<BR>Sent: Thu 6/14/2007 5:49 PM<BR>To: =
dl-Engineering<BR>Subject: New Perforce server =
online...almost<BR><BR><BR><BR>The new server hardware is up and running =
the latest version of the Perforce server software, and I've run some =
tests and things look okay ( and much faster to boot ). Currently all =
branches are locked so I can complete a baseline backup and perform a =
test build. The outage occurred on Tuesday, and since I rolled back to =
Monday's backup files I still have to add the changes that came in the =
last couple days by hand. This will bring the top-of-tree for the active =
branches up to date with all current work checked in. I will be working =
on this recovery operation this evening and as I finish rebuilding =
branches I will open them up to you.<BR><BR><BR><BR>The branches I need =
to fix up =
are:<BR><BR><BR><BR>support<BR><BR>FB-DELOREAN<BR><BR>ZONDA<BR><BR>R2_2_0=
_rel<BR><BR>R2_2_2_x_rel<BR><BR>cougar<BR><BR>FB-NOSAMBA<BR><BR><BR><BR>T=
hanks for your patience throughout all of this. The new server software =
has many new features, is much faster and smarter WRT integration =
histories, and is on a nice and fast 64-bit linux kernel with 64-bit =
Perforce serve binaries and double the RAM. Thanks to John R. for =
providing it, and to Andy for helping John bring up the Debian linux we =
used for the OS install.<BR><BR><BR><BR>I'll send out an email later =
when things are open for =
business.<BR><BR><BR><BR>-Ken<BR><BR><BR><BR><BR><BR><BR><BR><BR></FONT><=
/P></DIV></BODY></HTML>
------_=_NextPart_001_01C7AF10.0C6E1C5F--
