X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7AE0A.F96D6EBD@onstor-exch02.onstor.net>; Wed, 13 Jun 2007 14:34:16 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7AE0A.F96D6EBD"
Content-class: urn:content-classes:message
Subject: RE: Check-in of main-->cougar messed up.
Date: Wed, 13 Jun 2007 14:34:16 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E02F3D27D@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Check-in of main-->cougar messed up.
Thread-Index: AceuBqnvkH99KtWjQFa4Mwym66PurAAAf+zY
References: <BB375AF679D4A34E9CA8DFA650E2B04E02215850@onstor-exch02.onstor.net>
From: "Ken Renshaw" <ken.renshaw@onstor.com>
To: "Larry Scheer" <larry.scheer@onstor.com>,
	"dl-Cougar" <dl-Cougar@onstor.com>
Cc: "Ken Renshaw" <ken.renshaw@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7AE0A.F96D6EBD
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

That's what I was afraid of when I saw the short list. So far I've seen =
3 things wrong since yesterday's crash and burn:

-3 changelist descriptions lacked the "Affected files" list on the =
bottom of the text. The files were checked in and the changes themselves =
intact, but the description was lacking. I manually fixed up those three =
descriptions.

-Your main->cougar integration was beamed somewhere far, far away.

-a full sync of //depot/cougar/... fails, and I've whittled it down to =
//depot/cougar/openbsd/src/lib/libssl/... which when accessed yields the =
error a few have seen:

----------
Operation: user-sync

Operation 'dbscan' failed.

Database scan error on db.rev!

dbscan: db.rev: No child processes

Corrupt tree

----------

No other branch seems affected ( I've tested zonda, main, delorean, and =
R2_2_0_rel so far ). I've talked to Perforce support and the only way to =
correct this is to rebuild the db.* files, and the only way to do that =
is to restore from the last know good checkpoint and journal files. =
There have been precisely 3 checkins since my last checkpoint, one of =
which is the bad main->cougar merge so that leaves 2. What would happen =
is I would capture the revisions of the files for those couple checkins, =
rebuild the entire database as it was before the crash, and then =
manually add back in the changes so we wouldn't lose anything. Except of =
course the main->cougar merge. Larry, please replicate that source tree =
outside the client workspace and when the server is back up we'll =
rebuild the changelist that got lost. Perforce support highly recommends =
upgrading to the latest server version before playing back the =
checkpoint since a) the new version is faster/better at the replay and =
b) there are lots of bug fixes we really really want.=20

I need to come up with a game plan for reconstructing the server, =
starting hopefully with comandeering a faster system to bring up in =
parallel from John R. or something. Once I have a system and plan in =
place I'll let people know and we can schedule the outage ( I'm assuming =
sooner is far better than later...we don't want to continue very long =
with this corruption in place ).

In the meantime you are probably okay if you just sync =
//depot/cougar/nfx-tree/... and not the whole branch else you'll hit the =
openbsd/... issue.

-Ken


-----Original Message-----
From: Larry Scheer
Sent: Wed 6/13/2007 3:03 PM
To: dl-Cougar; Ken Renshaw
Subject: Check-in of main-->cougar messed up.
=20
From what I can see it looks like the majority of the files in the =
change list did not get submitted last night. When perforce went down it =
lost the bulk of the submittal.

Larry


------_=_NextPart_001_01C7AE0A.F96D6EBD
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Diso-8859-1">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7652.24">
<TITLE>RE: Check-in of main--&gt;cougar messed up.</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/plain format -->

<P><FONT SIZE=3D2>That's what I was afraid of when I saw the short list. =
So far I've seen 3 things wrong since yesterday's crash and burn:<BR>
<BR>
-3 changelist descriptions lacked the &quot;Affected files&quot; list on =
the bottom of the text. The files were checked in and the changes =
themselves intact, but the description was lacking. I manually fixed up =
those three descriptions.<BR>
<BR>
-Your main-&gt;cougar integration was beamed somewhere far, far =
away.<BR>
<BR>
-a full sync of //depot/cougar/... fails, and I've whittled it down to =
//depot/cougar/openbsd/src/lib/libssl/... which when accessed yields the =
error a few have seen:<BR>
<BR>
----------<BR>
Operation: user-sync<BR>
<BR>
Operation 'dbscan' failed.<BR>
<BR>
Database scan error on db.rev!<BR>
<BR>
dbscan: db.rev: No child processes<BR>
<BR>
Corrupt tree<BR>
<BR>
----------<BR>
<BR>
No other branch seems affected ( I've tested zonda, main, delorean, and =
R2_2_0_rel so far ). I've talked to Perforce support and the only way to =
correct this is to rebuild the db.* files, and the only way to do that =
is to restore from the last know good checkpoint and journal files. =
There have been precisely 3 checkins since my last checkpoint, one of =
which is the bad main-&gt;cougar merge so that leaves 2. What would =
happen is I would capture the revisions of the files for those couple =
checkins, rebuild the entire database as it was before the crash, and =
then manually add back in the changes so we wouldn't lose anything. =
Except of course the main-&gt;cougar merge. Larry, please replicate that =
source tree outside the client workspace and when the server is back up =
we'll rebuild the changelist that got lost. Perforce support highly =
recommends upgrading to the latest server version before playing back =
the checkpoint since a) the new version is faster/better at the replay =
and b) there are lots of bug fixes we really really want.<BR>
<BR>
I need to come up with a game plan for reconstructing the server, =
starting hopefully with comandeering a faster system to bring up in =
parallel from John R. or something. Once I have a system and plan in =
place I'll let people know and we can schedule the outage ( I'm assuming =
sooner is far better than later...we don't want to continue very long =
with this corruption in place ).<BR>
<BR>
In the meantime you are probably okay if you just sync =
//depot/cougar/nfx-tree/... and not the whole branch else you'll hit the =
openbsd/... issue.<BR>
<BR>
-Ken<BR>
<BR>
<BR>
-----Original Message-----<BR>
From: Larry Scheer<BR>
Sent: Wed 6/13/2007 3:03 PM<BR>
To: dl-Cougar; Ken Renshaw<BR>
Subject: Check-in of main--&gt;cougar messed up.<BR>
<BR>
From what I can see it looks like the majority of the files in the =
change list did not get submitted last night. When perforce went down it =
lost the bulk of the submittal.<BR>
<BR>
Larry<BR>
<BR>
</FONT>
</P>

</BODY>
</HTML>
------_=_NextPart_001_01C7AE0A.F96D6EBD--
