X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7AE12.504115C3@onstor-exch02.onstor.net>; Wed, 13 Jun 2007 15:26:48 -0800
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
Subject: RE: Check-in of main-->cougar messed up.
Date: Wed, 13 Jun 2007 15:26:47 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E042A48D5@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E02F3D27D@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Check-in of main-->cougar messed up.
Thread-Index: AceuBqnvkH99KtWjQFa4Mwym66PurAAAf+zYAAJcHoA=
From: "Tim Gardner" <tim.gardner@onstor.com>
To: "Ken Renshaw" <ken.renshaw@onstor.com>,
	"Larry Scheer" <larry.scheer@onstor.com>,
	"dl-Cougar" <dl-Cougar@onstor.com>

If we need to roll back, perhaps this could be used as an opportunity to
fix the checkin that resulted in having to integrate 65K files into
cougar.
Larry, could you please talk to Ken about our conversation last night
about
redoing the bad delorean integrate.

-----Original Message-----
From: Ken Renshaw=20
Sent: Wednesday, June 13, 2007 3:34 PM
To: Larry Scheer; dl-Cougar
Cc: Ken Renshaw
Subject: RE: Check-in of main-->cougar messed up.

That's what I was afraid of when I saw the short list. So far I've seen
3 things wrong since yesterday's crash and burn:

-3 changelist descriptions lacked the "Affected files" list on the
bottom of the text. The files were checked in and the changes themselves
intact, but the description was lacking. I manually fixed up those three
descriptions.

-Your main->cougar integration was beamed somewhere far, far away.

-a full sync of //depot/cougar/... fails, and I've whittled it down to
//depot/cougar/openbsd/src/lib/libssl/... which when accessed yields the
error a few have seen:

----------
Operation: user-sync

Operation 'dbscan' failed.

Database scan error on db.rev!

dbscan: db.rev: No child processes

Corrupt tree

----------

No other branch seems affected ( I've tested zonda, main, delorean, and
R2_2_0_rel so far ). I've talked to Perforce support and the only way to
correct this is to rebuild the db.* files, and the only way to do that
is to restore from the last know good checkpoint and journal files.
There have been precisely 3 checkins since my last checkpoint, one of
which is the bad main->cougar merge so that leaves 2. What would happen
is I would capture the revisions of the files for those couple checkins,
rebuild the entire database as it was before the crash, and then
manually add back in the changes so we wouldn't lose anything. Except of
course the main->cougar merge. Larry, please replicate that source tree
outside the client workspace and when the server is back up we'll
rebuild the changelist that got lost. Perforce support highly recommends
upgrading to the latest server version before playing back the
checkpoint since a) the new version is faster/better at the replay and
b) there are lots of bug fixes we really really want.=20

I need to come up with a game plan for reconstructing the server,
starting hopefully with comandeering a faster system to bring up in
parallel from John R. or something. Once I have a system and plan in
place I'll let people know and we can schedule the outage ( I'm assuming
sooner is far better than later...we don't want to continue very long
with this corruption in place ).

In the meantime you are probably okay if you just sync
//depot/cougar/nfx-tree/... and not the whole branch else you'll hit the
openbsd/... issue.

-Ken


-----Original Message-----
From: Larry Scheer
Sent: Wed 6/13/2007 3:03 PM
To: dl-Cougar; Ken Renshaw
Subject: Check-in of main-->cougar messed up.
=20
From what I can see it looks like the majority of the files in the
change list did not get submitted last night. When perforce went down it
lost the bulk of the submittal.

Larry

