AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:
CFG:
PT:0
S:andy.sharp@lsi.com
RQ:
SSV:mhbs.lsil.com
NSV:
SSH:
R:<Larry.Scheer@lsi.com>
MAID:2
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/LSI/INBOX	0	DEC609CD0E54B2448DAF023C89AE9755EB50C502@cosmail02.lsi.com
X-Sylpheed-End-Special-Headers: 1
Date: Wed, 24 Mar 2010 14:47:06 -0700
From: Andrew Sharp <andy.sharp@lsi.com>
To: "Scheer, Larry" <Larry.Scheer@lsi.com>
Subject: Re: K3 fsck completed /perforce back on line
Message-ID: <20100324144706.0e21a011@ripper.onstor.net>
In-Reply-To: <DEC609CD0E54B2448DAF023C89AE9755EB50C502@cosmail02.lsi.com>
References: <DEC609CD0E54B2448DAF023C89AE9755EB50C500@cosmail02.lsi.com>
	<20100324130023.3344176b@ripper.onstor.net>
	<DEC609CD0E54B2448DAF023C89AE9755EB50C502@cosmail02.lsi.com>
Organization: LSI
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

On Wed, 24 Mar 2010 14:48:46 -0600 "Scheer, Larry"
<Larry.Scheer@lsi.com> wrote:

> Well Ken but some stuff there that I am wondering what the heck it is
> such as /perforce/drivers. But since his departure Ed and I are the
> only people to use this volume. For us it is purely a build volume.
> Source is kept in perforce and the results of the builds are rsync'd
> to /n/Build-Trees and the upgrade server:
> 10.3.0.222:/tftpstor/ftp/upgrade
> 
> Essentially, all the important stuff is on our filers and in the
> perforce depot.

OK, nevermind that stuff about raid then.  I started to think there was
something seriously important on there, but then I *actually* started
to think.

> Larry
> ________________________________________
> From: Andrew Sharp [andy.sharp@lsi.com]
> Sent: Wednesday, March 24, 2010 1:00 PM
> To: Scheer, Larry
> Subject: Re: K3 fsck completed /perforce back on line
> 
> Yup.  We need these volumes to be on raid-5 at least, raid-1 best.
> How can we go about fixing that?  Or should we put them on MD?
> 
> I'm slightly unclear on what perforce data would be on k3.
> 
> 
> On Wed, 24 Mar 2010 13:52:08 -0600 "Scheer, Larry"
> <Larry.Scheer@lsi.com> wrote:
> 
> > I ran the fsck on /perforce. Fortunately the only files moved to
> > lost+found were all mipsel binaries generated by builds.
> >
> > /perforce is back.
> >
> > You can do your build now Ed.
> >
> > Andy, here are the errors from the log. We may have degrading media?
> >
> > The following SCSI errors were in the message log:
> > Mar 24 11:03:01 k3 kernel: SCSI error : <0 2 0 0> return code =
> > 0x70000 Mar 24 11:03:01 k3 kernel: end_request: I/O error, dev sda,
> > sector 843645236 Mar 24 11:03:01 k3 kernel: Buffer I/O error on
> > device dm-5, logical block 105480342 Mar 24 11:03:11 k3 kernel: SCSI
> > error : <0 2 0 0> return code = 0x70000 Mar 24 11:03:11 k3 kernel:
> > end_request: I/O error, dev sda, sector 843645236 Mar 24 11:03:11 k3
> > kernel: Buffer I/O error on device dm-5, logical block 105480342 Mar
> > 24 11:11:49 k3 kernel: SCSI error : <0 2 0 0> return code = 0x70000
> > Mar 24 11:11:49 k3 kernel: end_request: I/O error, dev sda, sector
> > 843890548 Mar 24 11:11:49 k3 kernel: Buffer I/O error on device
> > dm-5, logical block 105511006
> >
> > Dmesg shows these:
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543308
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 851543300
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 843890548
> > EXT3-fs error (device dm-5): ext3_readdir: directory #52744544
> > contains a hole at offset 0 Aborting journal on device dm-5.
> > ext3_abort called.
> > EXT3-fs error (device dm-5): ext3_journal_start_sb: Detected aborted
> > journal Remounting filesystem read-only
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 843645236
> > Buffer I/O error on device dm-5, logical block 105480342
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 843645236
> > Buffer I/O error on device dm-5, logical block 105480342
> > SCSI error : <0 2 0 0> return code = 0x70000
> > end_request: I/O error, dev sda, sector 843890548
> > Buffer I/O error on device dm-5, logical block 105511006
> > kjournald starting.  Commit interval 5 seconds
> > EXT3 FS on dm-5, internal journal
> > EXT3-fs: mounted filesystem with ordered data mode.
> > SELinux: initialized (dev dm-5, type ext3), uses xattr
