X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C71388.30C81623@onstor-exch02.onstor.net>; Tue, 28 Nov 2006 23:30:05 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C71388.30C81623"
Content-class: urn:content-classes:message
Subject: RE: quick Clio update
Date: Tue, 28 Nov 2006 23:30:05 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E013D25DB@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: quick Clio update
thread-index: AccTYoNq8hbp8v5rSpWC2ZzQHHvkQAABB+7yAAahHsAAAY7zIA==
References: <BB375AF679D4A34E9CA8DFA650E2B04E01899480@onstor-exch02.onstor.net>
From: "Paul Hammer" <paul.hammer@onstor.com>
To: "Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>,
	"Sandrine Boulanger" <sandrine.boulanger@onstor.com>,
	"Vikas Saini" <vikas.saini@onstor.com>,
	"dl-Clio" <dl-Clio@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C71388.30C81623
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Cool, all good inputs, thanks.=20
=20
Believe these eek runs were started before we moved to opt builds but =
not sure. Vikas?=20
=20
Not sure we can chalk up all of the time to Quotas though, hence we were =
trying to make sure the volumes were clean independent of the =
configurations so that we could continue to test post all the eek/fs =
corrections that are now in the release.
=20
-Paul.
=20
=20

________________________________

From: Jobi Ariyamannil
Sent: Tue 11/28/2006 10:47 PM
To: Sandrine Boulanger; Vikas Saini; dl-Clio
Subject: RE: quick Clio update



EEK is very slow processing quota records and QA have been reporting EEK =
running for long time when filesystems have large number of quota =
records.
On the other hand, I have not seen any customer reporting the same issue =
so far.

Are you running EEK with debug or opt build ? We should run EEK on such =
volumes with OPT builds only which may be faster than debug builds.
Also please check the output of "fscmd cachestats" to see whether we are =
bottlenecking on the caches.  If so, we can tune that and make eek =
finish faster.

Also I checked in a change (p4 change 22002) today which will add read =
ahead while eek processing quota records.  It would be good to check =
whether that improves the situation or not.

In any case, it is not good to hold the QA resources for 8 days (and =
more) just to verify quota records which from past experiences never =
reported any corruptions.  We can even provide an eek which skip that =
phase for QA testing.  (Also all the snapshots can be removed offline by =
passing "-c" option to eek).

Please let me any comments on this.

Regards,
Jobi

-----Original Message-----
From: Sandrine Boulanger
Sent: Tuesday, November 28, 2006 7:30 PM
To: Vikas Saini; dl-Clio
Subject: RE: quick Clio update

Cheetah soak has been running some traffic since last wednesday on the =
filer that is not running eek. We'll send out a report tomorrow.


-----Original Message-----
From: Vikas Saini
Sent: Tue 11/28/2006 7:00 PM
To: dl-Clio
Subject: quick Clio update

Defects

MF in Dev 26

MF in QA for Clio       38
NMF in QA for Clio      13
All in QA for Clio      51
All in QA               134

We have run eek on all the filers which are being used for Clio testing. =
Volumes which were part of Bobcat Soak and Cheetah Soak. We are still =
running eek on them. Eek is taking a lot of time on those volumes. Eek =
on a bobcat soak volume is running since last 8 days and has not =
completed yet. Below is the output for that eek. We have a similar =
volume on cheetah soak on which eek in running since last 6 days.


# fscmd eek eng49-3 lvolOnline -r -f
running eek in repair mode
Force mode: corrupt snapshots will be automatically removed.
volId 0x17a000000d0 is not online, state is 1
eek: trying to bring the logical volume online
Done.
Mon Nov 20 09:22:21 2006 =3D=3D eng49-3 =3D=3D verifying meta inodes =
=3D=3D
eng49-3: Inode (I#1) file invalid numblocks 3036141 in superblock, =
expected 3036142
Mon Nov 20 09:22:35 2006 =3D=3D eng49-3 =3D=3D mark and compare =
reference counts =3D=3D
Mon Nov 20 09:23:08 2006 =3D=3D eng49-3 =3D=3D mark used blocks =3D=3D
eng49-3: Inode (I#1) file invalid numblocks 3036141 in superblock, =
expected 3036142
processed 194314240 of 194314240 inodes

Mon Nov 20 10:20:38 2006 =3D=3D eng49-3 =3D=3D compare used blocks =
=3D=3D
processed 105574336 of 105574336 blocks

Mon Nov 20 11:31:12 2006 =3D=3D eng49-3 =3D=3D processing quota trees =
=3D=3D
processed 1 quota tree records
Mon Nov 20 11:31:12 2006 =3D=3D eng49-3 =3D=3D mark used inodes =3D=3D
proceprocessed 194314240 of 194314240 inodes, 16441522 of 16441522 =
directories

Tue Nov 21 01:42:55 2006 =3D=3D eng49-3 =3D=3D detecting unused quota =
trees =3D=3D
Tue Nov 21 01:42:55 2006 =3D=3D eng49-3 =3D=3D processing user quotas =
=3D=3D
processed 1220544 user quota records
processed 2466048 user quota records
processed 2671032 user quota records


On testing side from schedule point of view, we are couple of days =
behind but will catch up this week.

Soak env is not fully operational yet. Bobcat soak and cheetah soak are =
running eek and super soak is still in setup process.

More detailed Clio update will be sent in a day or two.

Thanks
Vikas





------_=_NextPart_001_01C71388.30C81623
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<HTML dir=3Dltr><HEAD><TITLE>RE: quick Clio update</TITLE>=0A=
<META http-equiv=3DContent-Type content=3D"text/html; charset=3Dunicode">=0A=
<META content=3D"MSHTML 6.00.2900.2995" name=3DGENERATOR></HEAD>=0A=
<BODY>=0A=
<DIV id=3DidOWAReplyText27844 dir=3Dltr>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 size=3D2>Cool, all =
good inputs, thanks. </FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 =
size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 size=3D2>Believe these =
eek runs were started before we moved to opt builds but not sure. Vikas? =
</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 =
size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 size=3D2>Not sure we =
can chalk up all of the time to Quotas though, hence we were trying to =
make sure the volumes were clean independent of the configurations so =
that we could continue to test post all the eek/fs corrections that are =
now in the release.</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 =
size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 =
size=3D2>-Paul.</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV></DIV>=0A=
<DIV dir=3Dltr><BR>=0A=
<HR tabIndex=3D-1>=0A=
<FONT face=3DTahoma size=3D2><B>From:</B> Jobi =
Ariyamannil<BR><B>Sent:</B> Tue 11/28/2006 10:47 PM<BR><B>To:</B> =
Sandrine Boulanger; Vikas Saini; dl-Clio<BR><B>Subject:</B> RE: quick =
Clio update<BR></FONT><BR></DIV>=0A=
<DIV>=0A=
<P><FONT size=3D2>EEK is very slow processing quota records and QA have =
been reporting EEK running for long time when filesystems have large =
number of quota records.<BR>On the other hand, I have not seen any =
customer reporting the same issue so far.<BR><BR>Are you running EEK =
with debug or opt build ? We should run EEK on such volumes with OPT =
builds only which may be faster than debug builds.<BR>Also please check =
the output of "fscmd cachestats" to see whether we are bottlenecking on =
the caches.&nbsp; If so, we can tune that and make eek finish =
faster.<BR><BR>Also I checked in a change (p4 change 22002) today which =
will add read ahead while eek processing quota records.&nbsp; It would =
be good to check whether that improves the situation or not.<BR><BR>In =
any case, it is not good to hold the QA resources for 8 days (and more) =
just to verify quota records which from past experiences never reported =
any corruptions.&nbsp; We can even provide an eek which skip that phase =
for QA testing.&nbsp; (Also all the snapshots can be removed offline by =
passing "-c" option to eek).<BR><BR>Please let me any comments on =
this.<BR><BR>Regards,<BR>Jobi<BR><BR>-----Original Message-----<BR>From: =
Sandrine Boulanger<BR>Sent: Tuesday, November 28, 2006 7:30 PM<BR>To: =
Vikas Saini; dl-Clio<BR>Subject: RE: quick Clio update<BR><BR>Cheetah =
soak has been running some traffic since last wednesday on the filer =
that is not running eek. We'll send out a report =
tomorrow.<BR><BR><BR>-----Original Message-----<BR>From: Vikas =
Saini<BR>Sent: Tue 11/28/2006 7:00 PM<BR>To: dl-Clio<BR>Subject: quick =
Clio update<BR><BR>Defects<BR><BR>MF in Dev 26<BR><BR>MF in QA for =
Clio&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 38<BR>NMF in QA for =
Clio&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 13<BR>All in QA for =
Clio&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 51<BR>All in =
QA&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 134<BR><BR>We have run eek on =
all the filers which are being used for Clio testing. Volumes which were =
part of Bobcat Soak and Cheetah Soak. We are still running eek on them. =
Eek is taking a lot of time on those volumes. Eek on a bobcat soak =
volume is running since last 8 days and has not completed yet. Below is =
the output for that eek. We have a similar volume on cheetah soak on =
which eek in running since last 6 days.<BR><BR><BR># fscmd eek eng49-3 =
lvolOnline -r -f<BR>running eek in repair mode<BR>Force mode: corrupt =
snapshots will be automatically removed.<BR>volId 0x17a000000d0 is not =
online, state is 1<BR>eek: trying to bring the logical volume =
online<BR>Done.<BR>Mon Nov 20 09:22:21 2006 =3D=3D eng49-3 =3D=3D =
verifying meta inodes =3D=3D<BR>eng49-3: Inode (I#1) file invalid =
numblocks 3036141 in superblock, expected 3036142<BR>Mon Nov 20 09:22:35 =
2006 =3D=3D eng49-3 =3D=3D mark and compare reference counts =
=3D=3D<BR>Mon Nov 20 09:23:08 2006 =3D=3D eng49-3 =3D=3D mark used =
blocks =3D=3D<BR>eng49-3: Inode (I#1) file invalid numblocks 3036141 in =
superblock, expected 3036142<BR>processed 194314240 of 194314240 =
inodes<BR><BR>Mon Nov 20 10:20:38 2006 =3D=3D eng49-3 =3D=3D compare =
used blocks =3D=3D<BR>processed 105574336 of 105574336 blocks<BR><BR>Mon =
Nov 20 11:31:12 2006 =3D=3D eng49-3 =3D=3D processing quota trees =
=3D=3D<BR>processed 1 quota tree records<BR>Mon Nov 20 11:31:12 2006 =
=3D=3D eng49-3 =3D=3D mark used inodes =3D=3D<BR>proceprocessed =
194314240 of 194314240 inodes, 16441522 of 16441522 =
directories<BR><BR>Tue Nov 21 01:42:55 2006 =3D=3D eng49-3 =3D=3D =
detecting unused quota trees =3D=3D<BR>Tue Nov 21 01:42:55 2006 =3D=3D =
eng49-3 =3D=3D processing user quotas =3D=3D<BR>processed 1220544 user =
quota records<BR>processed 2466048 user quota records<BR>processed =
2671032 user quota records<BR><BR><BR>On testing side from schedule =
point of view, we are couple of days behind but will catch up this =
week.<BR><BR>Soak env is not fully operational yet. Bobcat soak and =
cheetah soak are running eek and super soak is still in setup =
process.<BR><BR>More detailed Clio update will be sent in a day or =
two.<BR><BR>Thanks<BR>Vikas<BR><BR><BR></FONT></P></DIV></BODY></HTML>
------_=_NextPart_001_01C71388.30C81623--
