X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C709F5.0065F876@onstor-exch02.onstor.net>; Thu, 16 Nov 2006 19:03:48 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C709F5.0065F876"
Content-class: urn:content-classes:message
Subject: RE: Important Clio Updates 
Date: Thu, 16 Nov 2006 19:03:48 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E9D9646@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E016DA688@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Important Clio Updates 
thread-index: AccJFf7/mkyKAmwOTPilIXBR13Zx8wAAFXuwAAFQemAAAQ5qUAAwSbTAAAHqBvAAAEoDQAAB3wXQAAB67pAAAGXDEA==
From: "Henry Lau" <henry.lau@onstor.com>
To: "Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>,
	"Paul Hammer" <paul.hammer@onstor.com>,
	"Vikas Saini" <vikas.saini@onstor.com>,
	"dl-Clio" <dl-Clio@onstor.com>,
	"dl-QA" <dl-qa@onstor.com>,
	"dl-hcl-qa" <dl-hcl-qa@onstor.com>
Cc: "dl-Engineering" <dl-engineering@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C709F5.0065F876
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

As far as I know, there are 2 recently defects for eek that filed
yesterday and today.=20
TED 16367 (I am looking on it now)
TED 16409 (watch dog timeout)

Thanks,

Henry

_____________________________________________
From: Jobi Ariyamannil=20
Sent: Thursday, November 16, 2006 7:01 PM
To: Paul Hammer; Vikas Saini; dl-Clio; dl-QA; dl-hcl-qa
Cc: dl-Engineering
Subject: RE: Important Clio Updates=20

Both variants of the eek will freeze the filesystem - online eek being
scheduled by the system, don't have to worry about manually starting it,
which most often can be overlooked.
Also in 2.1, we may not be able to trace corruptions to other areas like
quotas and stuff, unless we plan to add more people.

Debug features with minimum overhead are turned on by default.  Others
(checksum, read verify after write, shadow allocators, shadow buffers
etc) are off and needs to be enabled when needed.

So far, I heard one defect found by having online eek enabled in the
last build.  Raj reported a watch dog timeout during eek.  It is good
that we found that issue before customer found it.
This panic could have happened while running eek if a filesystem has
extremely big file/directory.

Also everybody now knows whether their filesystem is clean or corrupted
:-)

Regards,
Jobi

_____________________________________________
From: Paul Hammer=20
Sent: Thursday, November 16, 2006 6:49 PM
To: Jobi Ariyamannil; Vikas Saini; dl-Clio; dl-QA; dl-hcl-qa
Cc: dl-Engineering
Subject: RE: Important Clio Updates=20

Thanks Jobi.

I am okay with the online eek as long as it does not freeze the FS or
interrupt the tests, still think we have to run eek often since it seems
to be the superset.

We should currently be running with all of the FS dbg flags turned on,
are we?

QA, how did the eek efforts go? Are we all clean? Did we find any
corruptions? Did Eek fix them?

Thanks,

-Paul

_____________________________________________
From: Jobi Ariyamannil=20
Sent: Thursday, November 16, 2006 5:51 PM
To: Vikas Saini; dl-Clio; dl-QA; dl-hcl-qa
Cc: dl-Engineering
Subject: RE: Important Clio Updates=20

There is an old wiki page - http://wiki/wiki/FS/Testing.

The information is little outdated - Henry is going to update that soon.

All the rcc (runtime consistency checks) are available with debug builds
only.
Run "fscmd rcc" to see the debug features enabled on your filer.
If you had run "filesystem rcc", from nfxsh, I am afraid that you turned
off all of them.

Differences between online eek and manual eek:

*	Online eek runs automatically at specified intervals.
*	Online eek does consistency checks on blocks and inodes in the
live view only.  It does not check things like quotas, security ids,
hard links etc.
*	Online eek will assert if some inconsistency is found.  Manual
eek would have just printed a message.
*	Filesystem repair can be done only using manual eek.

Regards,
Jobi


_____________________________________________
From: Vikas Saini=20
Sent: Thursday, November 16, 2006 5:42 PM
To: Jobi Ariyamannil; dl-Clio; dl-QA; dl-hcl-qa
Cc: dl-Engineering
Subject: RE: Important Clio Updates=20

Thanks Jobi...yes if you can share all the info, it will be really
great.  We might need to have one session about differences between
online eek and other eek, side effects of online eek apart from
performance. Other things include what information we should collect
incase of corruption issue etc...

Also I just tried "filesystem rcc" command but didn't get any
information. We might need more detail in there.


Thanks
Vikas

_____________________________________________
From: Jobi Ariyamannil=20
Sent: Thursday, November 16, 2006 5:04 PM
To: Vikas Saini; dl-Clio; dl-QA; dl-hcl-qa
Cc: dl-Engineering
Subject: RE: Important Clio Updates=20

Hi,

Looks like a well thought plan.  Some comments:

We don't need any manual effort to run eek periodically on debug builds
with the so called "online eek".
Online eek can be tuned to be run at periodic intervals ("fscmd eek
<vol> -o <interval in minutes>").

Online eek does consistency checks on filesystem periodically and during
that time access to the filesystem is blocked.
Thus on a large filesystem with lots of files and/or lot of big
directories, this could be time consuming.
(we improved the performance though)

We need to get hold of the filesystem at the first opportunity after the
corruption happened, so that we can trace back the sequence of
operations happened causing the filesystem corruption.
That's the reasoning behind for adding online eek to run periodically
(in my test environment, I used to run this every 2 or 10 minutes).

It is a sad thing that we don't have any tools to capture the corrupt
state of the filesystem for future analysis.
At veritas, we used to have a tool called "metasave", which saves the
filesystem metadata and development engineer could analyze the problem
by restoring the same to another device.
Also our fsdb needs a lot more improvements.

One thing we can certainly do here is supporting volume exports/imports
between various networks in the lab, so that development engineer could
import the corrupted filesystem to his own filer and do the analysis
without holding the QA resources.

When the plan below says, no known filesystem corruptions, I would like
to relax that a bit.  We will try to address filesystem corruption
issues which affect the integrity of the base filesystem.
We don't have enough resources to analyze corruption issues in other
peripheral areas like quotas, security, hard links etc.

Also it is highly recommended to turn on various dynamic debugging
features in filesystem code.  They can be used to catch problems quicker
while testing debug builds.
The commands for doing so, are "fscmd rcc" and "filesystem rcc" (from
nfxsh).  Some of them are turned on by default (fscmd rcc will list
them).

I should send out another email with details of data needed to be
collected when a problem with filesystem code is reported.  Often times,
we need to ask for different things in many iterations.
If some procedure is followed, all of us could save a lot of time.  Also
filesystem team is heavily overloaded :-)

Also I will send out a list of changes we made in clio branch so far.

Regards,
Jobi



_____________________________________________
From: Vikas Saini=20
Sent: Wednesday, November 15, 2006 6:06 PM
To: dl-Clio; dl-QA; dl-hcl-qa
Cc: dl-Engineering
Subject: Important Clio Updates=20
Importance: High

Hi All,

In today's project meeting, it was collectively agreed that we will not
be shipping Clio with any known FS corruption problems. FileSystem team
has made a lot of changes which will prevent known corruptions as well
as detecting and then fixing FS corruption issues (i.e. eek
improvements). We don't have a solid/easy way to figure out what is
causing that corruption on first place. In Clio, we are going to take a
focused approach to help towards resolving this issue.=20

Starting ASAP, we are going to perform all of our QE testing on a known
good filesystem. To do that, we will be eeking all the volumes in QE
testing env 's, using the -r option (repair) .=20
Incase we find any filesystem which eek can't fix, we will keep those
volumes for Dev to look at and figure out why eek is not able to repair
those volumes. If Eek does find a problem we will run eek multiple time
until it reports everything is fine.

Once we have a known good base(all known corruptions have been removed
and eek has passed), we are going to continue running our test
regression. We will run eek in verify mode (using only the -v option)
periodically (with every submittal) to make sure that filesystem is not
corrupted.=20

As soon as we hit any potential File System issues e.g. corruption or
volume exceptions, there will be a combined effort to diagnose the
problem.  This will involve understanding the test scenario which led to
that corruption as well as collecting all the information that could
help in fixing that issue. We will keep on doing this to the point that
there are zero known filesystem problems.


QE Team:

Please eek all the volumes and repair them to make sure that it is a
good known base. Eek should be able to fix any corrupted filesystem and
if it doesn't file a defect and keep that filesystem aside for
development debugging.


It will be a good idea to create brand new volumes and enable snapshots
on them (it's disabled by default for new volumes), keep them small and
eek them often, every few hours at first, then let the interval grow.

Please take note of new volumes created, ones that passed eek (and when,
day and submittal), and any volume that has not been eeked or has not
passed eek.

Run eek in verbose mode every 3-4 days or every submittal to make sure
that all the filesystems are still in good shape incase we detect any
filesystem corruption problem which requires eek to be run to fix it,
file a defect, raise the flag so that development can immediately take a
look at that and collect all the info they need to resolve that issue.


Enable Volume exception corefile feature, Unexpected volume exception
can be one indication of FS metadata corruption. Please run eek in
verbose mode immediately if you get any unexpected volume exception.

To enable volume exception corefile feature, run command "system modify
volumeexceptiondump -m on". This will generate a core file whenever
there is a volume exception. That corefile gets copied to mgmt volume.

FS hardening and No known Filesystem corruption is the objective of Clio
and we are going to make sure that we achieve that when we release Clio
on December 18th.


Thanks
Vikas









------_=_NextPart_001_01C709F5.0065F876
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7650.28">
<TITLE>RE: Important Clio Updates </TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">As far as I know,</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial"> there are 2 recently defects for =
eek</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial"> that filed yesterday and =
today</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> </SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">TED 16367</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial"> =
(</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">I am look</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">ing on it now)</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">TED 16409</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial"> (watch =
dog timeout)</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Thanks,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Henry</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Jobi Ariyamannil<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, November 16, =
2006 7:01 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Paul Hammer; Vikas Saini; =
dl-Clio; dl-QA; dl-hcl-qa<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Engineering<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: Important Clio Updates</FONT></SPAN><SPAN =
LANG=3D"en-us"> </SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Both variants of the eek will freeze the filesystem =
&#8211; online eek being scheduled by the system, don&#8217;t have to =
worry about manually starting it, which most often can be =
overlooked.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Also in 2.1, we may not be able to trace corruptions to =
other areas like quotas and stuff, unless we plan to add more =
people.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Debug features with minimum overhead are turned on by =
default.&nbsp; Others (checksum, read verify after write, shadow =
allocators, shadow buffers etc) are off and needs to be enabled when =
needed.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">So far, I heard one defect found by having online eek =
enabled in the last build.&nbsp; Raj reported a watch dog timeout during =
eek.&nbsp; It is good that we found that issue before customer found =
it.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">This panic could have happened while running eek if a =
filesystem has extremely big file/directory.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Also everybody now knows whether their filesystem is =
clean or corrupted</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT FACE=3D"Wingdings" SIZE=3D2>J</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Regards,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jobi</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Paul Hammer<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, November 16, =
2006 6:49 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Jobi Ariyamannil; Vikas =
Saini; dl-Clio; dl-QA; dl-hcl-qa<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Engineering<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: Important Clio Updates</FONT></SPAN><SPAN =
LANG=3D"en-us"> </SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Thanks Jobi.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">I am okay with the online eek as long as it does not =
freeze the FS or interrupt the tests, still think we have to run eek =
often since it seems to be the superset.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">We should currently be running with all of the FS dbg =
flags turned on, are we?</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">QA, how did the eek efforts go? Are we all clean? Did we =
find any corruptions? Did Eek fix them?</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Thanks,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">-Paul</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Jobi Ariyamannil<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, November 16, =
2006 5:51 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Vikas Saini; dl-Clio; =
dl-QA; dl-hcl-qa<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Engineering<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: Important Clio Updates</FONT></SPAN><SPAN =
LANG=3D"en-us"> </SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">There is an old wiki page -</FONT></SPAN><SPAN =
LANG=3D"en-us"> </SPAN><A HREF=3D"http://wiki/wiki/FS/Testing"><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><U><FONT COLOR=3D"#0000FF" =
SIZE=3D2 =
FACE=3D"Arial">http://wiki/wiki/FS/Testing</FONT></U></SPAN><SPAN =
LANG=3D"en-us"></SPAN></A><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">The information is little outdated &#8211; Henry is going =
to update that soon.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">All the rcc (runtime consistency checks) are available =
with debug builds only.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Run &#8220;fscmd rcc&#8221; to see the debug features =
enabled on your filer.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">If you had run &#8220;filesystem rcc&#8221;, from nfxsh, =
I am afraid that you turned off all of them.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Differences between online eek and manual =
eek:</FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Symbol">&#183;<FONT FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT></FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">Online eek runs automatically at specified =
intervals.</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Symbol">&#183;<FONT FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT></FONT> <FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Online eek does consistency =
checks on blocks and inodes in the live view only.&nbsp; It does not =
check things like quotas, security ids, hard links =
etc.</FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Symbol">&#183;<FONT FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT></FONT> <FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Online eek will assert if some =
inconsistency is found.&nbsp; Manual eek would have just printed a =
message.</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Symbol">&#183;<FONT FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT></FONT> <FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Filesystem repair can be done =
only using manual eek.</FONT></SPAN>
</P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Regards,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jobi</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Vikas Saini<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, November 16, =
2006 5:42 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Jobi Ariyamannil; =
dl-Clio; dl-QA; dl-hcl-qa<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Engineering<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: Important Clio Updates</FONT></SPAN><SPAN =
LANG=3D"en-us"> </SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Thanks Jobi&#8230;yes if you can share all the info, it =
will be really great.&nbsp; We might need to have one session about =
differences between online eek and other eek, side effects of online eek =
apart from performance. Other things include what information we should =
collect incase of corruption issue etc&#8230;</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Also I just tried &#8220;filesystem rcc&#8221; command =
but didn&#8217;t get any information. We might need more detail in =
there.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Thanks</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Vikas</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Jobi Ariyamannil<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, November 16, =
2006 5:04 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Vikas Saini; dl-Clio; =
dl-QA; dl-hcl-qa<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Engineering<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: Important Clio Updates</FONT></SPAN><SPAN =
LANG=3D"en-us"> </SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Hi,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Looks like a well thought plan.&nbsp; Some =
comments:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">We don&#8217;t need any manual effort to run eek =
periodically on debug builds with the so called &#8220;online =
eek&#8221;.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Online eek can be tuned to be run at periodic intervals =
(&#8220;fscmd eek &lt;vol&gt; -o &lt;interval in =
minutes&gt;&#8221;).</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Online eek does consistency checks on filesystem =
periodically and during that time access to the filesystem is =
blocked.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Thus on a large filesystem with lots of files and/or lot =
of big directories, this could be time consuming.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">(we improved the performance though)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">We need to get hold of the filesystem at the first =
opportunity after the corruption happened, so that we can trace back the =
sequence of operations happened causing the filesystem =
corruption.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">That&#8217;s the reasoning behind for adding online eek =
to run periodically (in my test environment, I used to run this every 2 =
or 10 minutes).</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">It is a sad thing that we don&#8217;t have any tools to =
capture the corrupt state of the filesystem for future =
analysis.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">At veritas, we used to have a tool called =
&#8220;metasave&#8221;, which saves the filesystem metadata and =
development engineer could analyze the problem by restoring the same to =
another device.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Also our fsdb needs a lot more =
improvements.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">One thing we can certainly do here is supporting volume =
exports/imports between various networks in the lab, so that development =
engineer could import the corrupted filesystem to his own filer and do =
the analysis without holding the QA resources.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">When the plan below says, no known filesystem =
corruptions, I would like to relax that a bit.&nbsp; We will try to =
address filesystem corruption issues which affect the integrity of the =
base filesystem.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">We don&#8217;t have enough resources to analyze =
corruption issues in other peripheral areas like quotas, security, hard =
links etc.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Also it is highly recommended to turn on various dynamic =
debugging features in filesystem code.&nbsp; They can be used to catch =
problems quicker while testing debug builds.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">The commands for doing so, are &#8220;fscmd rcc&#8221; =
and &#8220;filesystem rcc&#8221; (from nfxsh).&nbsp; Some of them are =
turned on by default (fscmd rcc will list them).</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">I should send out another email with details of data =
needed to be collected when a problem with filesystem code is =
reported.&nbsp; Often times, we need to ask for different things in many =
iterations.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">If some procedure is followed, all of us could save a lot =
of time.&nbsp; Also filesystem team is heavily =
overloaded</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT FACE=3D"Wingdings" SIZE=3D2>J</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Also I will send out a list of changes we made in clio =
branch so far.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Regards,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jobi</FONT></SPAN></P>
<BR>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Vikas Saini<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Wednesday, November 15, =
2006 6:06 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Clio; dl-QA; =
dl-hcl-qa<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Engineering<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Important Clio Updates<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Importance:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> High</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us">Hi All,</SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">In =
today&#8217;s project meeting, it was collectively agreed that we will =
not be shipping Clio with any known FS corruption problems. FileSystem =
team has made a lot of changes which will prevent known corruptions as =
well as detecting and then fixing FS corruption issues (i.e. eek =
improvements). We don&#8217;t have a solid/easy way to figure out what =
is causing that corruption on first place. In Clio, we are going to take =
a focused approach to help towards resolving this issue. =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting ASAP, we are going to perform all of our QE =
testing on a known good filesystem. To do that, we will be eeking all =
the volumes in QE testing env &#8216;s, using the &#8211;r option =
(repair) . </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Incase we find any filesystem which eek can&#8217;t fix, =
we will keep those volumes for Dev to look at and figure out why eek is =
not able to repair those volumes. If Eek does find a problem we will run =
eek multiple time until it reports everything is fine.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Once =
we have a known good base(all known corruptions have been removed and =
eek has passed), we are going to continue running our test regression. =
We will run eek in verify mode (using only the &#8211;v option) =
periodically (with every submittal) to make sure that filesystem is not =
corrupted. </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">As =
soon as we hit any potential File System issues e.g. corruption or =
volume exceptions, there will be a combined effort to diagnose the =
problem.&nbsp; This will involve understanding the test scenario which =
led to that corruption as well as collecting all the information that =
could help in fixing that issue. We will keep on doing this to the point =
that there are zero known filesystem problems.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">QE =
Team:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Please eek all the volumes and repair them to make sure =
that it is a good known base. Eek should be able to fix any corrupted =
filesystem and if it doesn&#8217;t file a defect and keep that =
filesystem aside for development debugging.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">It =
will be a good idea to create brand new volumes and enable snapshots on =
them (it&#8217;s disabled by default for new volumes), keep them small =
and eek them often, every few hours at first, then let the interval =
grow.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Please take note of new volumes created, ones that passed =
eek (and when, day and submittal), and any volume that has not been =
eeked or has not passed eek.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Run =
eek in verbose mode every 3-4 days or every submittal to make sure that =
all the filesystems are still in good shape incase we detect any =
filesystem corruption problem which requires eek to be run to fix it, =
file a defect, raise the flag so that development can immediately take a =
look at that and collect all the info they need to resolve that =
issue.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Enable Volume exception corefile feature, Unexpected =
volume exception can be one indication of FS metadata corruption. Please =
run eek in verbose mode immediately if you get any unexpected volume =
exception.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">To =
enable volume exception corefile feature, run command &#8220;system =
modify volumeexceptiondump -m on&#8221;. This will generate a core file =
whenever there is a volume exception. That corefile gets copied to mgmt =
volume.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">FS =
hardening and No known Filesystem corruption is the objective of Clio =
and we are going to make sure that we achieve that when we release Clio =
on December 18</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><SUP><FONT SIZE=3D2 =
FACE=3D"Arial">th</FONT></SUP></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Thanks</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Vikas</FONT></SPAN></P>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>

</BODY>
</HTML>
------_=_NextPart_001_01C709F5.0065F876--
