X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C709EA.C29941FD@onstor-exch02.onstor.net>; Thu, 16 Nov 2006 17:50:29 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C709EA.C29941FD"
Content-class: urn:content-classes:message
Subject: RE: Important Clio Updates 
Date: Thu, 16 Nov 2006 17:50:29 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E016DA615@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E03D481@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Important Clio Updates 
thread-index: AccJFf7/mkyKAmwOTPilIXBR13Zx8wAAFXuwAAFQemAAAQ5qUAAwSbTAAAHqBvAAAID9sA==
From: "Tim Gardner" <tim.gardner@onstor.com>
To: "Vikas Saini" <vikas.saini@onstor.com>,
	"Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>,
	"dl-Clio" <dl-Clio@onstor.com>,
	"dl-QA" <dl-qa@onstor.com>,
	"dl-hcl-qa" <dl-hcl-qa@onstor.com>
Cc: "dl-Engineering" <dl-engineering@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C709EA.C29941FD
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Henry is updating the wiki with this info. If there is something missing
from the wiki that you need,
please let Henry know.

Tim


_____________________________________________
From: Vikas Saini=20
Sent: Thursday, November 16, 2006 5:42 PM
To: Jobi Ariyamannil; dl-Clio; dl-QA; dl-hcl-qa
Cc: dl-Engineering
Subject: RE: Important Clio Updates=20

Thanks Jobi...yes if you can share all the info, it will be really
great.  We might need to have one session about differences between
online eek and other eek, side effects of online eek apart from
performance. Other things include what information we should collect
incase of corruption issue etc...

Also I just tried "filesystem rcc" command but didn't get any
information. We might need more detail in there.


Thanks
Vikas

_____________________________________________
From: Jobi Ariyamannil=20
Sent: Thursday, November 16, 2006 5:04 PM
To: Vikas Saini; dl-Clio; dl-QA; dl-hcl-qa
Cc: dl-Engineering
Subject: RE: Important Clio Updates=20

Hi,

Looks like a well thought plan.  Some comments:

We don't need any manual effort to run eek periodically on debug builds
with the so called "online eek".
Online eek can be tuned to be run at periodic intervals ("fscmd eek
<vol> -o <interval in minutes>").

Online eek does consistency checks on filesystem periodically and during
that time access to the filesystem is blocked.
Thus on a large filesystem with lots of files and/or lot of big
directories, this could be time consuming.
(we improved the performance though)

We need to get hold of the filesystem at the first opportunity after the
corruption happened, so that we can trace back the sequence of
operations happened causing the filesystem corruption.
That's the reasoning behind for adding online eek to run periodically
(in my test environment, I used to run this every 2 or 10 minutes).

It is a sad thing that we don't have any tools to capture the corrupt
state of the filesystem for future analysis.
At veritas, we used to have a tool called "metasave", which saves the
filesystem metadata and development engineer could analyze the problem
by restoring the same to another device.
Also our fsdb needs a lot more improvements.

One thing we can certainly do here is supporting volume exports/imports
between various networks in the lab, so that development engineer could
import the corrupted filesystem to his own filer and do the analysis
without holding the QA resources.

When the plan below says, no known filesystem corruptions, I would like
to relax that a bit.  We will try to address filesystem corruption
issues which affect the integrity of the base filesystem.
We don't have enough resources to analyze corruption issues in other
peripheral areas like quotas, security, hard links etc.

Also it is highly recommended to turn on various dynamic debugging
features in filesystem code.  They can be used to catch problems quicker
while testing debug builds.
The commands for doing so, are "fscmd rcc" and "filesystem rcc" (from
nfxsh).  Some of them are turned on by default (fscmd rcc will list
them).

I should send out another email with details of data needed to be
collected when a problem with filesystem code is reported.  Often times,
we need to ask for different things in many iterations.
If some procedure is followed, all of us could save a lot of time.  Also
filesystem team is heavily overloaded :-)

Also I will send out a list of changes we made in clio branch so far.

Regards,
Jobi



_____________________________________________
From: Vikas Saini=20
Sent: Wednesday, November 15, 2006 6:06 PM
To: dl-Clio; dl-QA; dl-hcl-qa
Cc: dl-Engineering
Subject: Important Clio Updates=20
Importance: High

Hi All,

In today's project meeting, it was collectively agreed that we will not
be shipping Clio with any known FS corruption problems. FileSystem team
has made a lot of changes which will prevent known corruptions as well
as detecting and then fixing FS corruption issues (i.e. eek
improvements). We don't have a solid/easy way to figure out what is
causing that corruption on first place. In Clio, we are going to take a
focused approach to help towards resolving this issue.=20

Starting ASAP, we are going to perform all of our QE testing on a known
good filesystem. To do that, we will be eeking all the volumes in QE
testing env 's, using the -r option (repair) .=20
Incase we find any filesystem which eek can't fix, we will keep those
volumes for Dev to look at and figure out why eek is not able to repair
those volumes. If Eek does find a problem we will run eek multiple time
until it reports everything is fine.

Once we have a known good base(all known corruptions have been removed
and eek has passed), we are going to continue running our test
regression. We will run eek in verify mode (using only the -v option)
periodically (with every submittal) to make sure that filesystem is not
corrupted.=20

As soon as we hit any potential File System issues e.g. corruption or
volume exceptions, there will be a combined effort to diagnose the
problem.  This will involve understanding the test scenario which led to
that corruption as well as collecting all the information that could
help in fixing that issue. We will keep on doing this to the point that
there are zero known filesystem problems.


QE Team:

Please eek all the volumes and repair them to make sure that it is a
good known base. Eek should be able to fix any corrupted filesystem and
if it doesn't file a defect and keep that filesystem aside for
development debugging.


It will be a good idea to create brand new volumes and enable snapshots
on them (it's disabled by default for new volumes), keep them small and
eek them often, every few hours at first, then let the interval grow.

Please take note of new volumes created, ones that passed eek (and when,
day and submittal), and any volume that has not been eeked or has not
passed eek.

Run eek in verbose mode every 3-4 days or every submittal to make sure
that all the filesystems are still in good shape incase we detect any
filesystem corruption problem which requires eek to be run to fix it,
file a defect, raise the flag so that development can immediately take a
look at that and collect all the info they need to resolve that issue.


Enable Volume exception corefile feature, Unexpected volume exception
can be one indication of FS metadata corruption. Please run eek in
verbose mode immediately if you get any unexpected volume exception.

To enable volume exception corefile feature, run command "system modify
volumeexceptiondump -m on". This will generate a core file whenever
there is a volume exception. That corefile gets copied to mgmt volume.

FS hardening and No known Filesystem corruption is the objective of Clio
and we are going to make sure that we achieve that when we release Clio
on December 18th.


Thanks
Vikas









------_=_NextPart_001_01C709EA.C29941FD
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7650.28">
<TITLE>RE: Important Clio Updates </TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Henry is updating the wiki with this info. If there is =
something missing from the wiki that you need,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">please let Henry know.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Tim</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Vikas Saini<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, November 16, =
2006 5:42 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Jobi Ariyamannil; =
dl-Clio; dl-QA; dl-hcl-qa<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Engineering<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: Important Clio Updates</FONT></SPAN><SPAN =
LANG=3D"en-us"> </SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Thanks Jobi&#8230;yes if you can share all the info, it =
will be really great.&nbsp; We might need to have one session about =
differences between online eek and other eek, side effects of online eek =
apart from performance. Other things include what information we should =
collect incase of corruption issue etc&#8230;</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Also I just tried &#8220;filesystem rcc&#8221; command =
but didn&#8217;t get any information. We might need more detail in =
there.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Thanks</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Vikas</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Jobi Ariyamannil<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, November 16, =
2006 5:04 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Vikas Saini; dl-Clio; =
dl-QA; dl-hcl-qa<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Engineering<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: Important Clio Updates</FONT></SPAN><SPAN =
LANG=3D"en-us"> </SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Hi,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Looks like a well thought plan.&nbsp; Some =
comments:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">We don&#8217;t need any manual effort to run eek =
periodically on debug builds with the so called &#8220;online =
eek&#8221;.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Online eek can be tuned to be run at periodic intervals =
(&#8220;fscmd eek &lt;vol&gt; -o &lt;interval in =
minutes&gt;&#8221;).</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Online eek does consistency checks on filesystem =
periodically and during that time access to the filesystem is =
blocked.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Thus on a large filesystem with lots of files and/or lot =
of big directories, this could be time consuming.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">(we improved the performance though)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">We need to get hold of the filesystem at the first =
opportunity after the corruption happened, so that we can trace back the =
sequence of operations happened causing the filesystem =
corruption.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">That&#8217;s the reasoning behind for adding online eek =
to run periodically (in my test environment, I used to run this every 2 =
or 10 minutes).</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">It is a sad thing that we don&#8217;t have any tools to =
capture the corrupt state of the filesystem for future =
analysis.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">At veritas, we used to have a tool called =
&#8220;metasave&#8221;, which saves the filesystem metadata and =
development engineer could analyze the problem by restoring the same to =
another device.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Also our fsdb needs a lot more =
improvements.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">One thing we can certainly do here is supporting volume =
exports/imports between various networks in the lab, so that development =
engineer could import the corrupted filesystem to his own filer and do =
the analysis without holding the QA resources.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">When the plan below says, no known filesystem =
corruptions, I would like to relax that a bit.&nbsp; We will try to =
address filesystem corruption issues which affect the integrity of the =
base filesystem.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">We don&#8217;t have enough resources to analyze =
corruption issues in other peripheral areas like quotas, security, hard =
links etc.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Also it is highly recommended to turn on various dynamic =
debugging features in filesystem code.&nbsp; They can be used to catch =
problems quicker while testing debug builds.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">The commands for doing so, are &#8220;fscmd rcc&#8221; =
and &#8220;filesystem rcc&#8221; (from nfxsh).&nbsp; Some of them are =
turned on by default (fscmd rcc will list them).</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">I should send out another email with details of data =
needed to be collected when a problem with filesystem code is =
reported.&nbsp; Often times, we need to ask for different things in many =
iterations.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">If some procedure is followed, all of us could save a lot =
of time.&nbsp; Also filesystem team is heavily =
overloaded</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT FACE=3D"Wingdings" SIZE=3D2>J</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Also I will send out a list of changes we made in clio =
branch so far.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Regards,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jobi</FONT></SPAN></P>
<BR>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Vikas Saini<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Wednesday, November 15, =
2006 6:06 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Clio; dl-QA; =
dl-hcl-qa<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Engineering<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Important Clio Updates<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Importance:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> High</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us">Hi All,</SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">In =
today&#8217;s project meeting, it was collectively agreed that we will =
not be shipping Clio with any known FS corruption problems. FileSystem =
team has made a lot of changes which will prevent known corruptions as =
well as detecting and then fixing FS corruption issues (i.e. eek =
improvements). We don&#8217;t have a solid/easy way to figure out what =
is causing that corruption on first place. In Clio, we are going to take =
a focused approach to help towards resolving this issue. =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting ASAP, we are going to perform all of our QE =
testing on a known good filesystem. To do that, we will be eeking all =
the volumes in QE testing env &#8216;s, using the &#8211;r option =
(repair) . </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Incase we find any filesystem which eek can&#8217;t fix, =
we will keep those volumes for Dev to look at and figure out why eek is =
not able to repair those volumes. If Eek does find a problem we will run =
eek multiple time until it reports everything is fine.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Once =
we have a known good base(all known corruptions have been removed and =
eek has passed), we are going to continue running our test regression. =
We will run eek in verify mode (using only the &#8211;v option) =
periodically (with every submittal) to make sure that filesystem is not =
corrupted. </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">As =
soon as we hit any potential File System issues e.g. corruption or =
volume exceptions, there will be a combined effort to diagnose the =
problem.&nbsp; This will involve understanding the test scenario which =
led to that corruption as well as collecting all the information that =
could help in fixing that issue. We will keep on doing this to the point =
that there are zero known filesystem problems.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">QE =
Team:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Please eek all the volumes and repair them to make sure =
that it is a good known base. Eek should be able to fix any corrupted =
filesystem and if it doesn&#8217;t file a defect and keep that =
filesystem aside for development debugging.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">It =
will be a good idea to create brand new volumes and enable snapshots on =
them (it&#8217;s disabled by default for new volumes), keep them small =
and eek them often, every few hours at first, then let the interval =
grow.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Please take note of new volumes created, ones that passed =
eek (and when, day and submittal), and any volume that has not been =
eeked or has not passed eek.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Run =
eek in verbose mode every 3-4 days or every submittal to make sure that =
all the filesystems are still in good shape incase we detect any =
filesystem corruption problem which requires eek to be run to fix it, =
file a defect, raise the flag so that development can immediately take a =
look at that and collect all the info they need to resolve that =
issue.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Enable Volume exception corefile feature, Unexpected =
volume exception can be one indication of FS metadata corruption. Please =
run eek in verbose mode immediately if you get any unexpected volume =
exception.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">To =
enable volume exception corefile feature, run command &#8220;system =
modify volumeexceptiondump -m on&#8221;. This will generate a core file =
whenever there is a volume exception. That corefile gets copied to mgmt =
volume.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">FS =
hardening and No known Filesystem corruption is the objective of Clio =
and we are going to make sure that we achieve that when we release Clio =
on December 18</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><SUP><FONT SIZE=3D2 =
FACE=3D"Arial">th</FONT></SUP></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Thanks</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Vikas</FONT></SPAN></P>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>

</BODY>
</HTML>
------_=_NextPart_001_01C709EA.C29941FD--
