X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C70D0D.068E1A71@onstor-exch02.onstor.net>; Mon, 20 Nov 2006 17:33:19 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C70D0D.068E1A71"
Content-class: urn:content-classes:message
Subject: RE: Important Clio Updates 
Date: Mon, 20 Nov 2006 17:33:19 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E0116DD80@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E01620681@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Important Clio Updates 
thread-index: AccJFf7/mkyKAmwOTPilIXBR13Zx8wAAFXuwAAFQemAAAQ5qUAD7AtAQ
From: "Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>
To: "Vikas Saini" <vikas.saini@onstor.com>,
	"dl-Clio" <dl-Clio@onstor.com>,
	"dl-QA" <dl-qa@onstor.com>,
	"dl-hcl-qa" <dl-hcl-qa@onstor.com>
Cc: "dl-File System" <dl-fs@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C70D0D.068E1A71
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

One more thing I would like to note:

It's a known fact that our current eek may not fix all the problems in
one pass.  So QE may have to run eek on their filesystems until no
errors are reported.
But if the second eek finds anymore errors (with this week's submittal
onwards), please open defects.
We would like to enhance eek to detect and fix all the problems in the
first pass itself.

Also please collect the eek logs if eek is taking more than 12 hours on
any filesystem and let us know.

Regards,
Jobi

_____________________________________________
From: Vikas Saini=20
Sent: Wednesday, November 15, 2006 6:06 PM
To: dl-Clio; dl-QA; dl-hcl-qa
Cc: dl-Engineering
Subject: Important Clio Updates=20
Importance: High

Hi All,

In today's project meeting, it was collectively agreed that we will not
be shipping Clio with any known FS corruption problems. FileSystem team
has made a lot of changes which will prevent known corruptions as well
as detecting and then fixing FS corruption issues (i.e. eek
improvements). We don't have a solid/easy way to figure out what is
causing that corruption on first place. In Clio, we are going to take a
focused approach to help towards resolving this issue.=20

Starting ASAP, we are going to perform all of our QE testing on a known
good filesystem. To do that, we will be eeking all the volumes in QE
testing env 's, using the -r option (repair) .=20
Incase we find any filesystem which eek can't fix, we will keep those
volumes for Dev to look at and figure out why eek is not able to repair
those volumes. If Eek does find a problem we will run eek multiple time
until it reports everything is fine.

Once we have a known good base(all known corruptions have been removed
and eek has passed), we are going to continue running our test
regression. We will run eek in verify mode (using only the -v option)
periodically (with every submittal) to make sure that filesystem is not
corrupted.=20

As soon as we hit any potential File System issues e.g. corruption or
volume exceptions, there will be a combined effort to diagnose the
problem.  This will involve understanding the test scenario which led to
that corruption as well as collecting all the information that could
help in fixing that issue. We will keep on doing this to the point that
there are zero known filesystem problems.


QE Team:

Please eek all the volumes and repair them to make sure that it is a
good known base. Eek should be able to fix any corrupted filesystem and
if it doesn't file a defect and keep that filesystem aside for
development debugging.


It will be a good idea to create brand new volumes and enable snapshots
on them (it's disabled by default for new volumes), keep them small and
eek them often, every few hours at first, then let the interval grow.

Please take note of new volumes created, ones that passed eek (and when,
day and submittal), and any volume that has not been eeked or has not
passed eek.

Run eek in verbose mode every 3-4 days or every submittal to make sure
that all the filesystems are still in good shape incase we detect any
filesystem corruption problem which requires eek to be run to fix it,
file a defect, raise the flag so that development can immediately take a
look at that and collect all the info they need to resolve that issue.


Enable Volume exception corefile feature, Unexpected volume exception
can be one indication of FS metadata corruption. Please run eek in
verbose mode immediately if you get any unexpected volume exception.

To enable volume exception corefile feature, run command "system modify
volumeexceptiondump -m on". This will generate a core file whenever
there is a volume exception. That corefile gets copied to mgmt volume.

FS hardening and No known Filesystem corruption is the objective of Clio
and we are going to make sure that we achieve that when we release Clio
on December 18th.


Thanks
Vikas









------_=_NextPart_001_01C70D0D.068E1A71
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7650.28">
<TITLE>RE: Important Clio Updates </TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Hi,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">One more thing I would like to note:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">I</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">t&#8217;s =
a</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial"> known fact that our current =
eek may not fix all the problems in one pass.&nbsp;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">So QE may have to run eek on their filesystems =
until no errors are</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">reported</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">But if the second</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">eek</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">find</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">s</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial"> anymore =
errors (with</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">this =
week</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">&#8217;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">s</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">submittal onwards), please open =
defects.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">We would like to</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">enhance</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial"></FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">eek =
to</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">detect</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial"> and</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> <FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">fix all the problems in the first pass =
itself.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Also please collect the eek logs if eek is taking more =
than 12 hours on any filesystem and let us know.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Regards,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Jobi</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Vikas Saini<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Wednesday, November 15, =
2006 6:06 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Clio; dl-QA; =
dl-hcl-qa<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Engineering<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Important Clio Updates<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Importance:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> High</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us">Hi All,</SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">In =
today&#8217;s project meeting, it was collectively agreed that we will =
not be shipping Clio with any known FS corruption problems. FileSystem =
team has made a lot of changes which will prevent known corruptions as =
well as detecting and then fixing FS corruption issues (i.e. eek =
improvements). We don&#8217;t have a solid/easy way to figure out what =
is causing that corruption on first place. In Clio, we are going to take =
a focused approach to help towards resolving this issue. =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Starting ASAP, we are going to perform all of our QE =
testing on a known good filesystem. To do that, we will be eeking all =
the volumes in QE testing env &#8216;s, using the &#8211;r option =
(repair) . </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Incase we find any filesystem which eek can&#8217;t fix, =
we will keep those volumes for Dev to look at and figure out why eek is =
not able to repair those volumes. If Eek does find a problem we will run =
eek multiple time until it reports everything is fine.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Once =
we have a known good base(all known corruptions have been removed and =
eek has passed), we are going to continue running our test regression. =
We will run eek in verify mode (using only the &#8211;v option) =
periodically (with every submittal) to make sure that filesystem is not =
corrupted. </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">As =
soon as we hit any potential File System issues e.g. corruption or =
volume exceptions, there will be a combined effort to diagnose the =
problem.&nbsp; This will involve understanding the test scenario which =
led to that corruption as well as collecting all the information that =
could help in fixing that issue. We will keep on doing this to the point =
that there are zero known filesystem problems.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">QE =
Team:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Please eek all the volumes and repair them to make sure =
that it is a good known base. Eek should be able to fix any corrupted =
filesystem and if it doesn&#8217;t file a defect and keep that =
filesystem aside for development debugging.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">It =
will be a good idea to create brand new volumes and enable snapshots on =
them (it&#8217;s disabled by default for new volumes), keep them small =
and eek them often, every few hours at first, then let the interval =
grow.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Please take note of new volumes created, ones that passed =
eek (and when, day and submittal), and any volume that has not been =
eeked or has not passed eek.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Run =
eek in verbose mode every 3-4 days or every submittal to make sure that =
all the filesystems are still in good shape incase we detect any =
filesystem corruption problem which requires eek to be run to fix it, =
file a defect, raise the flag so that development can immediately take a =
look at that and collect all the info they need to resolve that =
issue.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Enable Volume exception corefile feature, Unexpected =
volume exception can be one indication of FS metadata corruption. Please =
run eek in verbose mode immediately if you get any unexpected volume =
exception.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">To =
enable volume exception corefile feature, run command &#8220;system =
modify volumeexceptiondump -m on&#8221;. This will generate a core file =
whenever there is a volume exception. That corefile gets copied to mgmt =
volume.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">FS =
hardening and No known Filesystem corruption is the objective of Clio =
and we are going to make sure that we achieve that when we release Clio =
on December 18</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><SUP><FONT SIZE=3D2 =
FACE=3D"Arial">th</FONT></SUP></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">.</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Thanks</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Vikas</FONT></SPAN></P>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>

</BODY>
</HTML>
------_=_NextPart_001_01C70D0D.068E1A71--
