X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C8679A.5C229196@onstor-exch02.onstor.net>; Mon, 4 Feb 2008 18:56:44 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C8679A.5C229196"
Content-class: urn:content-classes:message
Subject: RE: 3.2 corruption update
Date: Mon, 4 Feb 2008 18:56:52 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E0812B70D@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E0812B708@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: 3.2 corruption update
Thread-Index: Achmt8jM3FGfeB2VSrWjodNwBbWifwAVIiiAABPrViAAAIIZoAAAFptwAAA36iAAAH0vcAAAd3BAAAAJz+AAANGg4AAAsotgAAAchrAADBY0oAAADOxw
References: <BB375AF679D4A34E9CA8DFA650E2B04E0812B3FF@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E0812B405@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E0812B708@onstor-exch02.onstor.net>
From: "Jonathan Goldick" <jonathan.goldick@onstor.com>
To: "Maxim Kozlovsky" <maxim.kozlovsky@onstor.com>,
	"Paul Hammer" <paul.hammer@onstor.com>,
	"Ed Kwan" <ed.kwan@onstor.com>,
	"Danqing Jin" <danqing.jin@onstor.com>,
	"Brian Nguyen" <brian.nguyen@onstor.com>
Cc: "Andy Sharp" <andy.sharp@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C8679A.5C229196
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Let's package it up and see what the Linux community sais on this point,
assuming there is no fix already available.

=20

CC'ing Andy since he reads more Linux news than I do.

=20

________________________________

From: Maxim Kozlovsky=20
Sent: Monday, February 04, 2008 5:55 PM
To: Paul Hammer; Ed Kwan; Danqing Jin; Brian Nguyen
Cc: Jonathan Goldick
Subject: RE: 3.2 corruption update

=20

I've checked that Linux clients to Linux server have the same problem.

=20

________________________________

From: Paul Hammer=20
Sent: Monday, February 04, 2008 12:10 PM
To: Maxim Kozlovsky; Ed Kwan; Danqing Jin; Brian Nguyen
Cc: Jonathan Goldick
Subject: RE: 3.2 corruption update

=20

Sounds good. Danqing and Brian N can you please do this ASAP?

=20

________________________________

From: Maxim Kozlovsky=20
Sent: Monday, February 04, 2008 12:08 PM
To: Maxim Kozlovsky; Paul Hammer; Ed Kwan; Danqing Jin
Cc: Jonathan Goldick
Subject: RE: 3.2 corruption update

=20

I'd like to see the 3.2 corruption where non-null characters where
inserted reproduced while collecting a network trace on all the clients.
I want to confirm that no client has written the bad data on its own
initiative.

=20

________________________________

From: Maxim Kozlovsky=20
Sent: Monday, February 04, 2008 11:53 AM
To: Paul Hammer; Ed Kwan; Danqing Jin
Cc: Jonathan Goldick
Subject: RE: 3.2 corruption update

=20

The null thing appears to be a client bug. I have a trace that shows
that, /homes/maximk/tmp/f1.eth.

=20

In the resulting file we had a sequence of 7 nulls at the offset 19292.
Here is what the client done:

=20

Frame 4640-4643 - the client wrote at offset 16384 2908 bytes, resulting
in file length 19292

=20

Frame 4644-4647 - couple of getattr calls - still returning file length
19292

=20

Access call at frame 4648-4649  - returning the file length 19299. The
file was updated by another client, this advanced the mtime/ctime.

=20

Frame 4652 - Now instead of reading back the new data, the client pads
the missing data at 19292-19299 with zeroes and proceeds with writing at
offset 16384 length 2922.=20

=20

I wonder if the client could pad the missing data with something else
instead of zeroes as well. This will explain the 3.2 corruption.

=20

=20

________________________________

From: Paul Hammer=20
Sent: Monday, February 04, 2008 11:23 AM
To: Ed Kwan; Maxim Kozlovsky; Danqing Jin
Cc: Jonathan Goldick
Subject: RE: 3.2 corruption update

=20

Thanks, please let me know what the next steps are, we need to keep
meeting and planning multiple times a day until we are close this one
out.

=20

________________________________

From: Ed Kwan=20
Sent: Monday, February 04, 2008 11:21 AM
To: Paul Hammer; Maxim Kozlovsky; Danqing Jin
Cc: Jonathan Goldick
Subject: RE: 3.2 corruption update

=20

Discussing this right now in the Escalation Meeting...

=20

________________________________

From: Paul Hammer=20
Sent: Monday, February 04, 2008 11:10 AM
To: Maxim Kozlovsky; Ed Kwan; Danqing Jin
Cc: Jonathan Goldick
Subject: FW: 3.2 corruption update

=20

Could the nulls be a problem with the test code/application?

=20

________________________________

From: Danqing Jin=20
Sent: Monday, February 04, 2008 10:54 AM
To: Paul Hammer
Cc: Ed Kwan
Subject: RE: 3.2 corruption update

=20

Yes.  BTW, the corruption is more subtle than what we saw with 3.2.

=20

________________________________

From: Paul Hammer=20
Sent: Monday, February 04, 2008 10:48 AM
To: Danqing Jin
Cc: Ed Kwan
Subject: RE: 3.2 corruption update

Just to confirm, this was a clean file (new or un-corrupted), and now
there are 10 instances of corruption? Please confirm asap.

=20

________________________________

From: Danqing Jin=20
Sent: Monday, February 04, 2008 10:47 AM
To: Paul Hammer
Cc: Ed Kwan
Subject: RE: 3.2 corruption update

=20

The test aborted around 4am because VPN connection on my home PC was
disconnected, but there are up to 10 or so sightings of NULL inserted
into the file.

=20

________________________________

From: Paul Hammer=20
Sent: Monday, February 04, 2008 10:31 AM
To: Danqing Jin
Cc: Ed Kwan
Subject: RE: 3.2 corruption update

Thanks, how did the over night testing go? Need an update as soon as you
can get me one. Thanks,

=20

=20

-Paul

=20

________________________________

From: Danqing Jin=20
Sent: Monday, February 04, 2008 1:44 AM
To: Paul Hammer
Subject: RE: 3.2 corruption update

=20

Paul,

=20

Jobi talked to me this evening, and both of us were thinking that we
probably should run the same test against 3.1 system as well just as
reference (or as a blank test), which may give us one more data point.
So I started running the same test against my filer over night.  Some
very initial result seems to indicate that issue we saw in sub 5 may
also exist in 3.1 where some content is replaced by NULL characters (or
the NULLs were inserted), I need still let the test run through the
night to get a larger sample.

=20

Thanks,

-Danqing-

=20

________________________________

From: Paul Hammer=20
Sent: Sunday, February 03, 2008 2:55 PM
To: Ed Kwan; Jobi Ariyamannil; Sandrine Boulanger; Brian Nguyen; John
Rogers; Eric Barrett; Danqing Jin; Jonathan Goldick; Maxim Kozlovsky
Cc: Caeli Collins; Bob Miller
Subject: 3.2 corruption update

Hi All,

=20

Please send your latest status updates to me or this list by 8:30
tomorrow morning; if you have any new news/understanding of the issue at
hand. I want us to have the most current information so that Bob, Caeli
and I can discuss our customer facing options in the morning.

=20

Thanks all,

=20

-Paul


------_=_NextPart_001_01C8679A.5C229196
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
 namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" =
name=3D"City"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"place"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:"MS Mincho";
	panose-1:2 2 6 9 4 2 5 8 3 4;}
@font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
	{font-family:"\@MS Mincho";
	panose-1:2 2 6 9 4 2 5 8 3 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle18
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle19
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle20
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle21
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle22
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle23
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle24
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle25
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle26
	{mso-style-type:personal-reply;
	font-family:Arial;
	color:navy;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Let&#8217;s package it up and see =
what the
Linux community <st1:place w:st=3D"on"><st1:City =
w:st=3D"on">sais</st1:City></st1:place>
on this point, assuming there is no fix already =
available.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>CC&#8217;ing Andy since he reads =
more
Linux news than I do.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> Maxim =
Kozlovsky <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, February =
04, 2008
5:55 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Paul
 Hammer</st1:PersonName>; Ed Kwan; <st1:PersonName w:st=3D"on">Danqing =
Jin</st1:PersonName>;
Brian Nguyen<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Jonathan Goldick<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: 3.2 =
corruption update</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>I&#8217;ve checked that Linux =
clients to
Linux server have the same problem.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div style=3D'border:none;border-left:solid blue 1.5pt;padding:0in 0in =
0in 4.0pt'>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">Paul Hammer</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, February =
04, 2008
12:10 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Maxim Kozlovsky; =
<st1:PersonName
w:st=3D"on">Ed Kwan</st1:PersonName>; <st1:PersonName =
w:st=3D"on">Danqing Jin</st1:PersonName>;
Brian Nguyen<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Jonathan
 Goldick</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: 3.2 =
corruption update</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Sounds good. Danqing and Brian N =
can you
please do this ASAP?<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">Maxim Kozlovsky</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, February =
04, 2008
12:08 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Maxim
 Kozlovsky</st1:PersonName>; <st1:PersonName w:st=3D"on">Paul =
Hammer</st1:PersonName>;
<st1:PersonName w:st=3D"on">Ed Kwan</st1:PersonName>; <st1:PersonName =
w:st=3D"on">Danqing
 Jin</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Jonathan
 Goldick</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: 3.2 =
corruption update</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>I&#8217;d like to see the 3.2 =
corruption
where non-null characters where inserted reproduced while collecting a =
network trace
on all the clients. I want to confirm that no client has written the bad =
data
on its own initiative.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div style=3D'border:none;border-left:solid blue 1.5pt;padding:0in 0in =
0in 4.0pt'>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">Maxim Kozlovsky</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, February =
04, 2008
11:53 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Paul
 Hammer</st1:PersonName>; <st1:PersonName w:st=3D"on">Ed =
Kwan</st1:PersonName>; <st1:PersonName
w:st=3D"on">Danqing Jin</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Jonathan
 Goldick</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: 3.2 =
corruption update</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>The null thing appears to be a =
client bug.
I have a trace that shows that, =
/homes/maximk/tmp/f1.eth.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>In the resulting file we had a =
sequence of
7 nulls at the offset 19292. Here is what the client =
done:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Frame 4640-4643 &#8211; the client =
wrote at
offset 16384 2908 bytes, resulting in file length =
19292<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Frame 4644-4647 - couple of getattr =
calls
&#8211; still returning file length 19292<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Access call at frame =
4648-4649&nbsp; -
returning the file length 19299. The file was updated by another client, =
this
advanced the mtime/ctime.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Frame 4652 - Now instead of reading =
back
the new data, the client pads the missing data at 19292-19299 with =
zeroes and
proceeds with writing at offset 16384 length 2922. =
<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>I wonder if the client could pad =
the
missing data with something else instead of zeroes as well. This will =
explain
the 3.2 corruption.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div style=3D'border:none;border-left:solid blue 1.5pt;padding:0in 0in =
0in 4.0pt'>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">Paul Hammer</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, February =
04, 2008
11:23 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Ed
 Kwan</st1:PersonName>; <st1:PersonName w:st=3D"on">Maxim =
Kozlovsky</st1:PersonName>;
<st1:PersonName w:st=3D"on">Danqing Jin</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Jonathan
 Goldick</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: 3.2 =
corruption update</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Thanks, please let me know what the =
next
steps are, we need to keep meeting and planning multiple times a day =
until we
are close this one out.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">Ed Kwan</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, February =
04, 2008
11:21 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Paul
 Hammer</st1:PersonName>; <st1:PersonName w:st=3D"on">Maxim =
Kozlovsky</st1:PersonName>;
<st1:PersonName w:st=3D"on">Danqing Jin</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Jonathan
 Goldick</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: 3.2 =
corruption update</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Discussing this right now in the
Escalation Meeting&#8230;<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div style=3D'border:none;border-left:solid blue 1.5pt;padding:0in 0in =
0in 4.0pt'>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">Paul Hammer</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, February =
04, 2008
11:10 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Maxim
 Kozlovsky</st1:PersonName>; <st1:PersonName w:st=3D"on">Ed =
Kwan</st1:PersonName>;
<st1:PersonName w:st=3D"on">Danqing Jin</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Jonathan
 Goldick</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> FW: 3.2 =
corruption update</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Could the nulls be a problem with =
the test
code/application?<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">Danqing Jin</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, February =
04, 2008
10:54 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Paul
 Hammer</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Ed
 Kwan</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: 3.2 =
corruption update</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:blue'>Yes.&nbsp; BTW, the corruption is =
more
subtle than what we saw with 3.2.</span></font><o:p></o:p></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;font-family:Tahoma'> <st1:PersonName
w:st=3D"on">Paul Hammer</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, February =
04, 2008
10:48 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Danqing
 Jin</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Ed
 Kwan</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: 3.2 =
corruption update</span></font><o:p></o:p></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Just to confirm, this was a clean =
file
(new or un-corrupted), and now there are 10 instances of corruption? =
Please
confirm asap.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">Danqing Jin</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, February =
04, 2008
10:47 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Paul
 Hammer</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Ed
 Kwan</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: 3.2 =
corruption update</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:blue'>The test aborted around 4am because =
VPN
connection on my home PC was disconnected, but there are up to 10 or so
sightings of NULL inserted into the file.</span></font><o:p></o:p></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;font-family:Tahoma'> <st1:PersonName
w:st=3D"on">Paul Hammer</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, February =
04, 2008
10:31 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Danqing
 Jin</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Ed
 Kwan</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: 3.2 =
corruption update</span></font><o:p></o:p></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Thanks, how did the over night =
testing go?
Need an update as soon as you can get me one. =
Thanks,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>-Paul<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">Danqing Jin</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, February =
04, 2008
1:44 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Paul
 Hammer</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: 3.2 =
corruption update</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:blue'>Paul,</span></font><o:p></o:p></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:blue'>Jobi talked to me this evening, and =
both
of us were thinking that we probably should run the same test against =
3.1 system
as well just as reference&nbsp;(or as&nbsp;a blank test), which may give =
us one
more data point.&nbsp; So I started running the same test against my =
filer over
night.&nbsp; Some very initial result seems to indicate that issue we =
saw in
sub 5 may also exist in 3.1 where some content is replaced by NULL =
characters
(or the NULLs were inserted), I need still let the test run through the =
night
to&nbsp;get a larger sample.</span></font><o:p></o:p></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:blue'>Thanks,</span></font><o:p></o:p></p>=


<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:blue'>-Danqing-</span></font><o:p></o:p></=
p>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

</div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;font-family:Tahoma'> <st1:PersonName
w:st=3D"on">Paul Hammer</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Sunday, February =
03, 2008
2:55 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Ed
 Kwan</st1:PersonName>; <st1:PersonName w:st=3D"on">Jobi =
Ariyamannil</st1:PersonName>;
<st1:PersonName w:st=3D"on">Sandrine Boulanger</st1:PersonName>; =
<st1:PersonName
w:st=3D"on">Brian Nguyen</st1:PersonName>; <st1:PersonName =
w:st=3D"on">John Rogers</st1:PersonName>;
<st1:PersonName w:st=3D"on">Eric Barrett</st1:PersonName>; =
<st1:PersonName w:st=3D"on">Danqing
 Jin</st1:PersonName>; <st1:PersonName w:st=3D"on">Jonathan =
Goldick</st1:PersonName>;
<st1:PersonName w:st=3D"on">Maxim Kozlovsky</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Caeli
 Collins</st1:PersonName>; <st1:PersonName w:st=3D"on">Bob =
Miller</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> 3.2 corruption =
update</span></font><o:p></o:p></p>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dblack face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:black'>Hi =
All,</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Please send your latest status updates to me or this =
list by
8:30 tomorrow morning; if you have any new news/understanding of the =
issue at
hand. I want us to have the most current information so that Bob, Caeli =
and I
can discuss our customer facing options in the =
morning.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Thanks all,</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>-Paul</span></font><o:p></o:p></p>

</div>

</div>

</div>

</div>

</div>

</div>

</body>

</html>

------_=_NextPart_001_01C8679A.5C229196--
