X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7FFC1.12150EB8@onstor-exch02.onstor.net>; Tue, 25 Sep 2007 14:11:50 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7FFC1.12150EB8"
Content-class: urn:content-classes:message
Subject: RE: NFS sequential write performance
Date: Tue, 25 Sep 2007 14:11:49 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E05B46A65@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E03E57195@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: NFS sequential write performance
Thread-Index: Acf/mipjyus+Vc6NTkKJpNKDXPsIPwAAPodNAACfvAAAAEmqxQAAwWig
References: <BB375AF679D4A34E9CA8DFA650E2B04E05B467AC@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E03E5718E@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E05B46809@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E03E57195@onstor-exch02.onstor.net>
From: "Maxim Kozlovsky" <maxim.kozlovsky@onstor.com>
To: "Paul Hammer" <paul.hammer@onstor.com>,
	"Fay Chong" <fay.chong@onstor.com>,
	"Jonathan Goldick" <jonathan.goldick@onstor.com>,
	"Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>,
	"Brian Montero" <brian.montero@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7FFC1.12150EB8
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

The problem with the file copy looks to be the networking. I've done
some copy tests with the dogfood, we are loosing a lot of packets during
the copy and I've got a trace to prove that. If we fail to reproduce the
problem with the link aggregation in the test environment, we will need
to ask support and Kevin to go figure out where the packets are going.
It could be just bad configuration on switch/network.

=20

________________________________

From: Paul Hammer=20
Sent: Tuesday, September 25, 2007 11:06 AM
To: Maxim Kozlovsky; Fay Chong; Jonathan Goldick; Jobi Ariyamannil;
Brian Montero; Andy Sharp
Cc: Amit Bothra; Bill Nadzam; Bob Miller; Brian DeForest
Subject: RE: NFS sequential write performance

=20

Great. Agree. Thanks Max.

=20

________________________________

From: Maxim Kozlovsky
Sent: Tue 9/25/2007 11:04 AM
To: Paul Hammer; Fay Chong; Jonathan Goldick; Jobi Ariyamannil; Brian
Montero; Andy Sharp
Cc: Amit Bothra; Bill Nadzam; Bob Miller; Brian DeForest
Subject: RE: NFS sequential write performance

Sure we can start now. I am still going to find out what is going on
with the 10x file copy slowdown in 19041. If it is caused by a different
issue I would like to fix that problem first.

=20

________________________________

From: Paul Hammer=20
Sent: Tuesday, September 25, 2007 10:56 AM
To: Maxim Kozlovsky; Fay Chong; Jonathan Goldick; Jobi Ariyamannil;
Brian Montero; Andy Sharp
Cc: Amit Bothra; Bill Nadzam; Bob Miller; Brian DeForest
Subject: RE: NFS sequential write performance

=20

Max,

=20

Very nice progress.=20

=20

Looks like we need to do this in R98; we can make the call later if this
drives the whole R98 schedule, i.e. pulls the ship date in possibly if
the work is done and tested earlier than 12/31.

=20

Can we start this work now?=20

=20

I copied Bill and Amit since they are both working on similar or perhaps
related issues, want to make sure we are all aligned and not working on
the same issues interpedently.=20

=20

Thanks,

=20

-Paul

=20

=20

=20

________________________________

From: Maxim Kozlovsky
Sent: Tue 9/25/2007 10:33 AM
To: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Brian
Montero; Andy Sharp
Subject: NFS sequential write performance

Here is the summary of yesterday's experiments:

Linux RH3 and BSD 6.2 performance was acceptable, BSD was somewhat
better than Linux. This can be attributed to the hardware difference. We
need to rerun the test on the identical hardware if anybody is
interested to find out what exactly is the difference between these two
OSs.

Linux RH5 - the problem is how we handle the unaligned writes. This bad
news is that it is not easy to fix. The good news is that I have written
most of the code to handle it. It will of course now take me twice the
time to finish the code to it could take if I got a change to work on it
back in March. In any case, this is a release project and not something
that can be patched.

BSD 4.1 - The problem is BSD committing too often. Not much we can do
about it except writing the data faster during commits. Implementing
large I/Os may help, which is another thing I was going to do in March.
This proves once again that we should do as I say and everything will be
fine.

I could not reproduce the issue with bug 19041 (slow copy of the files)
in the test environment. RH5 is slower than RH3 but not by a factor of
10 as when copying on compile2 with dogfood as the server. One of the
differences between the test system and dogfood is the link aggregation
on the dogfood. We need to setup the test system with the same link
configuration as dogfood and try again. We may also need to try the same
linux version as on compile2. Fay, could you please make the changes in
the test setup?

Max


------_=_NextPart_001_01C7FFC1.12150EB8
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<title>NFS sequential write performance</title>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"City"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"place"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
font-face=0A=
	{font-family:Tahoma;}

 /* Font Definitions */
 @font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:purple;
	text-decoration:underline;}
p
	{mso-margin-top-alt:auto;
	margin-right:0in;
	mso-margin-bottom-alt:auto;
	margin-left:0in;
	font-size:12.0pt;
	font-family:"Times New Roman";}
span.emailstyle18
	{font-family:Arial;
	color:navy;}
span.EmailStyle19
	{mso-style-type:personal-reply;
	font-family:Arial;
	color:navy;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>The problem with the file copy =
looks to be
the networking. I&#8217;ve done some copy tests with the dogfood, we are
loosing a lot of packets during the copy and I&#8217;ve got a trace to =
prove
that. If we fail to reproduce the problem with the link aggregation in =
the test
environment, we will need to ask support and Kevin to go figure out =
where the
packets are going. It could be just bad configuration on =
switch/network.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div style=3D'border:none;border-left:solid blue 1.5pt;padding:0in 0in =
0in 4.0pt'>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> Paul =
Hammer <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, September =
25, 2007
11:06 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Maxim Kozlovsky; Fay =
Chong;
Jonathan Goldick; Jobi Ariyamannil; Brian Montero; Andy Sharp<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Amit Bothra; Bill =
Nadzam; Bob
Miller; Brian DeForest<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: NFS =
sequential write
performance</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div id=3DidOWAReplyText50265>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dblack face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:black'>Great. Agree. Thanks =
Max.</span></font><o:p></o:p></p>

</div>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;font-family:Tahoma'> Maxim
Kozlovsky<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tue 9/25/2007 11:04 =
AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Paul Hammer; Fay =
Chong;
Jonathan Goldick; Jobi Ariyamannil; Brian Montero; Andy Sharp<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Amit Bothra; Bill =
Nadzam; Bob
Miller; Brian DeForest<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: NFS =
sequential write
performance</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Sure we can start now. I am still =
going to
find out what is going on with the 10x file copy slowdown in 19041. If =
it is
caused by a different issue I would like to fix that problem =
first.</span></font><o:p></o:p></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<div style=3D'border:none;border-left:solid blue 1.5pt;padding:0in 0in =
0in 4.0pt'>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> Paul =
Hammer <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, September =
25, 2007
10:56 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Maxim Kozlovsky; Fay =
Chong;
Jonathan Goldick; Jobi Ariyamannil; Brian Montero; Andy Sharp<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Amit Bothra; Bill =
Nadzam; Bob
Miller; Brian DeForest<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: NFS =
sequential write
performance</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<div id=3DidOWAReplyText1708>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dblack face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:black'>Max,</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Very nice progress. </span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Looks like we need to do this in R98; we can make the =
call later
if this drives the whole R98 schedule, i.e. pulls the ship date in =
possibly if
the work is done and tested earlier than =
12/31.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Can we start this work now? =
</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>I copied Bill and Amit since they are both working on
similar or perhaps related issues, want to make sure we are all aligned =
and not
working on the same issues interpedently. </span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Thanks,</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>-Paul</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;font-family:Tahoma'> Maxim
Kozlovsky<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tue 9/25/2007 10:33 =
AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Fay Chong; Paul =
Hammer;
Jonathan Goldick; Jobi Ariyamannil; Brian Montero; Andy Sharp<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> NFS sequential =
write
performance</span></font><o:p></o:p></p>

</div>

<div>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>Here
is the summary of yesterday&#8217;s =
experiments:</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>Linux
RH3 and BSD 6.2</span></font> <font size=3D2 face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial'>performance was acceptable, BSD</span></font> =
<font
size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>was</span></font>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>somewhat</span></font>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>better
than Linux.</span></font> <font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>This can be attributed to the hardware difference. We =
need
to</span></font> <font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>rerun</span></font> <font size=3D2 face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>the test on the identical =
hardware
if anybody is interested to find out</span></font> <font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>what exactly is the =
difference
between these two <st1:City w:st=3D"on"><st1:place =
w:st=3D"on">OSs</st1:place></st1:City>.</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>Linux
RH5</span></font> <font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>&#8211; the problem is how we handle the unaligned =
writes.
This bad news is that it is not easy to fix. The good news is that I =
have
written most of the code to handle it. It will</span></font> <font =
size=3D2
face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial'>of =
course now take
me twice the time</span></font> <font size=3D2 face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial'>to finish the code</span></font> <font =
size=3D2
face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>to</span></font> <font
size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>it could
take if I got a change to work on it back in March. In any case, this is =
a
release project and not something that can be =
patched.</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>BSD
4.1</span></font> <font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>&#8211;</span></font> <font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>The problem is BSD =
committing too
often.</span></font> <font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Not much we can do about it except writing the data =
faster
during commits. Implementing large I/Os may help, which is another thing =
I was
going to do in March.</span></font> <font size=3D2 face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>This</span></font> <font =
size=3D2
face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial'>proves =
once</span></font>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>again
that we should do as</span></font> <font size=3D2 face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>I say</span></font> <font =
size=3D2
face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial'>and =
everything will
be fine.</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>I
could not reproduce the issue with bug</span></font> <font size=3D2 =
face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>19041 (slow copy of the =
files) in
the test environment.</span></font> <font size=3D2 face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>RH5 is slower than RH3 but =
not by a
factor of 10 as when copying on compile2 with dogfood as</span></font> =
<font
size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>the</span></font>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>server.</span></font>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>One of
the differences between</span></font> <font size=3D2 face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>the</span></font> <font =
size=3D2
face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial'>test =
system and
dogfood is the link</span></font> <font size=3D2 face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>aggregation on the dogfood. =
We need
to</span></font> <font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>setup the test system with the same</span></font> =
<font
size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>link</span></font>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>configuration
as dogfood and try again. We may also need to try the</span></font> =
<font
size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>same linux
version as on compile2.</span></font> <font size=3D2 face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial'>Fay, could you please make =
the
changes</span></font> <font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>in the test setup?</span></font><o:p></o:p></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>Max</span></font><o:p></o:p>=
</p>

</div>

</div>

</div>

</div>

</div>

</body>

</html>

------_=_NextPart_001_01C7FFC1.12150EB8--
