X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7FF9D.4C81E052@onstor-exch02.onstor.net>; Tue, 25 Sep 2007 09:55:46 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7FF9D.4C81E052"
References: <BB375AF679D4A34E9CA8DFA650E2B04E05B467AC@onstor-exch02.onstor.net>
Content-class: urn:content-classes:message
Subject: RE: NFS sequential write performance
Date: Tue, 25 Sep 2007 09:55:46 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E03E5718E@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: NFS sequential write performance
Thread-Index: Acf/mipjyus+Vc6NTkKJpNKDXPsIPwAAPodN
From: "Paul Hammer" <paul.hammer@onstor.com>
To: "Maxim Kozlovsky" <maxim.kozlovsky@onstor.com>,
	"Fay Chong" <fay.chong@onstor.com>,
	"Jonathan Goldick" <jonathan.goldick@onstor.com>,
	"Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>,
	"Brian Montero" <brian.montero@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>
Cc: "Amit Bothra" <amit.bothra@onstor.com>,
	"Bill Nadzam" <bill.nadzam@onstor.com>,
	"Bob Miller" <bob.miller@onstor.com>,
	"Brian DeForest" <brian.deforest@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7FF9D.4C81E052
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Max,
=20
Very nice progress.=20
=20
Looks like we need to do this in R98; we can make the call later if this =
drives the whole R98 schedule, i.e. pulls the ship date in possibly if =
the work is done and tested earlier than 12/31.
=20
Can we start this work now?=20
=20
I copied Bill and Amit since they are both working on similar or perhaps =
related issues, want to make sure we are all aligned and not working on =
the same issues interpedently.=20
=20
Thanks,
=20
-Paul
=20
=20

________________________________

From: Maxim Kozlovsky
Sent: Tue 9/25/2007 10:33 AM
To: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Brian =
Montero; Andy Sharp
Subject: NFS sequential write performance



Here is the summary of yesterday's experiments:

Linux RH3 and BSD 6.2 performance was acceptable, BSD was somewhat =
better than Linux. This can be attributed to the hardware difference. We =
need to rerun the test on the identical hardware if anybody is =
interested to find out what exactly is the difference between these two =
OSs.

Linux RH5 - the problem is how we handle the unaligned writes. This bad =
news is that it is not easy to fix. The good news is that I have written =
most of the code to handle it. It will of course now take me twice the =
time to finish the code to it could take if I got a change to work on it =
back in March. In any case, this is a release project and not something =
that can be patched.

BSD 4.1 - The problem is BSD committing too often. Not much we can do =
about it except writing the data faster during commits. Implementing =
large I/Os may help, which is another thing I was going to do in March. =
This proves once again that we should do as I say and everything will be =
fine.

I could not reproduce the issue with bug 19041 (slow copy of the files) =
in the test environment. RH5 is slower than RH3 but not by a factor of =
10 as when copying on compile2 with dogfood as the server. One of the =
differences between the test system and dogfood is the link aggregation =
on the dogfood. We need to setup the test system with the same link =
configuration as dogfood and try again. We may also need to try the same =
linux version as on compile2. Fay, could you please make the changes in =
the test setup?

Max


------_=_NextPart_001_01C7FF9D.4C81E052
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<HTML dir=3Dltr><HEAD><TITLE>NFS sequential write performance</TITLE>=0A=
<META http-equiv=3DContent-Type content=3D"text/html; charset=3Dunicode">=0A=
<META content=3D"MSHTML 6.00.2900.3132" name=3DGENERATOR></HEAD>=0A=
<BODY>=0A=
<DIV id=3DidOWAReplyText1708 dir=3Dltr>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 =
size=3D2>Max,</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>Very nice progress. =
</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>Looks like we need to do this =
in R98; we can make the call later if this drives the whole R98 =
schedule, i.e. pulls the ship date in possibly if the work is done and =
tested earlier than 12/31.</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>Can we start this work now? =
</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>I copied Bill and Amit since =
they are both working on similar or perhaps related issues, want to make =
sure we are all aligned and not working on the same issues =
interpedently. </FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>Thanks,</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>-Paul</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV></DIV>=0A=
<DIV dir=3Dltr><BR>=0A=
<HR tabIndex=3D-1>=0A=
<FONT face=3DTahoma size=3D2><B>From:</B> Maxim =
Kozlovsky<BR><B>Sent:</B> Tue 9/25/2007 10:33 AM<BR><B>To:</B> Fay =
Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Brian Montero; =
Andy Sharp<BR><B>Subject:</B> NFS sequential write =
performance<BR></FONT><BR></DIV>=0A=
<DIV>=0A=
<P dir=3Dltr><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT =
face=3DArial size=3D2>Here is the summary of =
yesterday</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2>&#8217;</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT face=3DArial size=3D2>s =
experiments:</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us></SPAN></P>=0A=
<P dir=3Dltr><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us></SPAN></P>=0A=
<P dir=3Dltr><SPAN lang=3Den-us><FONT face=3DArial size=3D2>Linux RH3 =
and BSD 6.2</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> =
<FONT face=3DArial size=3D2>performance was =
acceptable,</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2> BSD</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT face=3DArial =
size=3D2>was</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> =
<FONT face=3DArial size=3D2>somewhat</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT face=3DArial =
size=3D2>better</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2> than Linux</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT face=3DArial =
size=3D2>.</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2></FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT face=3DArial =
size=3D2>This</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2> can be attributed to the =
hardware difference. We need to</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT face=3DArial =
size=3D2>rerun</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us> <FONT face=3DArial size=3D2>the</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT face=3DArial size=3D2> test =
on the identical hardware</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2> if anybody is interested to =
find out</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> =
<FONT face=3DArial size=3D2>what exactly is the difference between these =
two OSs.</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us></SPAN></P>=0A=
<P dir=3Dltr><SPAN lang=3Den-us><FONT face=3DArial size=3D2>Linux =
RH5</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT =
face=3DArial size=3D2>&#8211;</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT face=3DArial size=3D2> the =
problem is how we handle the unaligned writes. This bad news is that it =
is not easy to fix. The good news is that I have written most of the =
code to handle it</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2>. I</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT face=3DArial size=3D2>t =
will</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT =
face=3DArial size=3D2>of course</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT face=3DArial size=3D2> =
now</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT =
face=3DArial size=3D2> take me twice t</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT face=3DArial size=3D2>he =
time</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT =
face=3DArial size=3D2>to finish the code</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT face=3DArial =
size=3D2>to</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> =
<FONT face=3DArial size=3D2>it could take if I got a change to work on =
it back in March.</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2> In any case, this is a release =
project and not something that can be patched.</FONT></SPAN></P>=0A=
<P dir=3Dltr><SPAN lang=3Den-us><FONT face=3DArial size=3D2>BSD =
4.1</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT =
face=3DArial size=3D2>&#8211;</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT face=3DArial =
size=3D2></FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> =
<FONT face=3DArial size=3D2>The problem is BSD committing =
to</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT =
face=3DArial size=3D2>o</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2> often.</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT face=3DArial size=3D2>Not =
much we can do about it</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2> except writing the data =
faster</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT =
face=3DArial size=3D2> during commits</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT face=3DArial size=3D2>. =
Implementing large I/Os may help, which is another thing I was going to =
do</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT =
face=3DArial size=3D2> in March</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT face=3DArial =
size=3D2>.</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> =
<FONT face=3DArial size=3D2>This</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT face=3DArial =
size=3D2>proves once</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2></FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT face=3DArial =
size=3D2>again that we should do as</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT face=3DArial size=3D2>I =
say</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT =
face=3DArial size=3D2>and everything will be fine.</FONT></SPAN></P>=0A=
<P dir=3Dltr><SPAN lang=3Den-us><FONT face=3DArial size=3D2>I could not =
reproduce the issue with bug</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT face=3DArial =
size=3D2>19041</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2> (slow copy of the =
files)</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT =
face=3DArial size=3D2> in the test environment.</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT face=3DArial size=3D2>RH5 =
is slower than RH3 but not by a factor of 10 as when copying on compile2 =
with dogfood as</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us> <FONT face=3DArial size=3D2>the</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT face=3DArial =
size=3D2></FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> =
<FONT face=3DArial size=3D2>server</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us><FONT face=3DArial =
size=3D2>.</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> =
<FONT face=3DArial size=3D2>One of the differences =
between</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT =
face=3DArial size=3D2>the</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2></FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT face=3DArial size=3D2>test =
system and dogfood is the link</FONT></SPAN><SPAN =
lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT face=3DArial =
size=3D2>aggregation</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2> on the dogfood. We need =
to</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT =
face=3DArial size=3D2>setup the test system with the =
same</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT =
face=3DArial size=3D2>link</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us> <FONT face=3DArial size=3D2>configuration as dogfood and =
try again.</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us><FONT face=3DArial size=3D2> We may also need to try =
the</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT =
face=3DArial size=3D2>same linux version as on =
compile2.</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> =
<FONT face=3DArial size=3D2>Fay, could you please make the =
changes</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN lang=3Den-us> <FONT =
face=3DArial size=3D2>in the test setup?</FONT></SPAN></P>=0A=
<P dir=3Dltr><SPAN lang=3Den-us><FONT face=3DArial =
size=3D2>Max</FONT></SPAN><SPAN lang=3Den-us></SPAN><SPAN =
lang=3Den-us></SPAN></P></DIV></BODY></HTML>
------_=_NextPart_001_01C7FF9D.4C81E052--
