X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7FF9E.CFD3E015@onstor-exch02.onstor.net>; Tue, 25 Sep 2007 10:06:36 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7FF9E.CFD3E015"
References: <BB375AF679D4A34E9CA8DFA650E2B04E05B467AC@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E03E5718E@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E05B46809@onstor-exch02.onstor.net>
Content-class: urn:content-classes:message
Subject: RE: NFS sequential write performance
Date: Tue, 25 Sep 2007 10:06:26 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E03E57195@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: NFS sequential write performance
Thread-Index: Acf/mipjyus+Vc6NTkKJpNKDXPsIPwAAPodNAACfvAAAAEmqxQ==
From: "Paul Hammer" <paul.hammer@onstor.com>
To: "Maxim Kozlovsky" <maxim.kozlovsky@onstor.com>,
	"Fay Chong" <fay.chong@onstor.com>,
	"Jonathan Goldick" <jonathan.goldick@onstor.com>,
	"Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>,
	"Brian Montero" <brian.montero@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>
Cc: "Amit Bothra" <amit.bothra@onstor.com>,
	"Bill Nadzam" <bill.nadzam@onstor.com>,
	"Bob Miller" <bob.miller@onstor.com>,
	"Brian DeForest" <brian.deforest@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7FF9E.CFD3E015
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Great. Agree. Thanks Max.

________________________________

From: Maxim Kozlovsky
Sent: Tue 9/25/2007 11:04 AM
To: Paul Hammer; Fay Chong; Jonathan Goldick; Jobi Ariyamannil; Brian =
Montero; Andy Sharp
Cc: Amit Bothra; Bill Nadzam; Bob Miller; Brian DeForest
Subject: RE: NFS sequential write performance



Sure we can start now. I am still going to find out what is going on =
with the 10x file copy slowdown in 19041. If it is caused by a different =
issue I would like to fix that problem first.

=20

________________________________

From: Paul Hammer=20
Sent: Tuesday, September 25, 2007 10:56 AM
To: Maxim Kozlovsky; Fay Chong; Jonathan Goldick; Jobi Ariyamannil; =
Brian Montero; Andy Sharp
Cc: Amit Bothra; Bill Nadzam; Bob Miller; Brian DeForest
Subject: RE: NFS sequential write performance

=20

Max,

=20

Very nice progress.=20

=20

Looks like we need to do this in R98; we can make the call later if this =
drives the whole R98 schedule, i.e. pulls the ship date in possibly if =
the work is done and tested earlier than 12/31.

=20

Can we start this work now?=20

=20

I copied Bill and Amit since they are both working on similar or perhaps =
related issues, want to make sure we are all aligned and not working on =
the same issues interpedently.=20

=20

Thanks,

=20

-Paul

=20

=20

=20

________________________________

From: Maxim Kozlovsky
Sent: Tue 9/25/2007 10:33 AM
To: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Brian =
Montero; Andy Sharp
Subject: NFS sequential write performance

Here is the summary of yesterday's experiments:

Linux RH3 and BSD 6.2 performance was acceptable, BSD was somewhat =
better than Linux. This can be attributed to the hardware difference. We =
need to rerun the test on the identical hardware if anybody is =
interested to find out what exactly is the difference between these two =
OSs.

Linux RH5 - the problem is how we handle the unaligned writes. This bad =
news is that it is not easy to fix. The good news is that I have written =
most of the code to handle it. It will of course now take me twice the =
time to finish the code to it could take if I got a change to work on it =
back in March. In any case, this is a release project and not something =
that can be patched.

BSD 4.1 - The problem is BSD committing too often. Not much we can do =
about it except writing the data faster during commits. Implementing =
large I/Os may help, which is another thing I was going to do in March. =
This proves once again that we should do as I say and everything will be =
fine.

I could not reproduce the issue with bug 19041 (slow copy of the files) =
in the test environment. RH5 is slower than RH3 but not by a factor of =
10 as when copying on compile2 with dogfood as the server. One of the =
differences between the test system and dogfood is the link aggregation =
on the dogfood. We need to setup the test system with the same link =
configuration as dogfood and try again. We may also need to try the same =
linux version as on compile2. Fay, could you please make the changes in =
the test setup?

Max


------_=_NextPart_001_01C7FF9E.CFD3E015
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<HTML dir=3Dltr><HEAD><TITLE>NFS sequential write performance</TITLE>=0A=
<META http-equiv=3DContent-Type content=3D"text/html; charset=3Dunicode">=0A=
<META content=3D"MSHTML 6.00.2900.3132" name=3DGENERATOR>=0A=
<STYLE>=0A=
<!--=0A=
                       =0A=
 font-face=0A=
	{font-family:Tahoma;}=0A=
                        =0A=
 p.MsoNormal, li.MsoNormal, div.MsoNormal=0A=
	{margin:0in;=0A=
	margin-bottom:.0001pt;=0A=
	font-size:12.0pt;=0A=
	font-family:"Times New Roman";}=0A=
a:link, span.MsoHyperlink=0A=
	{color:blue;=0A=
	text-decoration:underline;}=0A=
a:visited, span.MsoHyperlinkFollowed=0A=
	{color:purple;=0A=
	text-decoration:underline;}=0A=
p=0A=
	{=0A=
	margin-right:0in;=0A=
	margin-left:0in;=0A=
	font-size:12.0pt;=0A=
	font-family:"Times New Roman";}=0A=
span.EmailStyle18=0A=
	{=0A=
	font-family:Arial;=0A=
	color:navy;}=0A=
=0A=
div.Section1=0A=
	{page:Section1;}=0A=
-->=0A=
</STYLE>=0A=
</HEAD>=0A=
<BODY lang=3DEN-US vLink=3Dpurple link=3Dblue>=0A=
<DIV id=3DidOWAReplyText50265 dir=3Dltr>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 size=3D2>Great. Agree. =
Thanks Max.</FONT></DIV></DIV>=0A=
<DIV dir=3Dltr><BR>=0A=
<HR tabIndex=3D-1>=0A=
<FONT face=3DTahoma size=3D2><B>From:</B> Maxim =
Kozlovsky<BR><B>Sent:</B> Tue 9/25/2007 11:04 AM<BR><B>To:</B> Paul =
Hammer; Fay Chong; Jonathan Goldick; Jobi Ariyamannil; Brian Montero; =
Andy Sharp<BR><B>Cc:</B> Amit Bothra; Bill Nadzam; Bob Miller; Brian =
DeForest<BR><B>Subject:</B> RE: NFS sequential write =
performance<BR></FONT><BR></DIV>=0A=
<DIV>=0A=
<DIV class=3DSection1>=0A=
<P class=3DMsoNormal><FONT face=3DArial color=3Dnavy size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">Sure we can =
start now. I am still going to find out what is going on with the 10x =
file copy slowdown in 19041. If it is caused by a different issue I =
would like to fix that problem first.</SPAN></FONT></P>=0A=
<P class=3DMsoNormal><FONT face=3DArial color=3Dnavy size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: =
Arial"></SPAN></FONT>&nbsp;</P>=0A=
<DIV style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 0in; BORDER-TOP: =
medium none; PADDING-LEFT: 4pt; PADDING-BOTTOM: 0in; BORDER-LEFT: blue =
1.5pt solid; PADDING-TOP: 0in; BORDER-BOTTOM: medium none">=0A=
<DIV>=0A=
<DIV class=3DMsoNormal style=3D"TEXT-ALIGN: center" align=3Dcenter><FONT =
face=3D"Times New Roman" size=3D3><SPAN style=3D"FONT-SIZE: 12pt">=0A=
<HR tabIndex=3D-1 align=3Dcenter width=3D"100%" SIZE=3D2>=0A=
</SPAN></FONT></DIV>=0A=
<P class=3DMsoNormal><B><FONT face=3DTahoma size=3D2><SPAN =
style=3D"FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: =
Tahoma">From:</SPAN></FONT></B><FONT face=3DTahoma size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"> Paul Hammer <BR><B><SPAN =
style=3D"FONT-WEIGHT: bold">Sent:</SPAN></B> Tuesday, September 25, 2007 =
10:56 AM<BR><B><SPAN style=3D"FONT-WEIGHT: bold">To:</SPAN></B> Maxim =
Kozlovsky; Fay Chong; Jonathan Goldick; Jobi Ariyamannil; Brian Montero; =
Andy Sharp<BR><B><SPAN style=3D"FONT-WEIGHT: bold">Cc:</SPAN></B> Amit =
Bothra; Bill Nadzam; Bob Miller; Brian DeForest<BR><B><SPAN =
style=3D"FONT-WEIGHT: bold">Subject:</SPAN></B> RE: NFS sequential write =
performance</SPAN></FONT></P></DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P>=0A=
<DIV id=3DidOWAReplyText1708>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3DArial color=3Dblack size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: =
Arial">Max,</SPAN></FONT></P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">Very nice progress. =
</SPAN></FONT></P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">Looks like we need to do =
this in R98; we can make the call later if this drives the whole R98 =
schedule, i.e. pulls the ship date in possibly if the work is done and =
tested earlier than 12/31.</SPAN></FONT></P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">Can we start this work =
now? </SPAN></FONT></P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">I copied Bill and Amit =
since they are both working on similar or perhaps related issues, want =
to make sure we are all aligned and not working on the same issues =
interpedently. </SPAN></FONT></P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">Thanks,</SPAN></FONT></P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">-Paul</SPAN></FONT></P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P></DIV></DIV>=0A=
<DIV>=0A=
<P class=3DMsoNormal><FONT face=3D"Times New Roman" size=3D3><SPAN =
style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P>=0A=
<DIV class=3DMsoNormal style=3D"TEXT-ALIGN: center" align=3Dcenter><FONT =
face=3D"Times New Roman" size=3D3><SPAN style=3D"FONT-SIZE: 12pt">=0A=
<HR tabIndex=3D-1 align=3Dcenter width=3D"100%" SIZE=3D2>=0A=
</SPAN></FONT></DIV>=0A=
<P class=3DMsoNormal style=3D"MARGIN-BOTTOM: 12pt"><B><FONT =
face=3DTahoma size=3D2><SPAN style=3D"FONT-WEIGHT: bold; FONT-SIZE: =
10pt; FONT-FAMILY: Tahoma">From:</SPAN></FONT></B><FONT face=3DTahoma =
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"> Maxim =
Kozlovsky<BR><B><SPAN style=3D"FONT-WEIGHT: bold">Sent:</SPAN></B> Tue =
9/25/2007 10:33 AM<BR><B><SPAN style=3D"FONT-WEIGHT: =
bold">To:</SPAN></B> Fay Chong; Paul Hammer; Jonathan Goldick; Jobi =
Ariyamannil; Brian Montero; Andy Sharp<BR><B><SPAN style=3D"FONT-WEIGHT: =
bold">Subject:</SPAN></B> NFS sequential write =
performance</SPAN></FONT></P></DIV>=0A=
<DIV>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">Here is the summary of yesterday&#8217;s =
experiments:</SPAN></FONT></P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">Linux RH3 and BSD 6.2</SPAN></FONT> <FONT =
face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">performance was acceptable, BSD</SPAN></FONT> <FONT face=3DArial =
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">was</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">somewhat</SPAN></FONT> =
<FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">better than Linux.</SPAN></FONT> <FONT face=3DArial =
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">This can be =
attributed to the hardware difference. We need to</SPAN></FONT> <FONT =
face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">rerun</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">the test on the identical =
hardware if anybody is interested to find out</SPAN></FONT> <FONT =
face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">what exactly is the difference between these two =
OSs.</SPAN></FONT></P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">Linux RH5</SPAN></FONT> <FONT face=3DArial =
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">&#8211; the =
problem is how we handle the unaligned writes. This bad news is that it =
is not easy to fix. The good news is that I have written most of the =
code to handle it. It will</SPAN></FONT> <FONT face=3DArial =
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">of course =
now take me twice the time</SPAN></FONT> <FONT face=3DArial =
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">to finish =
the code</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">to</SPAN></FONT> <FONT =
face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">it could take if I got a change to work on it back in March. In =
any case, this is a release project and not something that can be =
patched.</SPAN></FONT></P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">BSD 4.1</SPAN></FONT> <FONT face=3DArial =
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">&#8211;</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">The problem is BSD =
committing too often.</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">Not much we can do about =
it except writing the data faster during commits. Implementing large =
I/Os may help, which is another thing I was going to do in =
March.</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">This</SPAN></FONT> <FONT =
face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">proves once</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">again that we should do =
as</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: =
10pt; FONT-FAMILY: Arial">I say</SPAN></FONT> <FONT face=3DArial =
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">and =
everything will be fine.</SPAN></FONT></P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: Arial">I could not reproduce the issue with =
bug</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: =
10pt; FONT-FAMILY: Arial">19041 (slow copy of the files) in the test =
environment.</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">RH5 is slower than RH3 but =
not by a factor of 10 as when copying on compile2 with dogfood =
as</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: =
10pt; FONT-FAMILY: Arial">the</SPAN></FONT> <FONT face=3DArial =
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">server.</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">One of the differences =
between</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">the</SPAN></FONT> <FONT =
face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">test system and dogfood is the link</SPAN></FONT> <FONT =
face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">aggregation on the dogfood. We need to</SPAN></FONT> <FONT =
face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">setup the test system with the same</SPAN></FONT> <FONT =
face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">link</SPAN></FONT> <FONT face=3DArial size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">configuration as dogfood =
and try again. We may also need to try the</SPAN></FONT> <FONT =
face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">same linux version as on compile2.</SPAN></FONT> <FONT =
face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">Fay, could you please make the changes</SPAN></FONT> <FONT =
face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial">in the test setup?</SPAN></FONT></P>=0A=
<P><FONT face=3DArial size=3D2><SPAN style=3D"FONT-SIZE: 10pt; =
FONT-FAMILY: =
Arial">Max</SPAN></FONT></P></DIV></DIV></DIV></DIV></BODY></HTML>
------_=_NextPart_001_01C7FF9E.CFD3E015--
