X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7F6EE.46C9121D@onstor-exch02.onstor.net>; Fri, 14 Sep 2007 08:42:45 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7F6EE.46C9121D"
Content-class: urn:content-classes:message
Subject: RE: Free BSD for Yahoo test
Date: Fri, 14 Sep 2007 08:42:44 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E0587B1D6@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E0346A97B@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Free BSD for Yahoo test
Thread-Index: Acf26Im0tCY1uYWmRo2snDYAjReFggAAA2EgAACkK2AAAHMNIA==
From: "Eric Barrett" <eric.barrett@onstor.com>
To: "John Rogers" <john.rogers@onstor.com>,
	"Brian Montero" <brian.montero@onstor.com>
Cc: "Fay Chong" <fay.chong@onstor.com>,
	"Paul Hammer" <paul.hammer@onstor.com>,
	"Brian Baker" <brian.baker@onstor.com>,
	"Maxim Kozlovsky" <maxim.kozlovsky@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7F6EE.46C9121D
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

"Something is majorly wrong with NFS."

I don't know why, but for some reason, NFS is the first thing to break
every time a kernel is revved, regardless of operating system.

In the Linux world, distribution maintainers like RedHat and Debian
typically add extra patches to their kernels to make up for flaws in the
base code.  Usually they're from Trond Myklebust and Chuck Lever, two
kernel developers who run http://linux-nfs.org.  At one point they were
both on NetApp's payroll, but I think Chuck has since moved to Oracle,
which is understandable since they benefit greatly from his work.

A good example of how badly kernels can break -- we installed stock
Linux 2.6.22 on beast (the data warehouse system) and dd performance
dropped from 110MB/sec to 2.  Yes, two.

Since Yahoo is the only company of any size who uses FreeBSD, the
situation is worse.  There aren't a million splinters of FreeBSD out
there you can try until you find one that works.  And the only company
in the industry with experience tuning FreeBSD is...Yahoo.  (But they're
very good at it.)

I used to work with the Yahoo support team at NetApp, and I can tell you
NetApp did very little performance troubleshooting.  They sold on
features and price, and let Yahoo figure the performance stuff out.  It
may not be a viable approach for us, but that's what happened.

Good luck.  You'll need it.  =3D(


> _____________________________________________=20
> From: 	John Rogers =20
> Sent:	Friday, September 14, 2007 9:21 AM
> To:	John Rogers; Brian Montero
> Cc:	Fay Chong; Paul Hammer; Brian Baker; Maxim Kozlovsky; Andy
> Sharp; Eric Barrett
> Subject:	RE: Free BSD for Yahoo test
>=20
> Including Eric B.
>=20
> _____________________________________________
> From: John Rogers=20
> Sent: Friday, September 14, 2007 9:12 AM
> To: Brian Montero
> Cc: Fay Chong; Paul Hammer; Brian Baker; Maxim Kozlovsky; Andy Sharp
> Subject: RE: Free BSD for Yahoo test
>=20
> Brian,
>=20
> We are hearing this story too much!! We have a performance problem
> with modern kernels. The following list of Operating Systems have been
> reported to SUCK at nfs.
>=20
> Red Hat EL 4
> Red Hat EL 5
> Ubuntu Feisty Fawn
> Debian the latest version?
> SUSE 10
> FreeBSD 6.2
>=20
> Either something is majorly wrong with NFS or we have a problem
> handling it.
>=20
> While this latest occurrence may jeopardize our Yahoo deal, if we
> don't figure out what the problem is and how to fix it, we will be in
> deep doo doo in similar situations.
>=20
> I had reported previously that there were several red hat bugs that
> definitely could cause this. However this is starting to be far too
> prevelant. I know recently that NFS has become more complex adding
> additional overhead for checking various operations and supporting
> more file acls.
>=20
> John
>=20
> _____________________________________________
> From: Fay Chong=20
> Sent: Friday, September 14, 2007 9:02 AM
> To: John Rogers
> Cc: Brian Montero; Fay Chong
> Subject: Free BSD for Yahoo test
>=20
> John,
> As I mentioned yesterday, I see about 15 MB/sec doing dd with
> FreeBSD6.2 while using RHEL3 with the same dd statement yields about
> 50MB/sec. I downloaded FreeBSD 4.11 so I will burn the CD's and
> install on 10.3.62.21. I'll keep you posted.
> Thanks
> Fay
>=20
> Fay Chong
> Sr. Performance Engineer
> ONStor, Inc.
> fay.chong@onstor.com
> 408.376.3130 (w)
>=20
>=20

------_=_NextPart_001_01C7F6EE.46C9121D
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7652.24">
<TITLE>RE: Free BSD for Yahoo test</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P><FONT SIZE=3D2 FACE=3D"Arial">&quot;Something is majorly wrong with =
NFS.&quot;</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">I don't know why, but for some reason, =
NFS is the first thing to break every time a kernel is revved, =
regardless of operating system.</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">In the Linux world, distribution =
maintainers like RedHat and Debian typically add extra patches to their =
kernels to make up for flaws in the base code.&nbsp; Usually they're =
from Trond Myklebust and Chuck Lever, two kernel developers who run =
</FONT><A HREF=3D"http://linux-nfs.org"><U><FONT COLOR=3D"#0000FF" =
SIZE=3D2 FACE=3D"Arial">http://linux-nfs.org</FONT></U></A><FONT =
SIZE=3D2 FACE=3D"Arial">.&nbsp; At one point they were both on NetApp's =
payroll, but I think Chuck has since moved to Oracle, which is =
understandable since they benefit greatly from his work.</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">A good example of how badly kernels can =
break -- we installed stock Linux 2.6.22 on beast (the data warehouse =
system) and dd performance dropped from 110MB/sec to 2.&nbsp; Yes,<B> =
two</B>.</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Since Yahoo is the only company of any =
size who uses FreeBSD, the situation is worse.&nbsp; There aren't a =
million splinters of FreeBSD out there you can try until you find one =
that works.&nbsp; And the only company in the industry with experience =
tuning FreeBSD is...Yahoo.&nbsp; (But they're very good at =
it.)</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">I used to work with the Yahoo support =
team at NetApp, and I can tell you NetApp did very little performance =
troubleshooting.&nbsp; They sold on features and price, and let Yahoo =
figure the performance stuff out.&nbsp; It may not be a viable approach =
for us, but that's what happened.</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Good luck.&nbsp; You'll need it.&nbsp; =
=3D(</FONT>
</P>
<BR>

<P><FONT SIZE=3D1 =
FACE=3D"Tahoma">_____________________________________________ </FONT>

<BR><B><FONT SIZE=3D1 FACE=3D"Tahoma">From: &nbsp;</FONT></B> <FONT =
SIZE=3D1 FACE=3D"Tahoma">John Rogers&nbsp; </FONT>

<BR><B><FONT SIZE=3D1 FACE=3D"Tahoma">Sent:&nbsp;&nbsp;</FONT></B> <FONT =
SIZE=3D1 FACE=3D"Tahoma">Friday, September 14, 2007 9:21 AM</FONT>

<BR><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">To:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></B> <FONT SIZE=3D1 =
FACE=3D"Tahoma">John Rogers; Brian Montero</FONT>

<BR><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Cc:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></B> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Fay Chong; Paul Hammer; Brian Baker; Maxim Kozlovsky; =
Andy Sharp; Eric Barrett</FONT>

<BR><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Subject:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT>=
</B> <FONT SIZE=3D1 FACE=3D"Tahoma">RE: Free BSD for Yahoo test</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Including Eric =
B.</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> John Rogers<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Friday, September 14, 2007 9:12 AM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Brian Montero<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Cc:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Fay Chong; Paul Hammer; Brian Baker; Maxim Kozlovsky; =
Andy Sharp<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: Free BSD for Yahoo test</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Brian,</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">We are hearing this =
story too much!! We have a performance problem with modern kernels. The =
following list of Operating Systems have been reported to SUCK at =
nfs.</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Red Hat EL 4</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Red Hat EL 5</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Ubuntu Feisty =
Fawn</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Debian the latest =
version?</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">SUSE 10</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">FreeBSD 6.2</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Either something is =
majorly wrong with NFS or we have a problem handling it.</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">While this latest =
occurrence may jeopardize our Yahoo deal, if we don&#8217;t figure out =
what the problem is and how to fix it, we will be in deep doo doo in =
similar situations.</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">I had reported =
previously that there were several red hat bugs that definitely could =
cause this. However this is starting to be far too prevelant. I know =
recently that NFS has become more complex adding additional overhead for =
checking various operations and supporting more file acls.</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">John</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Fay Chong<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Friday, September 14, 2007 9:02 AM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> John Rogers<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Cc:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Brian Montero; Fay Chong<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> Free BSD for Yahoo test</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">John,</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">As I mentioned yesterday, I see about =
15 MB/sec doing dd with FreeBSD6.2 while using RHEL3 with the same dd =
statement yields about 50MB/sec. I downloaded FreeBSD 4.11 so I will =
burn the CD&#8217;s and install on 10.3.62.21. I&#8217;ll keep you =
posted.</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Thanks</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Fay</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Fay Chong</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Sr. Performance Engineer</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">ONStor, Inc.</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">fay.chong@onstor.com</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">408.376.3130 (w)</FONT>
</P>
<BR>

</BODY>
</HTML>
------_=_NextPart_001_01C7F6EE.46C9121D--
