X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C75F5F.66AADC50@onstor-exch02.onstor.net>; Mon, 5 Mar 2007 12:49:35 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C75F5F.66AADC50"
Content-class: urn:content-classes:message
Subject: RE: mount -sync ?
Date: Mon, 5 Mar 2007 12:49:35 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E02A95BE5@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04EF30A65@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: mount -sync ?
Thread-Index: AcdfNr+OYv3nwWLQQ1GwdvAzfpCMjQAAhWGkAAk9NhA=
From: "Eric Barrett" <eric.barrett@onstor.com>
To: "Huy Duong" <huy.duong@onstor.com>,
	"Steffen Thuemmel" <steffen.thuemmel@onstor.com>,
	"dl-se" <dl-se@onstor.com>,
	"dl-cstech" <dl-cstech@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C75F5F.66AADC50
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Steffen,

Is this Paragon?  I am the (recent) owner of their performance case,
4636.

To add to Huy's comments, 'sync' is definitely expected to lower
performance to the level you have documented on behalf of the customer
for streaming writes.  I believe your customer is hung up on streaming
write performance here, which is not completely applicable to the
database workload he's doing.

I am not aware of a requirement for RMAN to be mounted with 'sync'.  In
fact, this seems highly unlikely, and a bit of googling turns up no such
requirement nor any documentation of admins using 'sync' with RMAN on
NFS.  If the customer believes this, I would definitely challenge him to
provide documentation stating so.

What definitely *does* require sync is RAC.  The customer has said he is
using 10g RAC, which requires 'sync' for database log files, as they are
shared between nodes, and all I/O is expected to be committed to disk by
the server when the system call returns.

I have already recommended to the customer that, in lieu of 'sync', he
investigate "direct I/O".  This is supported on any modern Linux kernel
(RedHat 3.0 and up, for instance), but it does require enabling the
following init.ora paramter:

FILESYSTEM_IO_OPTIONS=3Ddirectio

Then you mount with
"hard,vers=3D3,tcp,bg,nointr,rsize=3D32768,wsize=3D32768,noac,timeo=3D600=
".  (Do
not deviate from those options -- for instance, do not tweak timeo, and
do not use UDP instead of TCP.)

Here is a document listing these requirements; they start at page 17:

http://www.oracle.com/technology/products/database/clustering/pdf/EMC_Be
stPracticeforOracle.pdf=20

While this is EMC-specific, I know that these recommendations hold true
for other NFS servers, because they applied to NetApp as well.  In fact,
I believe NetApp, in combination with Chuck Lever (a Linux kernel
developer) came up with most of these recommendations.

Finally, about nointr -- my experience with nointr is that enabling it
only confuses things.  If you interrupt a process (such as oraserver)
with Ctrl+C or kill, nointr will only delay the termination until a
non-responsive NFS server comes back.  The only purpose which I know it
serves is suppressing a bug in older Linux kernels where EAGAIN may
occasionally be returned from read(2) and write(2) system calls, which
crashes the database server instance (they're not supposed to return
EAGAIN when called in a blocking fashion, so Oracle doesn't expect it,
and thus goes down).


-----Original Message-----
From: Huy Duong=20
Sent: Monday, March 05, 2007 7:25 AM
To: Steffen Thuemmel; dl-se; dl-cstech
Subject: RE: mount -sync ?

Normally, an NFS client delays sending application write requests,
allowing application processing to overlap with NFS write operations. An
NFS client only causes an application to wait for writes to complete
when the application closes or flushes a file. When a client sends write
operations synchronously, however, the client causes applications to
wait for each write operation to complete at the server. This results in
much lower performance.

I am not familiar with RMAN, however what I know is that for databases
we generally recommend using the nointr flag, but can anyone confirm
this? This will allow NFS operations to NOT be interrupted while waiting
for a response from the server.=20
Does this slow down performance?
=20

-----Original Message-----
From: Steffen Thuemmel
Sent: Mon 3/5/2007 6:58 AM
To: dl-se; dl-cstech
Subject: mount -sync ?
=20
I've been called at a customer site to investigate slow performance.

It turned out to be the mount option "sync" which dropped the
performance. Using sync on a RH 4 client allows a read performance of
about 2.5 MB/s. Without that option, one client was around 60 MB/s, with
two clients I could max one IP port at about 120 MB/s.

=20

I know that "sync" impacts the performance, but I could not imagine that
the penalty is that high.

Here my questions:

-          Does the "sync" option has any impact on the server (the
BobCat) ?

-          Is the performance penalty introduced by "sync" dependened on
the client ? Do you have any experience with other clients ?

=20

The customer needs to use "sync" because they want to backup ORACLE DBs
using  RMAN to a NFS share. RMAN in ORA 10 requires "sync" mounted
Filesystems.  Do you have any suggestions for workarounds or
alternatives ?

=20

Thanks for your help,

=20

St.

=20

Steffen Thuemmel=20

Manager Systems Engineering

Central Europe

=20

ONStor

Schleussner Str. 42

D-63263 Neu-Isenburg

Germany

=20

telf.       +49 6102 884 84-0

mobil.     +49 173 673 3434

mail.       steffen.thuemmel@onstor.com
<mailto:steffen.thuemmel@onstor.com>=20

web.       www.onstor.com <http://www.onstor.com/>=20

=20



------_=_NextPart_001_01C75F5F.66AADC50
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7652.24">
<TITLE>RE: mount -sync ?</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Steffen,</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Is this =
Paragon?&nbsp; I am the (recent) owner of their performance case, =
4636.</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">To add to Huy's =
comments, 'sync' is definitely expected to lower performance to the =
level you have documented on behalf of the customer</FONT><FONT SIZE=3D2 =
FACE=3D"Arial"> for streaming writes.&nbsp; I believe your customer is =
hung up on streaming write performance here, which is not completely =
applicable to the database workload he's doing.</FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I am not aware of =
a requirement for RMAN to be mounted with 'sync'.&nbsp; In fact, this =
seems highly unlikely, and a bit of googling turns up no such =
requirement nor any documentation of admins using 'sync' with RMAN on =
NFS.&nbsp; If the customer believes this, I would definitely challenge =
him to provide documentation stating so.</FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">What definitely =
*does* require sync is RAC.&nbsp; The customer has said he is using 10g =
RAC, which requires 'sync' for database log files, as they are shared =
between nodes, and all I/O is expected to be committed to disk by the =
server when the system call returns.</FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I have already =
recommended to the customer that, in lieu of 'sync', he investigate =
&quot;direct I/O&quot;.&nbsp; This is supported on any modern Linux =
kernel (RedHat 3.0 and up, for instance), but it does require enabling =
the following init.ora paramter:</FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">FILESYSTEM_IO_OPTIONS=3Ddirectio</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Then you mount =
with =
&quot;hard,vers=3D3,tcp,bg,nointr,rsize=3D32768,wsize=3D32768,noac,timeo=3D=
600&quot;.&nbsp; (Do not deviate from those options -- for instance, do =
not tweak timeo, and do not use UDP instead of TCP.)</FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Here is a document =
listing these requirements; they start at page 17:</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"></SPAN><A =
HREF=3D"http://www.oracle.com/technology/products/database/clustering/pdf=
/EMC_BestPracticeforOracle.pdf"><SPAN LANG=3D"en-us"><U><FONT =
COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">http://www.oracle.com/technology/products/database/cluster=
ing/pdf/EMC_BestPracticeforOracle.pdf</FONT></U></SPAN></A><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial"> </FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">While this is =
EMC-specific, I know that these</FONT> <FONT SIZE=3D2 =
FACE=3D"Arial">recommendations hold true for other NFS servers, because =
they applied to NetApp as well.&nbsp; In fact, I believe NetApp, in =
combination with Chuck Lever (a Linux kernel developer) came up with =
most of these recommendations.</FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Finally, about =
nointr -- my experience with nointr is that enabling it only confuses =
things.&nbsp; If you interrupt a process (such as oraserver) with Ctrl+C =
or kill, nointr will only delay the termination until a non-responsive =
NFS server comes back.&nbsp; The only purpose which I know it serves is =
suppressing a bug in older Linux kernels where EAGAIN may occasionally =
be returned from read(2) and write(2) system calls, which crashes the =
database server instance (they're not supposed to return EAGAIN when =
called in a blocking fashion, so Oracle doesn't expect it, and thus goes =
down).</FONT></SPAN></P>
<BR>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">-----Original =
Message-----</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">From: Huy Duong =
</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Sent: Monday, =
March 05, 2007 7:25 AM</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">To: Steffen =
Thuemmel; dl-se; dl-cstech</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Subject: RE: =
mount -sync ?</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Normally, an NFS =
client delays sending application write requests, allowing application =
processing to overlap with NFS write operations. An NFS client only =
causes an application to wait for writes to complete when the =
application closes or flushes a file. When a client sends write =
operations synchronously, however, the client causes applications to =
wait for each write operation to complete at the server. This results in =
much lower performance.</FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I am not familiar =
with RMAN, however what I know is that for databases we generally =
recommend using the nointr flag, but can anyone confirm this? This will =
allow NFS operations to NOT be interrupted while waiting for a response =
from the server. </FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Does this slow =
down performance?</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">-----Original =
Message-----</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">From: Steffen =
Thuemmel</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Sent: Mon =
3/5/2007 6:58 AM</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">To: dl-se; =
dl-cstech</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Subject: mount =
-sync ?</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;</FONT></SPAN>

<BR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I've been called =
at a customer site to investigate slow performance.</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">It turned out to =
be the mount option &quot;sync&quot; which dropped the performance. =
Using sync on a RH 4 client allows a read performance of about 2.5 MB/s. =
Without that option, one client was around 60 MB/s, with two clients I =
could max one IP port at about 120 MB/s.</FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I know that =
&quot;sync&quot; impacts the performance, but I could not imagine that =
the penalty is that high.</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Here my =
questions:</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
Does the &quot;sync&quot; option has any impact on the server (the =
BobCat) ?</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
Is the performance penalty introduced by &quot;sync&quot; dependened on =
the client ? Do you have any experience with other clients =
?</FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">The customer needs =
to use &quot;sync&quot; because they want to backup ORACLE DBs =
using&nbsp; RMAN to a NFS share. RMAN in ORA 10 requires =
&quot;sync&quot; mounted Filesystems.&nbsp; Do you have any suggestions =
for workarounds or alternatives ?</FONT></SPAN></P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Thanks for your =
help,</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">St.</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Steffen Thuemmel =
</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Manager Systems =
Engineering</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Central =
Europe</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">ONStor</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Schleussner Str. =
42</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">D-63263 =
Neu-Isenburg</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Germany</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">telf.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; +49 6102 884 =
84-0</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">mobil.&nbsp;&nbsp;&nbsp;&nbsp; +49 173 673 =
3434</FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">mail.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
steffen.thuemmel@onstor.com &lt;</FONT></SPAN><A =
HREF=3D"mailto:steffen.thuemmel@onstor.com"><SPAN =
LANG=3D"en-us"><U><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">mailto:steffen.thuemmel@onstor.com</FONT></U></SPAN></A><S=
PAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">&gt; </FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">web.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </FONT></SPAN><A =
HREF=3D"file://www.onstor.com"><SPAN LANG=3D"en-us"><U><FONT =
COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">www.onstor.com</FONT></U></SPAN></A><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial"> &lt;</FONT></SPAN><A =
HREF=3D"http://www.onstor.com/"><SPAN LANG=3D"en-us"><U><FONT =
COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">http://www.onstor.com/</FONT></U></SPAN></A><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">&gt; </FONT></SPAN>
</P>

<P><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;</FONT></SPAN>
</P>
<BR>

</BODY>
</HTML>
------_=_NextPart_001_01C75F5F.66AADC50--
