X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C75F6D.DCCC85E7@onstor-exch02.onstor.net>; Mon, 5 Mar 2007 14:33:06 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C75F6D.DCCC85E7"
Content-class: urn:content-classes:message
Subject: RE: mount -sync ?
Date: Mon, 5 Mar 2007 14:33:05 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E02A95CB1@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E02A95C3A@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: mount -sync ?
Thread-Index: AcdfNr+OYv3nwWLQQ1GwdvAzfpCMjQAAhWGkAAk9NhAAAX6ZoAACgvuA
From: "Eric Barrett" <eric.barrett@onstor.com>
To: "Steffen Thuemmel" <steffen.thuemmel@onstor.com>,
	"Huy Duong" <huy.duong@onstor.com>,
	"dl-se" <dl-se@onstor.com>,
	"dl-cstech" <dl-cstech@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C75F6D.DCCC85E7
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Steffen,

As far as I know this is absolutely fine.


> _____________________________________________=20
> From: 	Steffen Thuemmel =20
> Sent:	Monday, March 05, 2007 12:29 PM
> To:	Eric Barrett; Huy Duong; dl-se; dl-cstech
> Subject:	RE: mount -sync ?
>=20
> Eric,
>=20
> You> '> re right > ->  it is RMAN for a 10g RAC. Sorry for the =
confusion.
>=20
> If the DB files are located on local storage, can RMAN write the B/U =
to a NFS mounted directory (no_sync) ?
>=20
> Thanks,
> St,
>=20
> Steffen Thuemmel=20
> =20
> ONStor
> Schleussner Str. 42
> D-63263 Neu-Isenburg
> Germany
> =20
> telf.       +49 6102 884 84-0
> mobil.     +49 173 673 3434
> mail.       steffen.thuemmel@onstor.com
> web.       www.onstor.com <http://www.onstor.com/>=20
> =20
>=20
> _____________________________________________
> From: Eric Barrett=20
> Sent: Montag, 5. M=E4rz 2007 20:50
> To: Huy Duong; Steffen Thuemmel; dl-se; dl-cstech
> Subject: RE: mount -sync ?
>=20
>=20
> Steffen,
>=20
> Is this Paragon?  I am the (recent) owner of their performance case, =
4636.
>=20
> To add to Huy's comments, 'sync' is definitely expected to lower =
performance to the level you have documented on behalf of the customer =
for streaming writes.  I believe your customer is hung up on streaming =
write performance here, which is not completely applicable to the =
database workload he's doing.
>=20
> I am not aware of a requirement for RMAN to be mounted with 'sync'.  =
In fact, this seems highly unlikely, and a bit of googling turns up no =
such requirement nor any documentation of admins using 'sync' with RMAN =
on NFS.  If the customer believes this, I would definitely challenge him =
to provide documentation stating so.
>=20
> What definitely *does* require sync is RAC.  The customer has said he =
is using 10g RAC, which requires 'sync' for database log files, as they =
are shared between nodes, and all I/O is expected to be committed to =
disk by the server when the system call returns.
>=20
> I have already recommended to the customer that, in lieu of 'sync', he =
investigate "direct I/O".  This is supported on any modern Linux kernel =
(RedHat 3.0 and up, for instance), but it does require enabling the =
following init.ora paramter:
>=20
> FILESYSTEM_IO_OPTIONS=3Ddirectio
>=20
> Then you mount with =
"hard,vers=3D3,tcp,bg,nointr,rsize=3D32768,wsize=3D32768,noac,timeo=3D600=
".  (Do not deviate from those options -- for instance, do not tweak =
timeo, and do not use UDP instead of TCP.)
>=20
> Here is a document listing these requirements; they start at page 17:
>=20
> =
http://www.oracle.com/technology/products/database/clustering/pdf/EMC_Bes=
tPracticeforOracle.pdf=20
>=20
> While this is EMC-specific, I know that these recommendations hold =
true for other NFS servers, because they applied to NetApp as well.  In =
fact, I believe NetApp, in combination with Chuck Lever (a Linux kernel =
developer) came up with most of these recommendations.
>=20
> Finally, about nointr -- my experience with nointr is that enabling it =
only confuses things.  If you interrupt a process (such as oraserver) =
with Ctrl+C or kill, nointr will only delay the termination until a =
non-responsive NFS server comes back.  The only purpose which I know it =
serves is suppressing a bug in older Linux kernels where EAGAIN may =
occasionally be returned from read(2) and write(2) system calls, which =
crashes the database server instance (they're not supposed to return =
EAGAIN when called in a blocking fashion, so Oracle doesn't expect it, =
and thus goes down).
>=20
>=20
> -----Original Message-----
> From: Huy Duong=20
> Sent: Monday, March 05, 2007 7:25 AM
> To: Steffen Thuemmel; dl-se; dl-cstech
> Subject: RE: mount -sync ?
>=20
> Normally, an NFS client delays sending application write requests, =
allowing application processing to overlap with NFS write operations. An =
NFS client only causes an application to wait for writes to complete =
when the application closes or flushes a file. When a client sends write =
operations synchronously, however, the client causes applications to =
wait for each write operation to complete at the server. This results in =
much lower performance.>=20
>=20
> I am not familiar with RMAN, however what I know is that for databases =
we generally recommend using the nointr flag, but can anyone confirm =
this? This will allow NFS operations to NOT be interrupted while waiting =
for a response from the server.=20
> Does this slow down performance?
> =20
>=20
> -----Original Message-----
> From: Steffen Thuemmel
> Sent: Mon 3/5/2007 6:58 AM
> To: dl-se; dl-cstech
> Subject: mount -sync ?
> =20
> I've been called at a customer site to investigate slow performance.
>=20
> It turned out to be the mount option "sync" which dropped the =
performance. Using sync on a RH 4 client allows a read performance of =
about 2.5 MB/s. Without that option, one client was around 60 MB/s, with =
two clients I could max one IP port at about 120 MB/s.
>=20
> =20
>=20
> I know that "sync" impacts the performance, but I could not imagine =
that the penalty is that high.
>=20
> Here my questions:
>=20
> -          Does the "sync" option has any impact on the server (the =
BobCat) ?
>=20
> -          Is the performance penalty introduced by "sync" dependened =
on the client ? Do you have any experience with other clients ?
>=20
> =20
>=20
> The customer needs to use "sync" because they want to backup ORACLE =
DBs using  RMAN to a NFS share. RMAN in ORA 10 requires "sync" mounted =
Filesystems.  Do you have any suggestions for workarounds or =
alternatives ?
>=20
> =20
>=20
> Thanks for your help,
>=20
> =20
>=20
> St.
>=20
> =20
>=20
> Steffen Thuemmel=20
>=20
> Manager Systems Engineering
>=20
> Central Europe
>=20
> =20
>=20
> ONStor
>=20
> Schleussner Str. 42
>=20
> D-63263 Neu-Isenburg
>=20
> Germany
>=20
> =20
>=20
> telf.       +49 6102 884 84-0
>=20
> mobil.     +49 173 673 3434
>=20
> mail.       steffen.thuemmel@onstor.com =
<mailto:steffen.thuemmel@onstor.com>=20
>=20
> web.       www.onstor.com <http://www.onstor.com/>=20
>=20
> =20
>=20
>=20

------_=_NextPart_001_01C75F6D.DCCC85E7
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Diso-8859-1">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7652.24">
<TITLE>RE: mount -sync ?</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P><FONT SIZE=3D2 FACE=3D"Arial">Steffen,</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">As far as I know this is absolutely =
fine.</FONT>
</P>
<BR>

<P><FONT SIZE=3D1 =
FACE=3D"Tahoma">_____________________________________________ </FONT>

<BR><B><FONT SIZE=3D1 FACE=3D"Tahoma">From: &nbsp;</FONT></B> <FONT =
SIZE=3D1 FACE=3D"Tahoma">Steffen Thuemmel&nbsp; </FONT>

<BR><B><FONT SIZE=3D1 FACE=3D"Tahoma">Sent:&nbsp;&nbsp;</FONT></B> <FONT =
SIZE=3D1 FACE=3D"Tahoma">Monday, March 05, 2007 12:29 PM</FONT>

<BR><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">To:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></B> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Eric Barrett; Huy Duong; dl-se; dl-cstech</FONT>

<BR><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Subject:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT>=
</B> <FONT SIZE=3D1 FACE=3D"Tahoma">RE: mount -sync ?</FONT>
</P>

<P><FONT COLOR=3D"#1F497D" FACE=3D"Calibri">Eric,</FONT>
</P>

<P><FONT COLOR=3D"#1F497D" FACE=3D"Calibri">You&#8217;re right &#8211; =
it is RMAN for a 10g RAC. Sorry for the confusion.</FONT>
</P>

<P><FONT COLOR=3D"#1F497D" FACE=3D"Calibri">If the DB files are located =
on local storage, can RMAN write the B/U to a NFS mounted directory =
(no_sync) ?</FONT>
</P>

<P><FONT COLOR=3D"#1F497D" FACE=3D"Calibri">Thanks,</FONT>

<BR><FONT COLOR=3D"#1F497D" FACE=3D"Calibri">St,</FONT>
</P>

<P><B><FONT COLOR=3D"#1F497D" FACE=3D"Arial">Steffen Thuemmel</FONT></B> =


<BR><FONT COLOR=3D"#1F497D" FACE=3D"Arial">=A0</FONT>

<BR><B><FONT COLOR=3D"#1F497D" FACE=3D"Arial">ONStor</FONT></B>

<BR><FONT COLOR=3D"#1F497D" FACE=3D"Arial">Schleussner Str. 42</FONT>

<BR><FONT COLOR=3D"#1F497D" FACE=3D"Arial">D-63263 Neu-Isenburg</FONT>

<BR><FONT COLOR=3D"#1F497D" FACE=3D"Arial">Germany</FONT>

<BR><FONT COLOR=3D"#1F497D" FACE=3D"Arial">=A0</FONT>

<BR><B><FONT COLOR=3D"#1F497D" FACE=3D"Arial">telf.</FONT><FONT =
COLOR=3D"#1F497D" SIZE=3D2 FACE=3D"Arial">=A0=A0=A0=A0=A0=A0</FONT> =
<FONT COLOR=3D"#1F497D" SIZE=3D1 FACE=3D"Arial">+49 6102 884 =
84-0</FONT></B>

<BR><B><FONT COLOR=3D"#1F497D" FACE=3D"Arial">mobil. =A0=A0=A0 =
</FONT><FONT COLOR=3D"#1F497D" SIZE=3D1 FACE=3D"Arial">+49 173 673 =
3434</FONT></B>

<BR><B><FONT COLOR=3D"#1F497D" FACE=3D"Arial">mail.=A0=A0=A0=A0=A0=A0 =
</FONT></B><A HREF=3D"mailto:steffen.thuemmel@onstor.com"><B><U><FONT =
COLOR=3D"#0000FF" =
FACE=3D"Arial">steffen.thuemmel@onstor.com</FONT></U></B></A><B></B>

<BR><B><FONT COLOR=3D"#1F497D" FACE=3D"Arial">web.=A0=A0=A0=A0=A0=A0 =
</FONT></B><A HREF=3D"http://www.onstor.com/"><B><U><FONT =
COLOR=3D"#0000FF" =
FACE=3D"Arial">www.onstor.com</FONT></U></B></A><B></B>

<BR><FONT COLOR=3D"#1F497D" FACE=3D"Arial">=A0</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Eric Barrett<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Montag, 5. M=E4rz 2007 20:50<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Huy Duong; Steffen Thuemmel; dl-se; dl-cstech<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: mount -sync ?</FONT>
</P>
<BR>

<P><FONT SIZE=3D2 FACE=3D"Arial">Steffen,</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Is this Paragon?&nbsp; I am the =
(recent) owner of their performance case, 4636.</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">To add to Huy's comments, 'sync' is =
definitely expected to lower performance to the level you have =
documented on behalf of the customer for streaming writes.&nbsp; I =
believe your customer is hung up on streaming write performance here, =
which is not completely applicable to the database workload he's =
doing.</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">I am not aware of a requirement for =
RMAN to be mounted with 'sync'.&nbsp; In fact, this seems highly =
unlikely, and a bit of googling turns up no such requirement nor any =
documentation of admins using 'sync' with RMAN on NFS.&nbsp; If the =
customer believes this, I would definitely challenge him to provide =
documentation stating so.</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">What definitely *does* require sync is =
RAC.&nbsp; The customer has said he is using 10g RAC, which requires =
'sync' for database log files, as they are shared between nodes, and all =
I/O is expected to be committed to disk by the server when the system =
call returns.</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">I have already recommended to the =
customer that, in lieu of 'sync', he investigate &quot;direct =
I/O&quot;.&nbsp; This is supported on any modern Linux kernel (RedHat =
3.0 and up, for instance), but it does require enabling the following =
init.ora paramter:</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">FILESYSTEM_IO_OPTIONS=3Ddirectio</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Then you mount with =
&quot;hard,vers=3D3,tcp,bg,nointr,rsize=3D32768,wsize=3D32768,noac,timeo=3D=
600&quot;.&nbsp; (Do not deviate from those options -- for instance, do =
not tweak timeo, and do not use UDP instead of TCP.)</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Here is a document listing these =
requirements; they start at page 17:</FONT>
</P>

<P><A =
HREF=3D"http://www.oracle.com/technology/products/database/clustering/pdf=
/EMC_BestPracticeforOracle.pdf"><U><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">http://www.oracle.com/technology/products/database/cluster=
ing/pdf/EMC_BestPracticeforOracle.pdf</FONT></U></A><FONT SIZE=3D2 =
FACE=3D"Arial"> </FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">While this is EMC-specific, I know that =
these recommendations hold true for other NFS servers, because they =
applied to NetApp as well.&nbsp; In fact, I believe NetApp, in =
combination with Chuck Lever (a Linux kernel developer) came up with =
most of these recommendations.</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Finally, about nointr -- my experience =
with nointr is that enabling it only confuses things.&nbsp; If you =
interrupt a process (such as oraserver) with Ctrl+C or kill, nointr will =
only delay the termination until a non-responsive NFS server comes =
back.&nbsp; The only purpose which I know it serves is suppressing a bug =
in older Linux kernels where EAGAIN may occasionally be returned from =
read(2) and write(2) system calls, which crashes the database server =
instance (they're not supposed to return EAGAIN when called in a =
blocking fashion, so Oracle doesn't expect it, and thus goes =
down).</FONT></P>
<BR>

<P><FONT SIZE=3D2 FACE=3D"Arial">-----Original Message-----</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">From: Huy Duong </FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Sent: Monday, March 05, 2007 7:25 =
AM</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">To: Steffen Thuemmel; dl-se; =
dl-cstech</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Subject: RE: mount -sync ?</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Normally, an NFS client delays sending =
application write requests, allowing application processing to overlap =
with NFS write operations. An NFS client only causes an application to =
wait for writes to complete when the application closes or flushes a =
file. When a client sends write operations synchronously, however, the =
client causes applications to wait for each write operation to complete =
at the server. This results in much lower performance.</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">I am not familiar with RMAN, however =
what I know is that for databases we generally recommend using the =
nointr flag, but can anyone confirm this? This will allow NFS operations =
to NOT be interrupted while waiting for a response from the server. =
</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Does this slow down performance?</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">-----Original Message-----</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">From: Steffen Thuemmel</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Sent: Mon 3/5/2007 6:58 AM</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">To: dl-se; dl-cstech</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Subject: mount -sync ?</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">I've been called at a customer site to =
investigate slow performance.</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">It turned out to be the mount option =
&quot;sync&quot; which dropped the performance. Using sync on a RH 4 =
client allows a read performance of about 2.5 MB/s. Without that option, =
one client was around 60 MB/s, with two clients I could max one IP port =
at about 120 MB/s.</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">I know that &quot;sync&quot; impacts =
the performance, but I could not imagine that the penalty is that =
high.</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Here my questions:</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Arial">-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
Does the &quot;sync&quot; option has any impact on the server (the =
BobCat) ?</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Arial">-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
Is the performance penalty introduced by &quot;sync&quot; dependened on =
the client ? Do you have any experience with other clients ?</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">The customer needs to use =
&quot;sync&quot; because they want to backup ORACLE DBs using&nbsp; RMAN =
to a NFS share. RMAN in ORA 10 requires &quot;sync&quot; mounted =
Filesystems.&nbsp; Do you have any suggestions for workarounds or =
alternatives ?</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Thanks for your help,</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">St.</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Steffen Thuemmel </FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Manager Systems Engineering</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Central Europe</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">ONStor</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Schleussner Str. 42</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">D-63263 Neu-Isenburg</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Germany</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Arial">telf.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; +49 6102 884 =
84-0</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">mobil.&nbsp;&nbsp;&nbsp;&nbsp; +49 173 =
673 3434</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Arial">mail.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
steffen.thuemmel@onstor.com &lt;</FONT><A =
HREF=3D"mailto:steffen.thuemmel@onstor.com"><U><FONT COLOR=3D"#0000FF" =
SIZE=3D2 =
FACE=3D"Arial">mailto:steffen.thuemmel@onstor.com</FONT></U></A><FONT =
SIZE=3D2 FACE=3D"Arial">&gt; </FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Arial">web.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </FONT><A =
HREF=3D"file://www.onstor.com"><U><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">www.onstor.com</FONT></U></A><FONT SIZE=3D2 =
FACE=3D"Arial"> &lt;</FONT><A HREF=3D"http://www.onstor.com/"><U><FONT =
COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">http://www.onstor.com/</FONT></U></A><FONT SIZE=3D2 =
FACE=3D"Arial">&gt; </FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">&nbsp;</FONT>
</P>
<BR>

</BODY>
</HTML>
------_=_NextPart_001_01C75F6D.DCCC85E7--
