X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7FF9C.F21FCC23@onstor-exch02.onstor.net>; Tue, 25 Sep 2007 09:53:14 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7FF9C.F21FCC23"
Content-class: urn:content-classes:message
Subject: RE: Lab reproduction for UK cases - RE: British Red Cross Case 5940
Date: Tue, 25 Sep 2007 09:53:14 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E05B467E3@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E05B466E3@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Lab reproduction for UK cases - RE: British Red Cross Case 5940
Thread-Index: Acf+9ejr/Ai3WViXTEqOj01eA2vKQwAg/jkgAAWJgIAAAw5PMA==
From: "Rich LaReau" <rich.lareau@onstor.com>
To: "Carolyn Sims" <carolyn.sims@onstor.com>,
	"Brian Nguyen" <brian.nguyen@onstor.com>,
	"Fay Chong" <fay.chong@onstor.com>,
	"Paul Hammer" <paul.hammer@onstor.com>,
	"Jonathan Goldick" <jonathan.goldick@onstor.com>,
	"Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>,
	"Brian Montero" <brian.montero@onstor.com>,
	"John Rogers" <john.rogers@onstor.com>,
	"Brian Nguyen" <brian.nguyen@onstor.com>,
	"Brian Bliem" <brian.bliem@onstor.com>
Cc: "Caeli Collins" <caeli.collins@onstor.com>,
	"Neil Cook" <neil.cook@onstor.com>,
	"David Crispin" <david.crispin@onstor.com>,
	"Tim O'Callaghan" <tim.ocallaghan@onstor.com>,
	"Sandrine Boulanger" <sandrine.boulanger@onstor.com>,
	"Paul Hammer" <paul.hammer@onstor.com>,
	"Jonathan Goldick" <jonathan.goldick@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7FF9C.F21FCC23
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Hi all,

Carolyn asked me to send this around cc-ing everybody, as it appears =
that we have a few loops going on at the same time.  Initial customer =
setup is below, followed by Max's findings of Fay's test comparisons for =
RH kernels 2.4 (fast) and 2.6 (slow) from a simple dd write.

Rich

_____________________________________________
From: Tim O'Callaghan=20
Sent: Tuesday, September 25, 2007 8:43 AM
To: Laurence Armiger
Cc: Carolyn Sims; Gerald Sommariva; Caeli Collins; Neil Cook; David =
Crispin
Subject: Re: British Red Cross Case 5940

Laurence,

HELP required.

I have finally managed to get the authority to reboot the Linux =
DS-System server in British Red Cross.=20

Earlier today I rebooted the system in Backup Technology and their =
performance levels resumed.

Rebooting the linux system has returned performance to its previous =
levels as Gerald suggested it might.

All the systems supported by Backup Technology are;

Kernel 2.6.9-5.Elsmp

RedHat Enterprise 4.

The volumes are mounted with the following options; nfs =
defaults,intr,rsize=3D32768,wsize=3D32768,async,nfsvers=3D3

Asigra recommend Red Hat Enterprise 4 which includes the 2.6.9-5.Elsmp =
kernel.

Any Asigra vault of significance will require an NFS file system, for =
the following reasons

1.  PostGreSql is 'better than MS SQL' in stability, efficiency and =
cost.
2.  I believe Linux can manage a greater than number of files than W2K3 =
server.
A typical Asigra 10TB vault will contain > 100 million files. (10 =
million files per TB)

Backup Technology are now at a crossroad with respect to the ONstor =
product - they feel that "since NFS is an open standard - it would never =
cause a problem, ONStor should work" Simon has intimated that unless we =
can get this fixed he has sold his last Bobcat - his retirement plan is =
based on moving the Asigra CIFs user base to NFS .

You will not believe the crap I am now enduring about this - can we put =
pressure on the engineering team to get the later kernel supported - =
given the Asigra recommendation of RedHat 4 I consider it unrealistic to =
think we can persuade Backup Technology to downgrade their DS-System =
operating system patch revison level.

To retain any hopes of future deals with these guys we need to get this =
fixed ASAP.

Regards

Tim

 http://www.onstor.com/images/footer/onstor_signature.gif


Tim O'Callaghan
Senior S.E.

ONStor, Inc.
office:  +44 1189 635866
mobile: +44 7767 435472=20

tim.ocallaghan@onstor.com
http://www.onstor.com
 http://www.onstor.com/images/footer/storage_mag_product_of_year.gif=09

_____________________________________________
From: Maxim Kozlovsky=20
Sent: Monday, September 24, 2007 4:38 PM
To: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Andy =
Sharp
Cc: Brian Montero; John Rogers
Subject: RE: Wireshark Red Hat 3 and 5 NFS sequential performance=20

Ok, the problem with the writes is unaligned and out of order writes =
with 2.6.18 kernel. See a trace at ~maximk/tmp/rh5.eth, frame =
70232/70247/70267/70287 and so on. We happen to handle this particularly =
badly as we don't coalesce the writes and read the partial blocks from =
disk. To make the matters worse, we hold the lock for the duration of =
this operation. At first I thought that I did not run the test right and =
did not delete the file, but Linux kernel really does that even with a =
new file. The KPI files are in ~maximk/tmp if anybody wants to take a =
look.

[maximk@compile2 tmp]$ =
/n/build-trees/R3.0.0.0/EverON-3.0.0.0/nfx-tree/Build/x86utils/kpi-ui -c =
NFS_V3_WRITE -f rh5.kpi=20
kpi                                        calls        total seconds    =
     seconds/call
NFS_V3_WRITE                              131089          1752.865365    =
         0.013371
  NFS_OPEN_HANDLE                           131089          1609.455383  =
           0.012277
  PKT_RPC_REASM                             131089            41.302495  =
           0.000315
  DCACHE_WRITE                              131089            96.842795  =
           0.000738
  PKT_ACPU_QUEUE                            131089             0.431612  =
           0.000003
  NFS_REPLY                                 131089             0.476795  =
           0.000003
  PKT_NCPU_QUEUE                            131089             0.186616  =
           0.000001
  PKT_TX                                    131089             0.338352  =
           0.000002

[maximk@compile2 tmp]$ =
/n/build-trees/R3.0.0.0/EverON-3.0.0.0/nfx-tree/Build/x86utils/kpi-ui -c =
NFS_V3_WRITE -f rh3.kpi
kpi                                        calls        total seconds    =
     seconds/call
NFS_V3_WRITE                              131072            56.102047    =
         0.000428
  NFS_OPEN_HANDLE                           131072             8.083762  =
           0.000061
  PKT_RPC_REASM                             131072            36.484091  =
           0.000278
  DCACHE_WRITE                              131072             6.565905  =
           0.000050
  PKT_ACPU_QUEUE                            131072             0.644989  =
           0.000004
  NFS_REPLY                                 131072             0.391506  =
           0.000002
  PKT_NCPU_QUEUE                            131072             0.241320  =
           0.000001
  PKT_TX                                    131072             0.287072  =
           0.000002

In the rh5 version, we spent almost all the time in NFS_OPEN_HANDLE, =
which tries to get the lock:

kpi                                        calls        total seconds    =
     seconds/call
NFS_V3_WRITE                              131089          1752.865365    =
         0.013371 =20
  NFS_OPEN_HANDLE                           131089          1609.455383  =
           0.012277
    FCB_LOCK                                   96311          =
1608.755907             0.016703
    AUTHOR_PERMISSION_CHECK                   131089             =
0.160068             0.000001

_____________________________________________
From: Maxim Kozlovsky=20
Sent: Monday, September 24, 2007 3:17 PM
To: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Andy =
Sharp
Cc: Brian Montero
Subject: RE: Wireshark Red Hat 3 and 5 NFS sequential performance=20

Can we find all "the other guys" and make sure they understand how to =
measure the performance of "dd'? The newer version of "dd" even prints =
the throughput after it is done for arithmetically challenged.

Ok, for the single direction write I really see the difference:

Rh5 - 33.6MB/s

131072+0 records in
131072+0 records out
4294967296 bytes (4.3 GB) copied, 127.949 seconds, 33.6 MB/s

real    2m7.977s
user    0m0.086s
sys     0m20.887s

rh3 - 60.8MB/Sec

time dd if=3D/dev/zero of=3D/1/file2 bs=3D32k count=3D131072
131072+0 records in
131072+0 records out

real    1m7.334s
user    0m0.140s
sys     0m13.100s

Are the rh5 and rh3 clients identical except the operating system?

_____________________________________________
From: Fay Chong=20
Sent: Monday, September 24, 2007 2:53 PM
To: Maxim Kozlovsky; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; =
Andy Sharp
Cc: Brian Montero
Subject: RE: Wireshark Red Hat 3 and 5 NFS sequential performance=20

Max,=20
I was doing single direction only. I did writes and then reads. These =
are the commands:

dd if=3D/dev/zero of=3Dfile1 bs=3D32k count=3D131072 & # write 4 GB
dd if=3Dfile1 of=3D/dev/null bs=3D32k count=3D131072 & # read 4 GB

I did not umount/mount or take the vol offline/online between tests. I =
normally do vol del and vol create when I do performance testing. Here =
the data rates are so far apart I did not think it made a difference. As =
far as vsvr stats go, they may not be absolutely accurate but that's =
what the other guys use for comparing tests.

Fay
_____________________________________________
From: Maxim Kozlovsky=20
Sent: Monday, September 24, 2007 2:40 PM
To: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Andy =
Sharp
Cc: Brian Montero
Subject: RE: Wireshark Red Hat 3 and 5 NFS sequential performance=20

What exactly was this test doing? Is it single direction, or =
bidirectional? The excel spreadsheet seems to imply that it is =
bidirectional.

I've tried bidirectional test and got completely different results with =
almost identical performance:

rhel3 - read 30MB/sec write 46MB/sec,=20
rhel5 - read 29.6MB/sec write 53.1MB/s

The test that I was running:

Rh3

time dd if=3D/1/file1 of=3D/dev/zero bs=3D32k ; skill -INT dd &
86187+0 records in
86187+0 records out

real    1m27.021s
user    0m0.090s
sys     0m11.740s

time dd if=3D/dev/zero of=3D/1/file2 bs=3D32k count=3D131072 ; skill =
-INT dd &
131072+0 records in
131072+0 records out

real    1m28.539s
user    0m0.160s
sys     0m19.460s

rh5:

time dd if=3D/dev/zero of=3D/1/file2 bs=3D32k count=3D131072 ; skill =
-INT dd&
131072+0 records in
131072+0 records out
4294967296 bytes (4.3 GB) copied, 80.8301 seconds, 53.1 MB/s

real    1m20.834s
user    0m0.081s
sys     0m14.097s

time dd if=3D/1/file1 of=3D/dev/zero bs=3D32k ; skill -INT dd &
72297+0 records in
72296+0 records out
2368995328 bytes (2.4 GB) copied, 80.0069 seconds, 29.6 MB/s


real    1m20.023s
user    0m0.047s
sys     0m1.866s

Couple of things:

Do not post the results of running "vs stat agg" as actual performance, =
who knows what this code is doing. For the case of dd the performance =
can be measured directly as shown above.

With this high volume of traffic the Wireshark is lossy, you can not =
rely on it to tell anything about dropped packets. Look at the TCP stats =
for the number of retransmitted packets on the client and on the filer =
instead (which by the way was 0 in my test on both rhel5 and rhel3).

In probably already doing this, but I thought I'll mention it just in =
case - you should unmount and remount the volume on the client and vol =
offline /vol online the volume on the filer between the tests to make =
sure consistent initial state is used.

Could you please create a script(s) which can be used to run your =
test(s) so we know we are on a same page?

Max

_____________________________________________
From: Fay Chong=20
Sent: Sunday, September 23, 2007 5:49 PM
To: Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Andy Sharp; Maxim =
Kozlovsky
Cc: Brian Montero; Fay Chong
Subject: Wireshark Red Hat 3 and 5 NFS sequential performance=20

Hi,

Attached are some results from the Wireshark trace experiments on NFS =
sequential read and write with Red Hat Linux release 3 and 5. The =
Wireshark summaries seemed to have a lot of dropped packets as well as =
TCP acked lost segment and TCP Previous segment lost messages. The =
traces were saved so they can be reviewed by others. Also the vsvr stat =
agg throughput results are included. Tests were run with and without the =
wireshark. Let's talk about reviewing the data and refining the =
experiment.

Thanks

Fay

 << File: wiresharkexp1.xls >>=20


Fay Chong
Sr. Performance Engineer
ONStor, Inc.
fay.chong@onstor.com
408.376.3130 (w)







------_=_NextPart_001_01C7FF9C.F21FCC23
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Diso-8859-1">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7652.24">
<TITLE>RE: Lab reproduction for UK cases - RE: British Red Cross Case =
5940</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->
<BR>

<P><FONT COLOR=3D"#000000" SIZE=3D2 FACE=3D"Arial">Hi all,</FONT>
</P>

<P><FONT COLOR=3D"#000000" SIZE=3D2 FACE=3D"Arial">Carolyn asked me to =
send this around cc-ing everybody, as it appears that we have a few =
loops going on at the same time.&nbsp; Initial customer setup is below, =
followed by Max's findings of Fay's test comparisons for RH kernels 2.4 =
(fast) and 2.6 (slow) from a simple dd write.</FONT></P>

<P><FONT COLOR=3D"#000000" SIZE=3D2 FACE=3D"Arial">Rich</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
<B>From:</B> Tim O'Callaghan<BR>
<B>Sent:</B> Tuesday, September 25, 2007 8:43 AM<BR>
<B>To:</B> Laurence Armiger<BR>
<B>Cc:</B> Carolyn Sims; Gerald Sommariva; Caeli Collins; Neil Cook; =
David Crispin<BR>
<B>Subject:</B> Re: British Red Cross Case 5940</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Laurence,</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">HELP required.</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">I have finally =
managed to get the authority to reboot the Linux DS-System server in =
British Red Cross. </FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Earlier today I =
rebooted the system in Backup Technology and their performance levels =
resumed.</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Rebooting the linux =
system has returned performance to its previous levels as Gerald =
suggested it might.</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">All the systems =
supported by Backup Technology are;</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Kernel =
2.6.9-5.Elsmp</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">RedHat Enterprise =
4.</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">The volumes are =
mounted with the following options; nfs =
defaults,intr,rsize=3D32768,wsize=3D32768,async,nfsvers=3D3</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Asigra recommend Red =
Hat Enterprise 4 which includes the 2.6.9-5.Elsmp kernel.</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Any Asigra vault of =
significance will require an NFS file system, for the following =
reasons</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">1.&nbsp; PostGreSql =
is &#8216;better than MS SQL&#8217; in stability, efficiency and =
cost.</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">2.&nbsp; I believe =
Linux can manage a greater than number of files than W2K3 server.</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">A typical Asigra =
10TB vault will contain &gt; 100 million files. (10 million files per =
TB)</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Backup Technology are =
now at a crossroad with respect to the ONstor product &#8211; they feel =
that &#8220;since NFS is an open standard &#8211; it would never cause a =
problem, ONStor should work&#8221; Simon has intimated that unless we =
can get this fixed he has sold his last Bobcat &#8211; his retirement =
plan is based on moving the Asigra CIFs user base to NFS .</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">You will not believe =
the crap I am now enduring about this &#8211; can we put pressure on the =
engineering team to get the later kernel supported &#8211; given the =
Asigra recommendation of RedHat 4 I consider it unrealistic to think we =
can persuade Backup Technology to downgrade their DS-System operating =
system patch revison level.</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">To retain any hopes =
of future deals with these guys we need to get this fixed ASAP.</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Regards</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Tim</FONT>
</P>

<P><FONT COLOR=3D"#000080" FACE=3D"Arial"></FONT>&nbsp;<A =
HREF=3D"http://www.onstor.com/images/footer/onstor_signature.gif"><U></U>=
<U><FONT COLOR=3D"#0000FF" FACE=3D"Times New =
Roman">http://www.onstor.com/images/footer/onstor_signature.gif</FONT></U=
></A><BR>
<BR>


<BR><B><FONT COLOR=3D"#000080" SIZE=3D1 FACE=3D"Verdana">Tim =
O'Callaghan</FONT></B><BR>
<FONT COLOR=3D"#000080" SIZE=3D1 FACE=3D"Verdana">Senior S.E.<BR>
<BR>
ONStor, Inc.<BR>
</FONT><B><FONT COLOR=3D"#000080" SIZE=3D1 =
FACE=3D"Verdana">office:</FONT></B><FONT COLOR=3D"#000080" SIZE=3D1 =
FACE=3D"Verdana">=A0 +44 1189 635866<BR>
</FONT><B><FONT COLOR=3D"#000080" SIZE=3D1 =
FACE=3D"Verdana">mobile:</FONT></B><FONT COLOR=3D"#000080" SIZE=3D1 =
FACE=3D"Verdana"> +44 7767 435472<BR>
</FONT>

<BR><A HREF=3D"mailto:tim.ocallaghan@onstor.com"><U><FONT =
COLOR=3D"#0000FF" SIZE=3D1 =
FACE=3D"Verdana">tim.ocallaghan@onstor.com</FONT></U></A><BR>
<A HREF=3D"http://www.onstor.com"><U></U><U><FONT COLOR=3D"#0000FF" =
SIZE=3D1 FACE=3D"Verdana">http://www.onstor.com</FONT></U></A><BR>
<FONT COLOR=3D"#000080" FACE=3D"Arial"></FONT>&nbsp;<A =
HREF=3D"http://www.onstor.com/images/footer/storage_mag_product_of_year.g=
if"><U></U><U><FONT COLOR=3D"#0000FF" FACE=3D"Times New =
Roman">http://www.onstor.com/images/footer/storage_mag_product_of_year.gi=
f</FONT></U></A>&nbsp;&nbsp;&nbsp;=20
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Maxim Kozlovsky<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, September 24, 2007 4:38 PM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Fay Chong; Paul Hammer; Jonathan Goldick; Jobi =
Ariyamannil; Andy Sharp<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Cc:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Brian Montero; John Rogers<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: Wireshark Red Hat 3 and 5 NFS sequential =
performance</FONT>=20
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Tahoma">Ok, the problem with =
the writes is unaligned and out of order writes with 2.6.18 kernel. See =
a trace at ~maximk/tmp/rh5.eth, frame 70232/70247/70267/70287 and so on. =
We happen to handle this particularly badly as we don&#8217;t coalesce =
the writes and read the partial blocks from disk. To make the matters =
worse, we hold the lock for the duration of this operation. At first I =
thought that I did not run the test right and did not delete the file, =
but Linux kernel really does that even with a new file. The KPI files =
are in ~maximk/tmp if anybody wants to take a look.</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier =
New">[maximk@compile2 tmp]$ =
/n/build-trees/R3.0.0.0/EverON-3.0.0.0/nfx-tree/Build/x86utils/kpi-ui -c =
NFS_V3_WRITE -f rh5.kpi </FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier =
New">kpi&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp; calls&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
total seconds&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
seconds/call</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier =
New">NFS_V3_WRITE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131089&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
1752.865365&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; 0.013371</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
NFS_OPEN_HANDLE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp; =
131089&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
1609.455383&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; 0.012277</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
PKT_RPC_REASM&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131089&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
41.302495&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp; 0.000315</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
DCACHE_WRITE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131089&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
96.842795&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp; 0.000738</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
PKT_ACPU_QUEUE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131089&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; =
0.431612&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp; 0.000003</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
NFS_REPLY&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131089&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; =
0.476795&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp; 0.000003</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
PKT_NCPU_QUEUE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131089&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; =
0.186616&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp; 0.000001</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
PKT_TX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131089&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; =
0.338352&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp; 0.000002</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier =
New">[maximk@compile2 tmp]$ =
/n/build-trees/R3.0.0.0/EverON-3.0.0.0/nfx-tree/Build/x86utils/kpi-ui -c =
NFS_V3_WRITE -f rh3.kpi</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier =
New">kpi&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp; calls&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
total seconds&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
seconds/call</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier =
New">NFS_V3_WRITE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131072&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
56.102047&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp; 0.000428</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
NFS_OPEN_HANDLE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp; =
131072&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; =
8.083762&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp; 0.000061</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
PKT_RPC_REASM&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131072&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
36.484091&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp; 0.000278</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
DCACHE_WRITE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131072&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; =
6.565905&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp; 0.000050</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
PKT_ACPU_QUEUE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131072&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; =
0.644989&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp; 0.000004</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
NFS_REPLY&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131072&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; =
0.391506&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp; 0.000002</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
PKT_NCPU_QUEUE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131072&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; =
0.241320&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp; 0.000001</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
PKT_TX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131072&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; =
0.287072&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp; 0.000002</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Tahoma">In the rh5 version, =
we spent almost all the time in NFS_OPEN_HANDLE, which tries to get the =
lock:</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier =
New">kpi&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp; calls&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
total seconds&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
seconds/call</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier =
New">NFS_V3_WRITE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131089&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
1752.865365&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; 0.013371&nbsp; </FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier New">&nbsp; =
NFS_OPEN_HANDLE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp; =
131089&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
1609.455383&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; 0.012277</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp; =
FCB_LOCK&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
96311&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
1608.755907&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; 0.016703</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Courier =
New">&nbsp;&nbsp;&nbsp; =
AUTHOR_PERMISSION_CHECK&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
131089&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; =
0.160068&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp; 0.000001</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Maxim Kozlovsky<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, September 24, 2007 3:17 PM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Fay Chong; Paul Hammer; Jonathan Goldick; Jobi =
Ariyamannil; Andy Sharp<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Cc:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Brian Montero<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: Wireshark Red Hat 3 and 5 NFS sequential =
performance</FONT>=20
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Can we find all =
&#8220;the other guys&#8221; and make sure they understand how to =
measure the performance of &#8220;dd&#8217;? The newer version of =
&#8220;dd&#8221; even prints the throughput after it is done for =
arithmetically challenged.</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Ok, for the single =
direction write I really see the difference:</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Rh5 &#8211; =
33.6MB/s</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">131072+0 records =
in</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">131072+0 records =
out</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">4294967296 bytes =
(4.3 GB) copied, 127.949 seconds, 33.6 MB/s</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">real&nbsp;&nbsp;&nbsp; 2m7.977s</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">user&nbsp;&nbsp;&nbsp; 0m0.086s</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">sys&nbsp;&nbsp;&nbsp;&nbsp; 0m20.887s</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">rh3 &#8211; =
60.8MB/Sec</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">time dd =
if=3D/dev/zero of=3D/1/file2 bs=3D32k count=3D131072</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">131072+0 records =
in</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">131072+0 records =
out</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">real&nbsp;&nbsp;&nbsp; 1m7.334s</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">user&nbsp;&nbsp;&nbsp; 0m0.140s</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">sys&nbsp;&nbsp;&nbsp;&nbsp; 0m13.100s</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Are the rh5 and rh3 =
clients identical except the operating system?</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Fay Chong<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, September 24, 2007 2:53 PM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Maxim Kozlovsky; Paul Hammer; Jonathan Goldick; Jobi =
Ariyamannil; Andy Sharp<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Cc:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Brian Montero<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: Wireshark Red Hat 3 and 5 NFS sequential =
performance</FONT>=20
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Max, </FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">I was doing single =
direction only. I did writes and then reads. These are the =
commands:</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">dd if=3D/dev/zero =
of=3Dfile1 bs=3D32k count=3D131072 &amp; # write 4 GB</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">dd if=3Dfile1 =
of=3D/dev/null bs=3D32k count=3D131072 &amp; # read 4 GB</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">I did not =
umount/mount or take the vol offline/online between tests. I normally do =
vol del and vol create when I do performance testing. Here the data =
rates are so far apart I did not think it made a difference. As far as =
vsvr stats go, they may not be absolutely accurate but that&#8217;s what =
the other guys use for comparing tests.</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Fay</FONT>

<BR><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Maxim Kozlovsky<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, September 24, 2007 2:40 PM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Fay Chong; Paul Hammer; Jonathan Goldick; Jobi =
Ariyamannil; Andy Sharp<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Cc:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Brian Montero<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: Wireshark Red Hat 3 and 5 NFS sequential =
performance</FONT>=20
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">What exactly was this =
test doing? Is it single direction, or bidirectional? The excel =
spreadsheet seems to imply that it is bidirectional.</FONT></P>
<BR>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">I&#8217;ve tried =
bidirectional test and got completely different results with almost =
identical performance:</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">rhel3 &#8211; read =
30MB/sec write 46MB/sec, </FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">rhel5 &#8211; read =
29.6MB/sec write 53.1MB/s</FONT>
<BR>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">The test that I was =
running:</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Rh3</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">time dd if=3D/1/file1 =
of=3D/dev/zero bs=3D32k ; skill &#8211;INT dd &amp;</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">86187+0 records =
in</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">86187+0 records =
out</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">real&nbsp;&nbsp;&nbsp; 1m27.021s</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">user&nbsp;&nbsp;&nbsp; 0m0.090s</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">sys&nbsp;&nbsp;&nbsp;&nbsp; 0m11.740s</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">time dd =
if=3D/dev/zero of=3D/1/file2 bs=3D32k count=3D131072 ; skill -INT dd =
&amp;</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">131072+0 records =
in</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">131072+0 records =
out</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">real&nbsp;&nbsp;&nbsp; 1m28.539s</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">user&nbsp;&nbsp;&nbsp; 0m0.160s</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">sys&nbsp;&nbsp;&nbsp;&nbsp; 0m19.460s</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">rh5:</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">time dd =
if=3D/dev/zero of=3D/1/file2 bs=3D32k count=3D131072 ; skill -INT =
dd&amp;</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">131072+0 records =
in</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">131072+0 records =
out</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">4294967296 bytes =
(4.3 GB) copied, 80.8301 seconds, 53.1 MB/s</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">real&nbsp;&nbsp;&nbsp; 1m20.834s</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">user&nbsp;&nbsp;&nbsp; 0m0.081s</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">sys&nbsp;&nbsp;&nbsp;&nbsp; 0m14.097s</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">time dd if=3D/1/file1 =
of=3D/dev/zero bs=3D32k ; skill -INT dd &amp;</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">72297+0 records =
in</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">72296+0 records =
out</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">2368995328 bytes =
(2.4 GB) copied, 80.0069 seconds, 29.6 MB/s</FONT>
</P>
<BR>

<P><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">real&nbsp;&nbsp;&nbsp; 1m20.023s</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">user&nbsp;&nbsp;&nbsp; 0m0.047s</FONT>

<BR><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">sys&nbsp;&nbsp;&nbsp;&nbsp; 0m1.866s</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Couple of =
things:</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Do not post the =
results of running &#8220;vs stat agg&#8221; as actual performance, who =
knows what this code is doing. For the case of dd the performance can be =
measured directly as shown above.</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">With this high volume =
of traffic the Wireshark is lossy, you can not rely on it to tell =
anything about dropped packets. Look at the TCP stats for the number of =
retransmitted packets on the client and on the filer instead (which by =
the way was 0 in my test on both rhel5 and rhel3).</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">In probably already =
doing this, but I thought I&#8217;ll mention it just in case - you =
should unmount and remount the volume on the client and vol offline /vol =
online the volume on the filer between the tests to make sure consistent =
initial state is used.</FONT></P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Could you please =
create a script(s) which can be used to run your test(s) so we know we =
are on a same page?</FONT>
</P>

<P><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Max</FONT>
</P>

<P><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Fay Chong<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Sunday, September 23, 2007 5:49 PM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Andy =
Sharp; Maxim Kozlovsky<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Cc:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Brian Montero; Fay Chong<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> Wireshark Red Hat 3 and 5 NFS sequential =
performance</FONT>=20
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Hi,</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Attached are some results from the =
Wireshark trace experiments on NFS sequential read and write with Red =
Hat Linux release 3 and 5. The Wireshark summaries seemed to have a lot =
of dropped packets as well as TCP acked lost segment and TCP Previous =
segment lost messages. The traces were saved so they can be reviewed by =
others. Also the vsvr stat agg throughput results are included. Tests =
were run with and without the wireshark. Let&#8217;s talk about =
reviewing the data and refining the experiment.</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Thanks</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Arial">Fay</FONT>
</P>

<P><FONT FACE=3D"Times New Roman">&nbsp;&lt;&lt; File: wiresharkexp1.xls =
&gt;&gt;</FONT>=20
</P>
<BR>

<P><FONT SIZE=3D2 FACE=3D"Arial">Fay Chong</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">Sr. Performance Engineer</FONT>

<BR><FONT SIZE=3D2 FACE=3D"Arial">ONStor, Inc.</FONT>

<BR><A HREF=3D"mailto:fay.chong@onstor.com"><U><FONT COLOR=3D"#0000FF" =
SIZE=3D2 FACE=3D"Arial">fay.chong@onstor.com</FONT></U></A>

<BR><FONT SIZE=3D2 FACE=3D"Arial">408.376.3130 (w)</FONT>
</P>
<BR>
<BR>
<BR>
<BR>
<BR>

</BODY>
</HTML>
------_=_NextPart_001_01C7FF9C.F21FCC23--
