X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7FF08.E59EE4D3@onstor-exch02.onstor.net>; Mon, 24 Sep 2007 16:13:28 -0800
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
Subject: RE: Wireshark Red Hat 3 and 5 NFS sequential performance
Date: Mon, 24 Sep 2007 16:13:27 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E05B4643B@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Wireshark Red Hat 3 and 5 NFS sequential performance
Thread-Index: Acf/B3kDGkYgQa/wQmOxgOHKz7QFYQAAMjpgAAAajsA=
References: <BB375AF679D4A34E9CA8DFA650E2B04E05A25C3D@onstor-exch02.onstor.net><BB375AF679D4A34E9CA8DFA650E2B04E05B46316@onstor-exch02.onstor.net><20070924161627.066895ed@ripper.onstor.net><BB375AF679D4A34E9CA8DFA650E2B04E05B463DD@onstor-exch02.onstor.net><20070924164228.57f49f9f@ripper.onstor.net><BB375AF679D4A34E9CA8DFA650E2B04E05B46405@onstor-exch02.onstor.net> <20070924170312.1f4fa1d8@ripper.onstor.net> 
From: "Maxim Kozlovsky" <maxim.kozlovsky@onstor.com>
To: "Maxim Kozlovsky" <maxim.kozlovsky@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>
Cc: "Fay Chong" <fay.chong@onstor.com>,
	"Paul Hammer" <paul.hammer@onstor.com>,
	"Jonathan Goldick" <jonathan.goldick@onstor.com>,
	"Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>,
	"Brian Montero" <brian.montero@onstor.com>


By the way, I wonder if I would ever learn to pick the file names as
real American people do.

>
>>-----Original Message-----
>>From: Andy Sharp
>>Sent: Monday, September 24, 2007 5:03 PM
>>To: Maxim Kozlovsky
>>Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Brian
>>Montero
>>Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential performance
>>
>>Let me put it another way, it might mean something.  This is what I
did:
>>
>>ripper$ time dd =
if=3Dpack-406c14816730814f196d4c27449b79d97aa36519.pack
>>of=3Dchewbaca bs=3D32k
>>9181+1 records in
>>9181+1 records out
>>300870117 bytes (301 MB) copied, 3.63615 seconds, 82.7 MB/s
>>
>>real    0m3.690s
>>user    0m0.003s
>>sys     0m0.674s
>>ripper$ time dd if=3Dchewbaca of=3Dapeman bs=3D32k
>>9181+1 records in
>>9181+1 records out
>>300870117 bytes (301 MB) copied, 246.651 seconds, 1.2 MB/s
>>
>>real    4m6.655s
>>user    0m0.000s
>>sys     0m0.940s
>>ripper$
>>
>>
>>So essentially the second time, it fell off a cliff.
>>
>>
>>
>>On Mon, 24 Sep 2007 16:52:09 -0700 "Maxim Kozlovsky"
>><maxim.kozlovsky@onstor.com> wrote:
>>
>>> I think this is just you, Andy:
>>>
>>> [root@c98r48-rhel5 ~]# time dd if=3D/2/file1 of=3D/2/file2 bs=3D32k
>>> count=3D131072
>>> 131072+0 records in
>>> 131072+0 records out
>>> 4294967296 bytes (4.3 GB) copied, 103.266 seconds, 41.6 MB/s
>>>
>>> real    1m43.282s
>>> user    0m0.109s
>>> sys     0m7.887s
>>>
>>>
>>>
>>> >-----Original Message-----
>>> >From: Andy Sharp
>>> >Sent: Monday, September 24, 2007 4:42 PM
>>> >To: Maxim Kozlovsky
>>> >Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil;
Brian
>>> >Montero
>>> >Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential performance
>>> >
>>> >Exqueezeme?
>>> >
>>> >I'd give my kingdom for any of those horses: 30, 50, 60, 80 MB/s.
>>> >Those differences amount to a big "so what" compared to ONE MB/s,
>>> >IMHO.
>>> >
>>> >For instance, if my race bike can't shift out of neutral, I'm gonna
>>> >want to fix that problem before I fix the top-end carburetion
>>> >problem.
>>> >
>>> >Fay, please note the kernel version number on these Redmondhat
>>> releases,
>>> >even though RH kernel versions correspond very little to real
kernel
>>> >version numbers, at least we'd have a shotgun's difference.  For
>>> >instance, I believe that RHEL3 is a 2.4 kernel, and RHEL5 is some
>>> >kind of 2.6 kernel.  Most likely you can use the 'uname -r' command
>>> >to show the kernel version number.
>>> >
>>> >Cheers,
>>> >
>>> >a
>>> >
>>> >On Mon, 24 Sep 2007 16:22:49 -0700 "Maxim Kozlovsky"
>>> ><maxim.kozlovsky@onstor.com> wrote:
>>> >
>>> >> Sure, but we are not there yet. For now, we have a problem with
>>> >> even running unidirectional write, so trying bidirectional test
>>> >> will not give a lot of new information.
>>> >>
>>> >> >-----Original Message-----
>>> >> >From: Andy Sharp
>>> >> >Sent: Monday, September 24, 2007 4:16 PM
>>> >> >To: Maxim Kozlovsky
>>> >> >Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil;
>>> Brian
>>> >> >Montero
>>> >> >Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential
performance
>>> >> >
>>> >> >The test that should demonstrate the problem is not two
>>> >> >unidirectional tests running at the same time, but a
bidirectional
>>> >> >test:
>>> >> >
>>> >> >
>>> >> >$ time dd if=3Dchewbaca of=3Dapeman bs=3D32k
>>> >> >9181+1 records in
>>> >> >9181+1 records out
>>> >> >300870117 bytes (301 MB) copied, 246.651 seconds, 1.2 MB/s
>>> >> >
>>> >> >real    4m6.655s
>>> >> >user    0m0.000s
>>> >> >sys     0m0.940s
>>> >> >
>>> >> >
>>> >> >Cheers,
>>> >> >
>>> >> >a
>>> >> >
>>> >> >On Mon, 24 Sep 2007 14:40:01 -0700 "Maxim Kozlovsky"
>>> >> ><maxim.kozlovsky@onstor.com> wrote:
>>> >> >
>>> >> >> What exactly was this test doing? Is it single direction, or
>>> >> >> bidirectional? The excel spreadsheet seems to imply that it is
>>> >> >> bidirectional.
>>> >> >>
>>> >> >> I've tried bidirectional test and got completely different
>>> >> >> results with almost identical performance:
>>> >> >>
>>> >> >> rhel3 - read 30MB/sec write 46MB/sec,
>>> >> >> rhel5 - read 29.6MB/sec write 53.1MB/s
>>> >> >>
>>> >> >> The test that I was running:
>>> >> >>
>>> >> >> Rh3
>>> >> >>
>>> >> >> time dd if=3D/1/file1 of=3D/dev/zero bs=3D32k ; skill -INT dd =
&
>>> >> >> 86187+0 records in
>>> >> >> 86187+0 records out
>>> >> >>
>>> >> >> real    1m27.021s
>>> >> >> user    0m0.090s
>>> >> >> sys     0m11.740s
>>> >> >>
>>> >> >> time dd if=3D/dev/zero of=3D/1/file2 bs=3D32k count=3D131072 ; =
skill
>>> >> >> -INT dd
>>> >> &
>>> >> >> 131072+0 records in
>>> >> >> 131072+0 records out
>>> >> >>
>>> >> >> real    1m28.539s
>>> >> >> user    0m0.160s
>>> >> >> sys     0m19.460s
>>> >> >>
>>> >> >> rh5:
>>> >> >>
>>> >> >> time dd if=3D/dev/zero of=3D/1/file2 bs=3D32k count=3D131072 ; =
skill
>>> >> >> -INT dd& 131072+0 records in
>>> >> >> 131072+0 records out
>>> >> >> 4294967296 bytes (4.3 GB) copied, 80.8301 seconds, 53.1 MB/s
>>> >> >>
>>> >> >> real    1m20.834s
>>> >> >> user    0m0.081s
>>> >> >> sys     0m14.097s
>>> >> >>
>>> >> >> time dd if=3D/1/file1 of=3D/dev/zero bs=3D32k ; skill -INT dd =
&
>>> >> >> 72297+0 records in
>>> >> >> 72296+0 records out
>>> >> >> 2368995328 bytes (2.4 GB) copied, 80.0069 seconds, 29.6 MB/s
>>> >> >>
>>> >> >>
>>> >> >> real    1m20.023s
>>> >> >> user    0m0.047s
>>> >> >> sys     0m1.866s
>>> >> >>
>>> >> >> Couple of things:
>>> >> >>
>>> >> >> Do not post the results of running "vs stat agg" as actual
>>> >> >> performance, who knows what this code is doing. For the case
of
>>> >> >> dd the performance can be measured directly as shown above.
>>> >> >>
>>> >> >> With this high volume of traffic the Wireshark is lossy, you
can
>>> >> >> not rely on it to tell anything about dropped packets. Look at
>>> >> >> the TCP stats for the number of retransmitted packets on the
>>> >> >> client and on the filer instead (which by the way was 0 in my
>>> >> >> test on both rhel5 and rhel3).
>>> >> >>
>>> >> >> In probably already doing this, but I thought I'll mention it
>>> >> >> just in case - you should unmount and remount the volume on
the
>>> >> >> client and
>>> >> vol
>>> >> >> offline /vol online the volume on the filer between the tests
to
>>> >> >> make sure consistent initial state is used.
>>> >> >>
>>> >> >> Could you please create a script(s) which can be used to run
>>> >> >> your test(s) so we know we are on a same page?
>>> >> >>
>>> >> >> Max
>>> >> >>
>>> >> >> _____________________________________________
>>> >> >> From: Fay Chong
>>> >> >> Sent: Sunday, September 23, 2007 5:49 PM
>>> >> >> To: Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Andy
Sharp;
>>> >> Maxim
>>> >> >> Kozlovsky
>>> >> >> Cc: Brian Montero; Fay Chong
>>> >> >> Subject: Wireshark Red Hat 3 and 5 NFS sequential performance
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> Attached are some results from the Wireshark trace experiments
>>> >> >> on NFS sequential read and write with Red Hat Linux release 3
>>> >> >> and 5. The Wireshark summaries seemed to have a lot of dropped
>>> >> >> packets as well
>>> >> as
>>> >> >> TCP acked lost segment and TCP Previous segment lost messages.
>>> >> >> The traces were saved so they can be reviewed by others. Also
>>> >> >> the vsvr stat agg throughput results are included. Tests were
>>> >> >> run with and without the wireshark. Let's talk about reviewing
>>> >> >> the data and refining the experiment.
>>> >> >>
>>> >> >> Thanks
>>> >> >>
>>> >> >> Fay
>>> >> >>
>>> >> >>  << File: wiresharkexp1.xls >>
>>> >> >>
>>> >> >>
>>> >> >> Fay Chong
>>> >> >> Sr. Performance Engineer
>>> >> >> ONStor, Inc.
>>> >> >> fay.chong@onstor.com
>>> >> >> 408.376.3130 (w)
>>> >> >>
>>> >> >>
