X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7FF0B.0FF60F58@onstor-exch02.onstor.net>; Mon, 24 Sep 2007 16:28:58 -0800
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
Subject: RE: Wireshark Red Hat 3 and 5 NFS sequential performance
Date: Mon, 24 Sep 2007 16:28:57 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E05B46456@onstor-exch02.onstor.net>
In-Reply-To: <20070924172222.5a11a3be@ripper.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Wireshark Red Hat 3 and 5 NFS sequential performance
Thread-Index: Acf/CiXQvrpPM745TAGQTJ2h0LaTgAAAC1qA
References: <BB375AF679D4A34E9CA8DFA650E2B04E05A25C3D@onstor-exch02.onstor.net><BB375AF679D4A34E9CA8DFA650E2B04E05B46316@onstor-exch02.onstor.net><20070924161627.066895ed@ripper.onstor.net><BB375AF679D4A34E9CA8DFA650E2B04E05B463DD@onstor-exch02.onstor.net><20070924164228.57f49f9f@ripper.onstor.net><BB375AF679D4A34E9CA8DFA650E2B04E05B46405@onstor-exch02.onstor.net><20070924170312.1f4fa1d8@ripper.onstor.net><BB375AF679D4A34E9CA8DFA650E2B04E05B4643B@onstor-exch02.onstor.net> <20070924172222.5a11a3be@ripper.onstor.net>
From: "Maxim Kozlovsky" <maxim.kozlovsky@onstor.com>
To: "Andy Sharp" <andy.sharp@onstor.com>
Cc: "Fay Chong" <fay.chong@onstor.com>,
	"Paul Hammer" <paul.hammer@onstor.com>,
	"Jonathan Goldick" <jonathan.goldick@onstor.com>,
	"Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>,
	"Brian Montero" <brian.montero@onstor.com>

I could tell you what Russian people think about Americans trying to
speak Russian, but this usually involves unprintable expressions.
"Woosie" does not even start to describe it.=20

>-----Original Message-----
>From: Andy Sharp
>Sent: Monday, September 24, 2007 5:22 PM
>To: Maxim Kozlovsky
>Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Brian
>Montero
>Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential performance
>
>It's a gift, what can I say?  Besides, I thought you were a real
>'merican?  I mean, apart from that woosie Russian accent.
>
>On Mon, 24 Sep 2007 17:13:27 -0700 "Maxim Kozlovsky"
><maxim.kozlovsky@onstor.com> wrote:
>
>>
>> By the way, I wonder if I would ever learn to pick the file names as
>> real American people do.
>>
>> >
>> >>-----Original Message-----
>> >>From: Andy Sharp
>> >>Sent: Monday, September 24, 2007 5:03 PM
>> >>To: Maxim Kozlovsky
>> >>Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil;
>> >>Brian Montero
>> >>Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential performance
>> >>
>> >>Let me put it another way, it might mean something.  This is what I
>> did:
>> >>
>> >>ripper$ time dd
>> >>if=3Dpack-406c14816730814f196d4c27449b79d97aa36519.pack =
of=3Dchewbaca
>> >>bs=3D32k 9181+1 records in
>> >>9181+1 records out
>> >>300870117 bytes (301 MB) copied, 3.63615 seconds, 82.7 MB/s
>> >>
>> >>real    0m3.690s
>> >>user    0m0.003s
>> >>sys     0m0.674s
>> >>ripper$ time dd if=3Dchewbaca of=3Dapeman bs=3D32k
>> >>9181+1 records in
>> >>9181+1 records out
>> >>300870117 bytes (301 MB) copied, 246.651 seconds, 1.2 MB/s
>> >>
>> >>real    4m6.655s
>> >>user    0m0.000s
>> >>sys     0m0.940s
>> >>ripper$
>> >>
>> >>
>> >>So essentially the second time, it fell off a cliff.
>> >>
>> >>
>> >>
>> >>On Mon, 24 Sep 2007 16:52:09 -0700 "Maxim Kozlovsky"
>> >><maxim.kozlovsky@onstor.com> wrote:
>> >>
>> >>> I think this is just you, Andy:
>> >>>
>> >>> [root@c98r48-rhel5 ~]# time dd if=3D/2/file1 of=3D/2/file2 =
bs=3D32k
>> >>> count=3D131072
>> >>> 131072+0 records in
>> >>> 131072+0 records out
>> >>> 4294967296 bytes (4.3 GB) copied, 103.266 seconds, 41.6 MB/s
>> >>>
>> >>> real    1m43.282s
>> >>> user    0m0.109s
>> >>> sys     0m7.887s
>> >>>
>> >>>
>> >>>
>> >>> >-----Original Message-----
>> >>> >From: Andy Sharp
>> >>> >Sent: Monday, September 24, 2007 4:42 PM
>> >>> >To: Maxim Kozlovsky
>> >>> >Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil;
>> Brian
>> >>> >Montero
>> >>> >Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential
performance
>> >>> >
>> >>> >Exqueezeme?
>> >>> >
>> >>> >I'd give my kingdom for any of those horses: 30, 50, 60, 80
MB/s.
>> >>> >Those differences amount to a big "so what" compared to ONE
MB/s,
>> >>> >IMHO.
>> >>> >
>> >>> >For instance, if my race bike can't shift out of neutral, I'm
>> >>> >gonna want to fix that problem before I fix the top-end
>> >>> >carburetion problem.
>> >>> >
>> >>> >Fay, please note the kernel version number on these Redmondhat
>> >>> releases,
>> >>> >even though RH kernel versions correspond very little to real
>> kernel
>> >>> >version numbers, at least we'd have a shotgun's difference.  For
>> >>> >instance, I believe that RHEL3 is a 2.4 kernel, and RHEL5 is
some
>> >>> >kind of 2.6 kernel.  Most likely you can use the 'uname -r'
>> >>> >command to show the kernel version number.
>> >>> >
>> >>> >Cheers,
>> >>> >
>> >>> >a
>> >>> >
>> >>> >On Mon, 24 Sep 2007 16:22:49 -0700 "Maxim Kozlovsky"
>> >>> ><maxim.kozlovsky@onstor.com> wrote:
>> >>> >
>> >>> >> Sure, but we are not there yet. For now, we have a problem
with
>> >>> >> even running unidirectional write, so trying bidirectional
test
>> >>> >> will not give a lot of new information.
>> >>> >>
>> >>> >> >-----Original Message-----
>> >>> >> >From: Andy Sharp
>> >>> >> >Sent: Monday, September 24, 2007 4:16 PM
>> >>> >> >To: Maxim Kozlovsky
>> >>> >> >Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi
>> >>> >> >Ariyamannil;
>> >>> Brian
>> >>> >> >Montero
>> >>> >> >Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential
>> performance
>> >>> >> >
>> >>> >> >The test that should demonstrate the problem is not two
>> >>> >> >unidirectional tests running at the same time, but a
>> bidirectional
>> >>> >> >test:
>> >>> >> >
>> >>> >> >
>> >>> >> >$ time dd if=3Dchewbaca of=3Dapeman bs=3D32k
>> >>> >> >9181+1 records in
>> >>> >> >9181+1 records out
>> >>> >> >300870117 bytes (301 MB) copied, 246.651 seconds, 1.2 MB/s
>> >>> >> >
>> >>> >> >real    4m6.655s
>> >>> >> >user    0m0.000s
>> >>> >> >sys     0m0.940s
>> >>> >> >
>> >>> >> >
>> >>> >> >Cheers,
>> >>> >> >
>> >>> >> >a
>> >>> >> >
>> >>> >> >On Mon, 24 Sep 2007 14:40:01 -0700 "Maxim Kozlovsky"
>> >>> >> ><maxim.kozlovsky@onstor.com> wrote:
>> >>> >> >
>> >>> >> >> What exactly was this test doing? Is it single direction,
or
>> >>> >> >> bidirectional? The excel spreadsheet seems to imply that it
>> >>> >> >> is bidirectional.
>> >>> >> >>
>> >>> >> >> I've tried bidirectional test and got completely different
>> >>> >> >> results with almost identical performance:
>> >>> >> >>
>> >>> >> >> rhel3 - read 30MB/sec write 46MB/sec,
>> >>> >> >> rhel5 - read 29.6MB/sec write 53.1MB/s
>> >>> >> >>
>> >>> >> >> The test that I was running:
>> >>> >> >>
>> >>> >> >> Rh3
>> >>> >> >>
>> >>> >> >> time dd if=3D/1/file1 of=3D/dev/zero bs=3D32k ; skill -INT =
dd &
>> >>> >> >> 86187+0 records in
>> >>> >> >> 86187+0 records out
>> >>> >> >>
>> >>> >> >> real    1m27.021s
>> >>> >> >> user    0m0.090s
>> >>> >> >> sys     0m11.740s
>> >>> >> >>
>> >>> >> >> time dd if=3D/dev/zero of=3D/1/file2 bs=3D32k =
count=3D131072 ;
skill
>> >>> >> >> -INT dd
>> >>> >> &
>> >>> >> >> 131072+0 records in
>> >>> >> >> 131072+0 records out
>> >>> >> >>
>> >>> >> >> real    1m28.539s
>> >>> >> >> user    0m0.160s
>> >>> >> >> sys     0m19.460s
>> >>> >> >>
>> >>> >> >> rh5:
>> >>> >> >>
>> >>> >> >> time dd if=3D/dev/zero of=3D/1/file2 bs=3D32k =
count=3D131072 ;
skill
>> >>> >> >> -INT dd& 131072+0 records in
>> >>> >> >> 131072+0 records out
>> >>> >> >> 4294967296 bytes (4.3 GB) copied, 80.8301 seconds, 53.1
MB/s
>> >>> >> >>
>> >>> >> >> real    1m20.834s
>> >>> >> >> user    0m0.081s
>> >>> >> >> sys     0m14.097s
>> >>> >> >>
>> >>> >> >> time dd if=3D/1/file1 of=3D/dev/zero bs=3D32k ; skill -INT =
dd &
>> >>> >> >> 72297+0 records in
>> >>> >> >> 72296+0 records out
>> >>> >> >> 2368995328 bytes (2.4 GB) copied, 80.0069 seconds, 29.6
MB/s
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> real    1m20.023s
>> >>> >> >> user    0m0.047s
>> >>> >> >> sys     0m1.866s
>> >>> >> >>
>> >>> >> >> Couple of things:
>> >>> >> >>
>> >>> >> >> Do not post the results of running "vs stat agg" as actual
>> >>> >> >> performance, who knows what this code is doing. For the
case
>> of
>> >>> >> >> dd the performance can be measured directly as shown above.
>> >>> >> >>
>> >>> >> >> With this high volume of traffic the Wireshark is lossy,
you
>> can
>> >>> >> >> not rely on it to tell anything about dropped packets. Look
>> >>> >> >> at the TCP stats for the number of retransmitted packets on
>> >>> >> >> the client and on the filer instead (which by the way was 0
>> >>> >> >> in my test on both rhel5 and rhel3).
>> >>> >> >>
>> >>> >> >> In probably already doing this, but I thought I'll mention
>> >>> >> >> it just in case - you should unmount and remount the volume
>> >>> >> >> on
>> the
>> >>> >> >> client and
>> >>> >> vol
>> >>> >> >> offline /vol online the volume on the filer between the
>> >>> >> >> tests
>> to
>> >>> >> >> make sure consistent initial state is used.
>> >>> >> >>
>> >>> >> >> Could you please create a script(s) which can be used to
run
>> >>> >> >> your test(s) so we know we are on a same page?
>> >>> >> >>
>> >>> >> >> Max
>> >>> >> >>
>> >>> >> >> _____________________________________________
>> >>> >> >> From: Fay Chong
>> >>> >> >> Sent: Sunday, September 23, 2007 5:49 PM
>> >>> >> >> To: Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Andy
>> Sharp;
>> >>> >> Maxim
>> >>> >> >> Kozlovsky
>> >>> >> >> Cc: Brian Montero; Fay Chong
>> >>> >> >> Subject: Wireshark Red Hat 3 and 5 NFS sequential
>> >>> >> >> performance
>> >>> >> >>
>> >>> >> >> Hi,
>> >>> >> >>
>> >>> >> >> Attached are some results from the Wireshark trace
>> >>> >> >> experiments on NFS sequential read and write with Red Hat
>> >>> >> >> Linux release 3 and 5. The Wireshark summaries seemed to
>> >>> >> >> have a lot of dropped packets as well
>> >>> >> as
>> >>> >> >> TCP acked lost segment and TCP Previous segment lost
>> >>> >> >> messages. The traces were saved so they can be reviewed by
>> >>> >> >> others. Also the vsvr stat agg throughput results are
>> >>> >> >> included. Tests were run with and without the wireshark.
>> >>> >> >> Let's talk about reviewing the data and refining the
>> >>> >> >> experiment.
>> >>> >> >>
>> >>> >> >> Thanks
>> >>> >> >>
>> >>> >> >> Fay
>> >>> >> >>
>> >>> >> >>  << File: wiresharkexp1.xls >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> Fay Chong
>> >>> >> >> Sr. Performance Engineer
>> >>> >> >> ONStor, Inc.
>> >>> >> >> fay.chong@onstor.com
>> >>> >> >> 408.376.3130 (w)
>> >>> >> >>
>> >>> >> >>
