AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20070924171836.3fa49e80@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<maxim.kozlovsky@onstor.com>,<fay.chong@onstor.com>,<paul.hammer@onstor.com>,<jonathan.goldick@onstor.com>,<jobi.ariyamannil@onstor.com>,<brian.montero@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@onstor-exch02.onstor.net/INBOX	0	BB375AF679D4A34E9CA8DFA650E2B04E05B46437@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Mon, 24 Sep 2007 17:19:05 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: "Maxim Kozlovsky" <maxim.kozlovsky@onstor.com>
Cc: "Fay Chong" <fay.chong@onstor.com>, "Paul Hammer"
 <paul.hammer@onstor.com>, "Jonathan Goldick" <jonathan.goldick@onstor.com>,
 "Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>, "Brian Montero"
 <brian.montero@onstor.com>
Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential performance
Message-ID: <20070924171905.7eb2bf30@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E05B46437@onstor-exch02.onstor.net>
References: <BB375AF679D4A34E9CA8DFA650E2B04E05A25C3D@onstor-exch02.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05B46316@onstor-exch02.onstor.net>
	<20070924161627.066895ed@ripper.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05B463DD@onstor-exch02.onstor.net>
	<20070924164228.57f49f9f@ripper.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05B46405@onstor-exch02.onstor.net>
	<20070924170312.1f4fa1d8@ripper.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05B46437@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

I guess the first run at a combined 162MB/s is so outstanding the gods
just punish me the second time around.

There shouldn't be anything "special" about the env on ripper.  It's
running a 2.6.22.3 kernel.  As always, feel free to use it to your
heart's desire if it will help.  Perhaps there were further
"improvements" in the kernel post 2.6.18.

Cheers,

a

On Mon, 24 Sep 2007 17:11:19 -0700 "Maxim Kozlovsky"
<maxim.kozlovsky@onstor.com> wrote:

> Don't see it here:
> 
> [root@c98r48-rhel5 ~]# dd if=/dev/zero of=/2/file3 bs=32k count=10000
> 10000+0 records in
> 10000+0 records out
> 327680000 bytes (328 MB) copied, 4.79523 seconds, 68.3 MB/s
> [root@c98r48-rhel5 ~]# dd if=/2/file3 of=/2/file4 bs=32k 
> 10000+0 records in
> 10000+0 records out
> 327680000 bytes (328 MB) copied, 6.99971 seconds, 46.8 MB/s
> [root@c98r48-rhel5 ~]# dd if=/2/file4 of=/2/file5 bs=32k 
> 10000+0 records in
> 10000+0 records out
> 327680000 bytes (328 MB) copied, 7.23644 seconds, 45.3 MB/s
> 
> Could be something else you are doing on ripper.
> 
> >-----Original Message-----
> >From: Andy Sharp
> >Sent: Monday, September 24, 2007 5:03 PM
> >To: Maxim Kozlovsky
> >Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Brian
> >Montero
> >Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential performance
> >
> >Let me put it another way, it might mean something.  This is what I
> did:
> >
> >ripper$ time dd if=pack-406c14816730814f196d4c27449b79d97aa36519.pack
> >of=chewbaca bs=32k
> >9181+1 records in
> >9181+1 records out
> >300870117 bytes (301 MB) copied, 3.63615 seconds, 82.7 MB/s
> >
> >real    0m3.690s
> >user    0m0.003s
> >sys     0m0.674s
> >ripper$ time dd if=chewbaca of=apeman bs=32k
> >9181+1 records in
> >9181+1 records out
> >300870117 bytes (301 MB) copied, 246.651 seconds, 1.2 MB/s
> >
> >real    4m6.655s
> >user    0m0.000s
> >sys     0m0.940s
> >ripper$
> >
> >
> >So essentially the second time, it fell off a cliff.
> >
> >
> >
> >On Mon, 24 Sep 2007 16:52:09 -0700 "Maxim Kozlovsky"
> ><maxim.kozlovsky@onstor.com> wrote:
> >
> >> I think this is just you, Andy:
> >>
> >> [root@c98r48-rhel5 ~]# time dd if=/2/file1 of=/2/file2 bs=32k
> >> count=131072
> >> 131072+0 records in
> >> 131072+0 records out
> >> 4294967296 bytes (4.3 GB) copied, 103.266 seconds, 41.6 MB/s
> >>
> >> real    1m43.282s
> >> user    0m0.109s
> >> sys     0m7.887s
> >>
> >>
> >>
> >> >-----Original Message-----
> >> >From: Andy Sharp
> >> >Sent: Monday, September 24, 2007 4:42 PM
> >> >To: Maxim Kozlovsky
> >> >Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil;
> Brian
> >> >Montero
> >> >Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential performance
> >> >
> >> >Exqueezeme?
> >> >
> >> >I'd give my kingdom for any of those horses: 30, 50, 60, 80 MB/s.
> >> >Those differences amount to a big "so what" compared to ONE MB/s,
> >> >IMHO.
> >> >
> >> >For instance, if my race bike can't shift out of neutral, I'm
> >> >gonna want to fix that problem before I fix the top-end
> >> >carburetion problem.
> >> >
> >> >Fay, please note the kernel version number on these Redmondhat
> >> releases,
> >> >even though RH kernel versions correspond very little to real
> >> >kernel version numbers, at least we'd have a shotgun's
> >> >difference.  For instance, I believe that RHEL3 is a 2.4 kernel,
> >> >and RHEL5 is some kind of 2.6 kernel.  Most likely you can use
> >> >the 'uname -r' command to show the kernel version number.
> >> >
> >> >Cheers,
> >> >
> >> >a
> >> >
> >> >On Mon, 24 Sep 2007 16:22:49 -0700 "Maxim Kozlovsky"
> >> ><maxim.kozlovsky@onstor.com> wrote:
> >> >
> >> >> Sure, but we are not there yet. For now, we have a problem with
> >> >> even running unidirectional write, so trying bidirectional test
> >> >> will not give a lot of new information.
> >> >>
> >> >> >-----Original Message-----
> >> >> >From: Andy Sharp
> >> >> >Sent: Monday, September 24, 2007 4:16 PM
> >> >> >To: Maxim Kozlovsky
> >> >> >Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil;
> >> Brian
> >> >> >Montero
> >> >> >Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential
> >> >> >performance
> >> >> >
> >> >> >The test that should demonstrate the problem is not two
> >> >> >unidirectional tests running at the same time, but a
> bidirectional
> >> >> >test:
> >> >> >
> >> >> >
> >> >> >$ time dd if=chewbaca of=apeman bs=32k
> >> >> >9181+1 records in
> >> >> >9181+1 records out
> >> >> >300870117 bytes (301 MB) copied, 246.651 seconds, 1.2 MB/s
> >> >> >
> >> >> >real    4m6.655s
> >> >> >user    0m0.000s
> >> >> >sys     0m0.940s
> >> >> >
> >> >> >
> >> >> >Cheers,
> >> >> >
> >> >> >a
> >> >> >
> >> >> >On Mon, 24 Sep 2007 14:40:01 -0700 "Maxim Kozlovsky"
> >> >> ><maxim.kozlovsky@onstor.com> wrote:
> >> >> >
> >> >> >> What exactly was this test doing? Is it single direction, or
> >> >> >> bidirectional? The excel spreadsheet seems to imply that it
> >> >> >> is bidirectional.
> >> >> >>
> >> >> >> I've tried bidirectional test and got completely different
> >> >> >> results with almost identical performance:
> >> >> >>
> >> >> >> rhel3 - read 30MB/sec write 46MB/sec,
> >> >> >> rhel5 - read 29.6MB/sec write 53.1MB/s
> >> >> >>
> >> >> >> The test that I was running:
> >> >> >>
> >> >> >> Rh3
> >> >> >>
> >> >> >> time dd if=/1/file1 of=/dev/zero bs=32k ; skill -INT dd &
> >> >> >> 86187+0 records in
> >> >> >> 86187+0 records out
> >> >> >>
> >> >> >> real    1m27.021s
> >> >> >> user    0m0.090s
> >> >> >> sys     0m11.740s
> >> >> >>
> >> >> >> time dd if=/dev/zero of=/1/file2 bs=32k count=131072 ; skill
> >> >> >> -INT dd
> >> >> &
> >> >> >> 131072+0 records in
> >> >> >> 131072+0 records out
> >> >> >>
> >> >> >> real    1m28.539s
> >> >> >> user    0m0.160s
> >> >> >> sys     0m19.460s
> >> >> >>
> >> >> >> rh5:
> >> >> >>
> >> >> >> time dd if=/dev/zero of=/1/file2 bs=32k count=131072 ; skill
> >> >> >> -INT dd& 131072+0 records in
> >> >> >> 131072+0 records out
> >> >> >> 4294967296 bytes (4.3 GB) copied, 80.8301 seconds, 53.1 MB/s
> >> >> >>
> >> >> >> real    1m20.834s
> >> >> >> user    0m0.081s
> >> >> >> sys     0m14.097s
> >> >> >>
> >> >> >> time dd if=/1/file1 of=/dev/zero bs=32k ; skill -INT dd &
> >> >> >> 72297+0 records in
> >> >> >> 72296+0 records out
> >> >> >> 2368995328 bytes (2.4 GB) copied, 80.0069 seconds, 29.6 MB/s
> >> >> >>
> >> >> >>
> >> >> >> real    1m20.023s
> >> >> >> user    0m0.047s
> >> >> >> sys     0m1.866s
> >> >> >>
> >> >> >> Couple of things:
> >> >> >>
> >> >> >> Do not post the results of running "vs stat agg" as actual
> >> >> >> performance, who knows what this code is doing. For the case
> >> >> >> of dd the performance can be measured directly as shown
> >> >> >> above.
> >> >> >>
> >> >> >> With this high volume of traffic the Wireshark is lossy, you
> can
> >> >> >> not rely on it to tell anything about dropped packets. Look
> >> >> >> at the TCP stats for the number of retransmitted packets on
> >> >> >> the client and on the filer instead (which by the way was 0
> >> >> >> in my test on both rhel5 and rhel3).
> >> >> >>
> >> >> >> In probably already doing this, but I thought I'll mention it
> >> >> >> just in case - you should unmount and remount the volume on
> >> >> >> the client and
> >> >> vol
> >> >> >> offline /vol online the volume on the filer between the tests
> to
> >> >> >> make sure consistent initial state is used.
> >> >> >>
> >> >> >> Could you please create a script(s) which can be used to run
> >> >> >> your test(s) so we know we are on a same page?
> >> >> >>
> >> >> >> Max
> >> >> >>
> >> >> >> _____________________________________________
> >> >> >> From: Fay Chong
> >> >> >> Sent: Sunday, September 23, 2007 5:49 PM
> >> >> >> To: Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Andy
> Sharp;
> >> >> Maxim
> >> >> >> Kozlovsky
> >> >> >> Cc: Brian Montero; Fay Chong
> >> >> >> Subject: Wireshark Red Hat 3 and 5 NFS sequential performance
> >> >> >>
> >> >> >> Hi,
> >> >> >>
> >> >> >> Attached are some results from the Wireshark trace
> >> >> >> experiments on NFS sequential read and write with Red Hat
> >> >> >> Linux release 3 and 5. The Wireshark summaries seemed to
> >> >> >> have a lot of dropped packets as well
> >> >> as
> >> >> >> TCP acked lost segment and TCP Previous segment lost
> >> >> >> messages. The traces were saved so they can be reviewed by
> >> >> >> others. Also the vsvr stat agg throughput results are
> >> >> >> included. Tests were run with and without the wireshark.
> >> >> >> Let's talk about reviewing the data and refining the
> >> >> >> experiment.
> >> >> >>
> >> >> >> Thanks
> >> >> >>
> >> >> >> Fay
> >> >> >>
> >> >> >>  << File: wiresharkexp1.xls >>
> >> >> >>
> >> >> >>
> >> >> >> Fay Chong
> >> >> >> Sr. Performance Engineer
> >> >> >> ONStor, Inc.
> >> >> >> fay.chong@onstor.com
> >> >> >> 408.376.3130 (w)
> >> >> >>
> >> >> >>
