AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20070924165729.774e6591@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<maxim.kozlovsky@onstor.com>,<fay.chong@onstor.com>,<paul.hammer@onstor.com>,<jonathan.goldick@onstor.com>,<jobi.ariyamannil@onstor.com>,<brian.montero@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@onstor-exch02.onstor.net/INBOX	0	BB375AF679D4A34E9CA8DFA650E2B04E05B46405@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Mon, 24 Sep 2007 16:57:57 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: "Maxim Kozlovsky" <maxim.kozlovsky@onstor.com>
Cc: "Fay Chong" <fay.chong@onstor.com>, "Paul Hammer"
 <paul.hammer@onstor.com>, "Jonathan Goldick" <jonathan.goldick@onstor.com>,
 "Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>, "Brian Montero"
 <brian.montero@onstor.com>
Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential performance
Message-ID: <20070924165757.27474609@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E05B46405@onstor-exch02.onstor.net>
References: <BB375AF679D4A34E9CA8DFA650E2B04E05A25C3D@onstor-exch02.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05B46316@onstor-exch02.onstor.net>
	<20070924161627.066895ed@ripper.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05B463DD@onstor-exch02.onstor.net>
	<20070924164228.57f49f9f@ripper.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05B46405@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

How many times did you try it?

On Mon, 24 Sep 2007 16:52:09 -0700 "Maxim Kozlovsky"
<maxim.kozlovsky@onstor.com> wrote:

> I think this is just you, Andy:
> 
> [root@c98r48-rhel5 ~]# time dd if=/2/file1 of=/2/file2 bs=32k
> count=131072
> 131072+0 records in
> 131072+0 records out
> 4294967296 bytes (4.3 GB) copied, 103.266 seconds, 41.6 MB/s
> 
> real    1m43.282s
> user    0m0.109s
> sys     0m7.887s
> 
> 
> 
> >-----Original Message-----
> >From: Andy Sharp
> >Sent: Monday, September 24, 2007 4:42 PM
> >To: Maxim Kozlovsky
> >Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Brian
> >Montero
> >Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential performance
> >
> >Exqueezeme?
> >
> >I'd give my kingdom for any of those horses: 30, 50, 60, 80 MB/s.
> >Those differences amount to a big "so what" compared to ONE MB/s,
> >IMHO.
> >
> >For instance, if my race bike can't shift out of neutral, I'm gonna
> >want to fix that problem before I fix the top-end carburetion
> >problem.
> >
> >Fay, please note the kernel version number on these Redmondhat
> releases,
> >even though RH kernel versions correspond very little to real kernel
> >version numbers, at least we'd have a shotgun's difference.  For
> >instance, I believe that RHEL3 is a 2.4 kernel, and RHEL5 is some
> >kind of 2.6 kernel.  Most likely you can use the 'uname -r' command
> >to show the kernel version number.
> >
> >Cheers,
> >
> >a
> >
> >On Mon, 24 Sep 2007 16:22:49 -0700 "Maxim Kozlovsky"
> ><maxim.kozlovsky@onstor.com> wrote:
> >
> >> Sure, but we are not there yet. For now, we have a problem with
> >> even running unidirectional write, so trying bidirectional test
> >> will not give a lot of new information.
> >>
> >> >-----Original Message-----
> >> >From: Andy Sharp
> >> >Sent: Monday, September 24, 2007 4:16 PM
> >> >To: Maxim Kozlovsky
> >> >Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil;
> Brian
> >> >Montero
> >> >Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential performance
> >> >
> >> >The test that should demonstrate the problem is not two
> >> >unidirectional tests running at the same time, but a bidirectional
> >> >test:
> >> >
> >> >
> >> >$ time dd if=chewbaca of=apeman bs=32k
> >> >9181+1 records in
> >> >9181+1 records out
> >> >300870117 bytes (301 MB) copied, 246.651 seconds, 1.2 MB/s
> >> >
> >> >real    4m6.655s
> >> >user    0m0.000s
> >> >sys     0m0.940s
> >> >
> >> >
> >> >Cheers,
> >> >
> >> >a
> >> >
> >> >On Mon, 24 Sep 2007 14:40:01 -0700 "Maxim Kozlovsky"
> >> ><maxim.kozlovsky@onstor.com> wrote:
> >> >
> >> >> What exactly was this test doing? Is it single direction, or
> >> >> bidirectional? The excel spreadsheet seems to imply that it is
> >> >> bidirectional.
> >> >>
> >> >> I've tried bidirectional test and got completely different
> >> >> results with almost identical performance:
> >> >>
> >> >> rhel3 - read 30MB/sec write 46MB/sec,
> >> >> rhel5 - read 29.6MB/sec write 53.1MB/s
> >> >>
> >> >> The test that I was running:
> >> >>
> >> >> Rh3
> >> >>
> >> >> time dd if=/1/file1 of=/dev/zero bs=32k ; skill -INT dd &
> >> >> 86187+0 records in
> >> >> 86187+0 records out
> >> >>
> >> >> real    1m27.021s
> >> >> user    0m0.090s
> >> >> sys     0m11.740s
> >> >>
> >> >> time dd if=/dev/zero of=/1/file2 bs=32k count=131072 ; skill
> >> >> -INT dd
> >> &
> >> >> 131072+0 records in
> >> >> 131072+0 records out
> >> >>
> >> >> real    1m28.539s
> >> >> user    0m0.160s
> >> >> sys     0m19.460s
> >> >>
> >> >> rh5:
> >> >>
> >> >> time dd if=/dev/zero of=/1/file2 bs=32k count=131072 ; skill
> >> >> -INT dd& 131072+0 records in
> >> >> 131072+0 records out
> >> >> 4294967296 bytes (4.3 GB) copied, 80.8301 seconds, 53.1 MB/s
> >> >>
> >> >> real    1m20.834s
> >> >> user    0m0.081s
> >> >> sys     0m14.097s
> >> >>
> >> >> time dd if=/1/file1 of=/dev/zero bs=32k ; skill -INT dd &
> >> >> 72297+0 records in
> >> >> 72296+0 records out
> >> >> 2368995328 bytes (2.4 GB) copied, 80.0069 seconds, 29.6 MB/s
> >> >>
> >> >>
> >> >> real    1m20.023s
> >> >> user    0m0.047s
> >> >> sys     0m1.866s
> >> >>
> >> >> Couple of things:
> >> >>
> >> >> Do not post the results of running "vs stat agg" as actual
> >> >> performance, who knows what this code is doing. For the case of
> >> >> dd the performance can be measured directly as shown above.
> >> >>
> >> >> With this high volume of traffic the Wireshark is lossy, you can
> >> >> not rely on it to tell anything about dropped packets. Look at
> >> >> the TCP stats for the number of retransmitted packets on the
> >> >> client and on the filer instead (which by the way was 0 in my
> >> >> test on both rhel5 and rhel3).
> >> >>
> >> >> In probably already doing this, but I thought I'll mention it
> >> >> just in case - you should unmount and remount the volume on the
> >> >> client and
> >> vol
> >> >> offline /vol online the volume on the filer between the tests to
> >> >> make sure consistent initial state is used.
> >> >>
> >> >> Could you please create a script(s) which can be used to run
> >> >> your test(s) so we know we are on a same page?
> >> >>
> >> >> Max
> >> >>
> >> >> _____________________________________________
> >> >> From: Fay Chong
> >> >> Sent: Sunday, September 23, 2007 5:49 PM
> >> >> To: Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Andy Sharp;
> >> Maxim
> >> >> Kozlovsky
> >> >> Cc: Brian Montero; Fay Chong
> >> >> Subject: Wireshark Red Hat 3 and 5 NFS sequential performance
> >> >>
> >> >> Hi,
> >> >>
> >> >> Attached are some results from the Wireshark trace experiments
> >> >> on NFS sequential read and write with Red Hat Linux release 3
> >> >> and 5. The Wireshark summaries seemed to have a lot of dropped
> >> >> packets as well
> >> as
> >> >> TCP acked lost segment and TCP Previous segment lost messages.
> >> >> The traces were saved so they can be reviewed by others. Also
> >> >> the vsvr stat agg throughput results are included. Tests were
> >> >> run with and without the wireshark. Let's talk about reviewing
> >> >> the data and refining the experiment.
> >> >>
> >> >> Thanks
> >> >>
> >> >> Fay
> >> >>
> >> >>  << File: wiresharkexp1.xls >>
> >> >>
> >> >>
> >> >> Fay Chong
> >> >> Sr. Performance Engineer
> >> >> ONStor, Inc.
> >> >> fay.chong@onstor.com
> >> >> 408.376.3130 (w)
> >> >>
> >> >>
