AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20070924172033.26484b90@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<maxim.kozlovsky@onstor.com>,<fay.chong@onstor.com>,<paul.hammer@onstor.com>,<jonathan.goldick@onstor.com>,<jobi.ariyamannil@onstor.com>,<brian.montero@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@onstor-exch02.onstor.net/INBOX	0	BB375AF679D4A34E9CA8DFA650E2B04E05B4643B@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Mon, 24 Sep 2007 17:22:22 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: "Maxim Kozlovsky" <maxim.kozlovsky@onstor.com>
Cc: "Fay Chong" <fay.chong@onstor.com>, "Paul Hammer"
 <paul.hammer@onstor.com>, "Jonathan Goldick" <jonathan.goldick@onstor.com>,
 "Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>, "Brian Montero"
 <brian.montero@onstor.com>
Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential performance
Message-ID: <20070924172222.5a11a3be@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E05B4643B@onstor-exch02.onstor.net>
References: <BB375AF679D4A34E9CA8DFA650E2B04E05A25C3D@onstor-exch02.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05B46316@onstor-exch02.onstor.net>
	<20070924161627.066895ed@ripper.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05B463DD@onstor-exch02.onstor.net>
	<20070924164228.57f49f9f@ripper.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05B46405@onstor-exch02.onstor.net>
	<20070924170312.1f4fa1d8@ripper.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05B4643B@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

It's a gift, what can I say?  Besides, I thought you were a real
'merican?  I mean, apart from that woosie Russian accent.

On Mon, 24 Sep 2007 17:13:27 -0700 "Maxim Kozlovsky"
<maxim.kozlovsky@onstor.com> wrote:

> 
> By the way, I wonder if I would ever learn to pick the file names as
> real American people do.
> 
> >
> >>-----Original Message-----
> >>From: Andy Sharp
> >>Sent: Monday, September 24, 2007 5:03 PM
> >>To: Maxim Kozlovsky
> >>Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil;
> >>Brian Montero
> >>Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential performance
> >>
> >>Let me put it another way, it might mean something.  This is what I
> did:
> >>
> >>ripper$ time dd
> >>if=pack-406c14816730814f196d4c27449b79d97aa36519.pack of=chewbaca
> >>bs=32k 9181+1 records in
> >>9181+1 records out
> >>300870117 bytes (301 MB) copied, 3.63615 seconds, 82.7 MB/s
> >>
> >>real    0m3.690s
> >>user    0m0.003s
> >>sys     0m0.674s
> >>ripper$ time dd if=chewbaca of=apeman bs=32k
> >>9181+1 records in
> >>9181+1 records out
> >>300870117 bytes (301 MB) copied, 246.651 seconds, 1.2 MB/s
> >>
> >>real    4m6.655s
> >>user    0m0.000s
> >>sys     0m0.940s
> >>ripper$
> >>
> >>
> >>So essentially the second time, it fell off a cliff.
> >>
> >>
> >>
> >>On Mon, 24 Sep 2007 16:52:09 -0700 "Maxim Kozlovsky"
> >><maxim.kozlovsky@onstor.com> wrote:
> >>
> >>> I think this is just you, Andy:
> >>>
> >>> [root@c98r48-rhel5 ~]# time dd if=/2/file1 of=/2/file2 bs=32k
> >>> count=131072
> >>> 131072+0 records in
> >>> 131072+0 records out
> >>> 4294967296 bytes (4.3 GB) copied, 103.266 seconds, 41.6 MB/s
> >>>
> >>> real    1m43.282s
> >>> user    0m0.109s
> >>> sys     0m7.887s
> >>>
> >>>
> >>>
> >>> >-----Original Message-----
> >>> >From: Andy Sharp
> >>> >Sent: Monday, September 24, 2007 4:42 PM
> >>> >To: Maxim Kozlovsky
> >>> >Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi Ariyamannil;
> Brian
> >>> >Montero
> >>> >Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential performance
> >>> >
> >>> >Exqueezeme?
> >>> >
> >>> >I'd give my kingdom for any of those horses: 30, 50, 60, 80 MB/s.
> >>> >Those differences amount to a big "so what" compared to ONE MB/s,
> >>> >IMHO.
> >>> >
> >>> >For instance, if my race bike can't shift out of neutral, I'm
> >>> >gonna want to fix that problem before I fix the top-end
> >>> >carburetion problem.
> >>> >
> >>> >Fay, please note the kernel version number on these Redmondhat
> >>> releases,
> >>> >even though RH kernel versions correspond very little to real
> kernel
> >>> >version numbers, at least we'd have a shotgun's difference.  For
> >>> >instance, I believe that RHEL3 is a 2.4 kernel, and RHEL5 is some
> >>> >kind of 2.6 kernel.  Most likely you can use the 'uname -r'
> >>> >command to show the kernel version number.
> >>> >
> >>> >Cheers,
> >>> >
> >>> >a
> >>> >
> >>> >On Mon, 24 Sep 2007 16:22:49 -0700 "Maxim Kozlovsky"
> >>> ><maxim.kozlovsky@onstor.com> wrote:
> >>> >
> >>> >> Sure, but we are not there yet. For now, we have a problem with
> >>> >> even running unidirectional write, so trying bidirectional test
> >>> >> will not give a lot of new information.
> >>> >>
> >>> >> >-----Original Message-----
> >>> >> >From: Andy Sharp
> >>> >> >Sent: Monday, September 24, 2007 4:16 PM
> >>> >> >To: Maxim Kozlovsky
> >>> >> >Cc: Fay Chong; Paul Hammer; Jonathan Goldick; Jobi
> >>> >> >Ariyamannil;
> >>> Brian
> >>> >> >Montero
> >>> >> >Subject: Re: Wireshark Red Hat 3 and 5 NFS sequential
> performance
> >>> >> >
> >>> >> >The test that should demonstrate the problem is not two
> >>> >> >unidirectional tests running at the same time, but a
> bidirectional
> >>> >> >test:
> >>> >> >
> >>> >> >
> >>> >> >$ time dd if=chewbaca of=apeman bs=32k
> >>> >> >9181+1 records in
> >>> >> >9181+1 records out
> >>> >> >300870117 bytes (301 MB) copied, 246.651 seconds, 1.2 MB/s
> >>> >> >
> >>> >> >real    4m6.655s
> >>> >> >user    0m0.000s
> >>> >> >sys     0m0.940s
> >>> >> >
> >>> >> >
> >>> >> >Cheers,
> >>> >> >
> >>> >> >a
> >>> >> >
> >>> >> >On Mon, 24 Sep 2007 14:40:01 -0700 "Maxim Kozlovsky"
> >>> >> ><maxim.kozlovsky@onstor.com> wrote:
> >>> >> >
> >>> >> >> What exactly was this test doing? Is it single direction, or
> >>> >> >> bidirectional? The excel spreadsheet seems to imply that it
> >>> >> >> is bidirectional.
> >>> >> >>
> >>> >> >> I've tried bidirectional test and got completely different
> >>> >> >> results with almost identical performance:
> >>> >> >>
> >>> >> >> rhel3 - read 30MB/sec write 46MB/sec,
> >>> >> >> rhel5 - read 29.6MB/sec write 53.1MB/s
> >>> >> >>
> >>> >> >> The test that I was running:
> >>> >> >>
> >>> >> >> Rh3
> >>> >> >>
> >>> >> >> time dd if=/1/file1 of=/dev/zero bs=32k ; skill -INT dd &
> >>> >> >> 86187+0 records in
> >>> >> >> 86187+0 records out
> >>> >> >>
> >>> >> >> real    1m27.021s
> >>> >> >> user    0m0.090s
> >>> >> >> sys     0m11.740s
> >>> >> >>
> >>> >> >> time dd if=/dev/zero of=/1/file2 bs=32k count=131072 ; skill
> >>> >> >> -INT dd
> >>> >> &
> >>> >> >> 131072+0 records in
> >>> >> >> 131072+0 records out
> >>> >> >>
> >>> >> >> real    1m28.539s
> >>> >> >> user    0m0.160s
> >>> >> >> sys     0m19.460s
> >>> >> >>
> >>> >> >> rh5:
> >>> >> >>
> >>> >> >> time dd if=/dev/zero of=/1/file2 bs=32k count=131072 ; skill
> >>> >> >> -INT dd& 131072+0 records in
> >>> >> >> 131072+0 records out
> >>> >> >> 4294967296 bytes (4.3 GB) copied, 80.8301 seconds, 53.1 MB/s
> >>> >> >>
> >>> >> >> real    1m20.834s
> >>> >> >> user    0m0.081s
> >>> >> >> sys     0m14.097s
> >>> >> >>
> >>> >> >> time dd if=/1/file1 of=/dev/zero bs=32k ; skill -INT dd &
> >>> >> >> 72297+0 records in
> >>> >> >> 72296+0 records out
> >>> >> >> 2368995328 bytes (2.4 GB) copied, 80.0069 seconds, 29.6 MB/s
> >>> >> >>
> >>> >> >>
> >>> >> >> real    1m20.023s
> >>> >> >> user    0m0.047s
> >>> >> >> sys     0m1.866s
> >>> >> >>
> >>> >> >> Couple of things:
> >>> >> >>
> >>> >> >> Do not post the results of running "vs stat agg" as actual
> >>> >> >> performance, who knows what this code is doing. For the case
> of
> >>> >> >> dd the performance can be measured directly as shown above.
> >>> >> >>
> >>> >> >> With this high volume of traffic the Wireshark is lossy, you
> can
> >>> >> >> not rely on it to tell anything about dropped packets. Look
> >>> >> >> at the TCP stats for the number of retransmitted packets on
> >>> >> >> the client and on the filer instead (which by the way was 0
> >>> >> >> in my test on both rhel5 and rhel3).
> >>> >> >>
> >>> >> >> In probably already doing this, but I thought I'll mention
> >>> >> >> it just in case - you should unmount and remount the volume
> >>> >> >> on
> the
> >>> >> >> client and
> >>> >> vol
> >>> >> >> offline /vol online the volume on the filer between the
> >>> >> >> tests
> to
> >>> >> >> make sure consistent initial state is used.
> >>> >> >>
> >>> >> >> Could you please create a script(s) which can be used to run
> >>> >> >> your test(s) so we know we are on a same page?
> >>> >> >>
> >>> >> >> Max
> >>> >> >>
> >>> >> >> _____________________________________________
> >>> >> >> From: Fay Chong
> >>> >> >> Sent: Sunday, September 23, 2007 5:49 PM
> >>> >> >> To: Paul Hammer; Jonathan Goldick; Jobi Ariyamannil; Andy
> Sharp;
> >>> >> Maxim
> >>> >> >> Kozlovsky
> >>> >> >> Cc: Brian Montero; Fay Chong
> >>> >> >> Subject: Wireshark Red Hat 3 and 5 NFS sequential
> >>> >> >> performance
> >>> >> >>
> >>> >> >> Hi,
> >>> >> >>
> >>> >> >> Attached are some results from the Wireshark trace
> >>> >> >> experiments on NFS sequential read and write with Red Hat
> >>> >> >> Linux release 3 and 5. The Wireshark summaries seemed to
> >>> >> >> have a lot of dropped packets as well
> >>> >> as
> >>> >> >> TCP acked lost segment and TCP Previous segment lost
> >>> >> >> messages. The traces were saved so they can be reviewed by
> >>> >> >> others. Also the vsvr stat agg throughput results are
> >>> >> >> included. Tests were run with and without the wireshark.
> >>> >> >> Let's talk about reviewing the data and refining the
> >>> >> >> experiment.
> >>> >> >>
> >>> >> >> Thanks
> >>> >> >>
> >>> >> >> Fay
> >>> >> >>
> >>> >> >>  << File: wiresharkexp1.xls >>
> >>> >> >>
> >>> >> >>
> >>> >> >> Fay Chong
> >>> >> >> Sr. Performance Engineer
> >>> >> >> ONStor, Inc.
> >>> >> >> fay.chong@onstor.com
> >>> >> >> 408.376.3130 (w)
> >>> >> >>
> >>> >> >>
