AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20090215171617.677776e8@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:exch1.onstor.net
NSV:
SSH:
R:<patrick.haverty@onstor.com>,<dl-Leopard@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@exch1.onstor.net/INBOX	66151	2779531E7C760D4491C96305019FEEB51851F49354@exch1.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Sun, 15 Feb 2009 17:17:12 -0800
From: Andrew Sharp <andy.sharp@onstor.com>
To: Patrick Haverty <patrick.haverty@onstor.com>
Cc: dl-Leopard <dl-Leopard@onstor.com>
Subject: Re: Finding Leopard Performance
Message-ID: <20090215171712.04907199@ripper.onstor.net>
In-Reply-To: <2779531E7C760D4491C96305019FEEB51851F49354@exch1.onstor.net>
References: <2779531E7C760D4491C96305019FEEB51851F49354@exch1.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

On Thu, 12 Feb 2009 21:02:13 -0800 Patrick Haverty
<patrick.haverty@onstor.com> wrote:

> 
> 
> If you go into "\\Mightydog\Program<file://\\Mightydog\Program>
> Management\Leopard\Performance" and open
> Leopard_Perf_Data_2009-02-12.pdf you'll see I've developed "Report
> Formatting Schizophrenia"  I've collected lots of data running dd,
> sio, bonnie++ benchmark, and Iometer.  I also added some of the dd
> and sio results to that folder if anyone wants a look before it can
> be compiled into something easier to read.  I've spent a lot of time
> running tests and rerunning with different parameters (transfer block
> size, default ZFS block size, file size, number of processes),
> different JBODs, different storage pool configurations, different
> number of clients, and a couple different types of Leopard hardware.
> And I've spent more time trying to collect the data and pare it into
> some sort of useful information.  Unfortunately, for various reasons,
> I still don't have a complete set of performance data for our
> intended standard Leopard system.
> 
> Leopard_Perf_Data_2009-02-12.pdf is a collection of some of what I
> thought were the highlights, mostly for comparing the various storage
> pools, more than performance testing a system.  If you don't want to
> get lost in the minutia, just look at the first page which compares
> some initial dd and sio results (NFS) to a Cougar-Fujitsu system.
> I'm not sure that's even a valid comparison because the Leopard is
> using a DotHill array with SAS drives and I think at the time there
> were some client side limitations on the Leopard.  The lower chart on
> that page just shows the total throughput seen by 1, 2, 3 and 4
> client(s) at a time accessing the Leopard system.
> 
> The data used for the rest of the document was generated from
> bonnie++ benchmark tests and Iometer.  The bonnie++ benchmark test is
> great in that it only requires the Leopard be attached to storage to
> run, no clients needed.  It's a good way to see and compare the
> performance of the disk subsystem.  So far I've only been able to run
> Iometer over a single GbE link, which is easily saturated by some
> tests, so it hasn't been as useful to find peak performance yet.  And
> I haven't yet been able to define the correlation between the results
> indicated by bonnie++ to what is seen by the other type tests.

Bonnie directly on the hardware might be legitimate for the single
purpose of comparing different raid setups.  But then again it might
not be, since you never quite know how the extra overhead of
CIFS/NFS/TCP/IP might affect other things.  But I might be tempted to
use if for that anyway ~:^)  A fast array setup that uses a lot of CPU
might be slower with local bonnie because bonnie itself uses a lot of
CPU.

I'm curious as to what might be causing the iometer restriction you
mention -- only over a single GigE link?  Is that just lab
setup/resource restrictions?  I'm also curious as to what you mean,
since the first page shows results with multiple clients and numbers
that exceed what 1 GigE link can produce.

Since you've been running bonnie directly on the hardware it is missing
the networking processing component that the other tests have, which is
probably why the results are not really correlateable.  Alert Webster's.

> I think sheet seven is the most interesting and there are two reasons
> for that.  But you all are going to have to judge for yourselves.
> BTW, that chart not only compares performance with 8, 16, and 32GB
> DRAM, but also compares a pool, Standard, that is created from two
> hardware protected (4+1 RAID5 and 5+1 RAID5) volumes presented to ZFS
> as virtual devices (which do not have RAID protection by ZFS enabled)
> to a pool that is created from raw devices, from a JBOD, presented to
> ZFS and configured as a pool, Standard-lsi, consisting of a 4+1
> RAIDZ1 group and a 5+1 RAIDZ1 group.

On page 5 & 6, it looks like something got funky with the cached 8k
random writes.  Since it's cached, the two should be the same, just
like all the other cached tests (is this "cached" v. "uncached" thing a
ZFS or Slowaris thing?  I've not seen this before).

Page 7, well, what can I say, I'm dying to know what it looks like when
NFS/TCP/IP is in the mix.  I think you'll see these boxes change shape
in interesting ways that definitely do matter.

I think page 10 is the most interesting.  It backs up what I've been
suggesting all along: the best performer is the single quad-core
opteron with 4GB of memory ~:^)  Although, in the paraphrased words of
the immortal bard Sir Mix-A-Lot, I like Butser.  It might be even
faster if it had opterons.

I'd like to get more detail on the methodology used for pages 12 and
13.  Without it I'm not able to digest what the numbers might mean.
Like 0.07 MB/s throughput?  2.29 MB/s.  Really?

> I welcome any comments towards formatting data to make the
> information pop, but I'll tolerate criticisms to that end. :)

A lot of hard work obviously went into all this.  Thanks for that.
Now, can we get you to do a bunch more?  ~:^)  But seriously, as a
general question to the core team folks: is there a plan to do more
stringent, client based performance benching with Leopard?  I would
assume that we would need to do this in a bigger lab setup like in the
Campbell Elab.  The Spec setup doesn't make sense, but access to more
clients, switches and whatnot.

Really good job collecting and collating all this data.

Thanks,

a
