AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:
CFG:
PT:0
S:andy.sharp@lsi.com
RQ:
SSV:mhbs.lsil.com
NSV:
SSH:
R:<ralf@linux-mips.org>
MAID:2
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/LSI/INBOX	0	20091202202458.GA26457@linux-mips.org
X-Sylpheed-End-Special-Headers: 1
Date: Wed, 2 Dec 2009 13:38:56 -0800
From: Andrew Sharp <andy.sharp@lsi.com>
To: Ralf Baechle <ralf@linux-mips.org>
Subject: Re: NUMA development for sb1 processors
Message-ID: <20091202133856.3c18ed05@ripper.onstor.net>
In-Reply-To: <20091202202458.GA26457@linux-mips.org>
References: <20091130143634.58416df2@ripper.onstor.net>
	<20091201012442.GC31892@linux-mips.org>
	<20091202114756.5b40ea39@ripper.onstor.net>
	<20091202202458.GA26457@linux-mips.org>
Organization: LSI
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

On Wed, 2 Dec 2009 13:24:58 -0700 Ralf Baechle <ralf@linux-mips.org>
wrote:

> On Wed, Dec 02, 2009 at 11:47:56AM -0800, Andrew Sharp wrote:
> 
> > Perhaps odd was the wrong word.  It seems to be capable of much more
> > complexity than I think I need ... nodes with/without local memory,
> > heirarchical topology, etc.
> 
> True though the networking topology isn't that complex.  It's a
> hypercube with 1 or 2 CPUs per node.  A 16 node system would be a 4
> dimensional hypercube.  Even larger systems may have short cuts known
> as express links installed which basically add diagonals through such
> a hypercube.  This whole scheme scales up to 512 CPUs / 256 nodes.  I
> think 1024 CPUs / 512 nodes was possible but no such configuration
> was ever sold.  Larger systems would be clusters.  ASCI Blue Mountain
> was a cluster of 48 128 CPU systems for a total of 6144 CPUs and
> 1.5TB memory.  That's '98 technology btw. and of course that box held
> the top spot of the top 500 list.
> 
> > > I'm not sure if flattened device tree covers NUMA but if so that
> > > would be strong recommendation.  IP27 btw. passes all sorts of
> > > information from the PROM - see
> > > arch/mips/include/asm/sn/klconfig.h.  The key there is that using
> > > klconfig does not involve using any calls into the firmware which
> > > traditionally are problems - what are the calling conventions
> > > (o32, N32, N64), stack alignment, locking conventions etc.  A
> > > data structure is so much easier to handle.
> > 
> > Flattened device tree ... where to look?  I'll take a gander at
> > klconfig.h.
> 
> I don't have any specific URLs at hand but your favorite search engine
> should find plenty of material about the Flattened Device Tree.  In
> short it's based on the way that OpenFirmware on SPARC and PPC
> systems is passing configuration information.  Flattened because it's
> a simple, flat data structure.
> 
> >  BTW, I'm curious what does the 'sn' v. ip27 mean?  Is that
> > shared with a platform besides ip27?
> 
> Ah, complicated :-) SN stands for Scalable Node. There were:
> 
>  * SN00 specifically only stands for the Origin 200
>  * SN0 is IP27 which was marketed as Origin 200 / Origin 2000.  If
> equipped with graphics an Origin 2000 was sold as Onyx 2.
>  * SN1 did exist twice:
> 
>     * MIPS-based as IP35 aka Origin 300 / 3000 rsp Onyx 3 with
> graphics.
>     * An IA64-based version was in the works but the product was
> cancelled. That project was also known as SNIA.
>  * SN2 which was only available based on the IA64, also known as
> Altix.
> 
> > > I'm a bit surprised that you're mailing me about this - it is my
> > > understanding that for Broadcom the SB1-based stuff is more or
> > > less dead since a long time and that the NUMA stuff apparently is
> > > fairly slow though that may to a degree also depend on factors
> > > outside the BCM1480.
> > 
> > Heh.  Interesting questions.  Since we just kicked off a new build
> > of 100 boxes, ie. 200 1480s, Broadcom said the demand was so high
> > the lead time would be 3 months.  And there are plans for selling
> > boxes numbering in the 4 figures.  They'll keep making them if they
> > keep selling them.
> 
> In my opinion a very well designed CPU core and SOC with only very few
> rough edges.  I'd love to work on more of these.
> 
> > According to our sources at Broadcom, we are the only dual 1480
> > design they know of, never mind NUMA.  We are running the HT bus at
> > 600 MHz, which I'm guessing will be fast enough.  If necessary,
> > I'll try and kick in the feature where the kernel text is copied to
> > both nodes.
> 
> Indeed cool stuff.  Send me a box for xmas ;-)

I could probably make that happen.  Would you be interested
in a dual 1250 until then?  I could probably make that happen quite
easily.  Bonus features of the dual 1250: it also has a 900MHz dual core
R9000 as well as some other "interesting" architectural details ~:^)

Seriously.  What's your delivery address?  I'll make some calls.

>   Ralf
