X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C8006A.47A3B69A@onstor-exch02.onstor.net>; Wed, 26 Sep 2007 10:23:04 -0800
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
Subject: RE: CF accesses in PROM
Date: Wed, 26 Sep 2007 10:23:07 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E05C0237B@onstor-exch02.onstor.net>
In-Reply-To: <20070926103628.76b9e75a@ripper.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: CF accesses in PROM
Thread-Index: AcgAY8XQ+7oX2wpJRKGs3qUo3XpfdgAA7OzA
References: <BB375AF679D4A34E9CA8DFA650E2B04E05A25BE9@onstor-exch02.onstor.net><20070925165150.59e1b88c@ripper.onstor.net><20070926093941.5f93e1d5@ripper.onstor.net><BB375AF679D4A34E9CA8DFA650E2B04E05B46EA0@onstor-exch02.onstor.net> <20070926103628.76b9e75a@ripper.onstor.net>
From: "Brian Stark" <brian.stark@onstor.com>
To: "Andy Sharp" <andy.sharp@onstor.com>


More below...


> -----Original Message-----
> From: Andy Sharp=20
> Sent: Wednesday, September 26, 2007 10:36 AM
> To: Brian Stark
> Subject: Re: CF accesses in PROM
>=20
> On Wed, 26 Sep 2007 09:55:48 -0700 "Brian Stark"
> <brian.stark@onstor.com> wrote:
>=20
> > See below...
>=20
> See below ~:^)
>  =20
> >=20
> > > -----Original Message-----
> > > From: Andy Sharp
> > > Sent: Wednesday, September 26, 2007 9:40 AM
> > > To: Brian Stark
> > > Subject: Re: CF accesses in PROM
> > >=20
> > > On Tue, 25 Sep 2007 16:51:50 -0700 Andrew Sharp=20
> > > <andy.sharp@onstor.com>
> > > wrote:
> > >=20
> > > > You see?  You have to use the original email title if you
> > > want me to
> > > > remember it.  I only hash these things by title in my=20
> brain, there=20
> > > > isn't room to hash the contents as well.  That's the=20
> limits of a=20
> > > > walnut-sized brain.
> > > >=20
> > > > On Sun, 23 Sep 2007 11:15:55 -0700 "Brian Stark"
> > > > <brian.stark@onstor.com> wrote:
> > > >=20
> > > > > Just wanted to let you know that we now have CF accesses
> > > working in
> > > > > PROM. This confirms that the hardware path to and from
> > > flash is ok
> > > > > and also incorporates all the address changes we had=20
> to make to=20
> > > > > support the CF from a SiByte.
> > > > > =20
> > > > > We tried getting the CF to be configured as memory
> > > instead of i/o,
> > > > > but we hit some roadblocks with this.  For now, we are=20
> > > > > supporting the same scheme that we've used before on=20
> Bobcat when
> > > accessing CF,
> > > > > which is a mixture of i/o and memory accesses.  Here's
> > > the address
> > > > > map that we're using on PCI in PROM:
> > > > > =20
> > > > > =20
> > > > > Physical Address	 Size	 Function
> > > > > Notes 0000.0400	 256	 CF Controller I/O socket A
> > > > > Accessed at base of 0xdc000000; set in ExCA i/o 0
> > > >=20
> > > > uh, right off the bat we aren't communicating.  this thing
> > > is located
> > > > at phys 0x400???  i'm guessing that what you meant is=20
> some kind of=20
> > > > offset here?  this sounds like the register offset.
> > >=20
> > > OK, I get it now.  0xdc00.0000 is the phys and 400 is the offset=20
> > > from there.  Check.
> >=20
> > Yep, you got it.
> >=20
> >=20
> >=20
> > >=20
> > > > > 0000.0800	 256	 CF Controller I/O socket B
> > > > > Accessed at base of 0xdc000000; set in ExCA i/o 0
> > > >=20
> > > > ditto of course
> > >=20
> > > ditto
> > >=20
> > > > > 0001.0000	 64KB	 CF Controller I/O ExCA
> > > > > Accessed at base of 0xdc000000; uses base/index scheme
> > > >=20
> > > > double ditto.  this one i remember from boobcat days.
> > > >=20
> > > > > 0080.0000	 64KB	 CF Controller Memory
> > > > > Socket memory accesses (set in ExCA memory 0);=20
> accessed at base=20
> > > > > of 0xf8.0000.0000
> > > >=20
> > > > whoa nellie, 0xf8.0000.0000?  we have 40 bit physical
> > > addresses?  did
> > > > you mean 0xf800.0000?
>=20
> Christ I'm a moron, we have 44 bit physical addresses, doh!
>=20
> > > The code I'm looking at has mem_resource starting at physical=20
> > > 0x4000.0000, which I don't quite get, because it also=20
> defines these:
> > >=20
> > > #define A_PHYS_LDTPCI_IO_MATCH_BYTES_32 (0x0040000000)=20
> > > #define A_PHYS_LDTPCI_IO_MATCH_BITS_32  (0x0060000000)
> > >=20
> > > #define A_PHYS_LDT_SPECIAL_MATCH_BYTES  (0x00D8000000)
> > > #define A_PHYS_LDTPCI_IO_MATCH_BYTES    (0x00DC000000)
> > > #define A_PHYS_LDTPCI_CFG_MATCH_BYTES   (0x00DE000000)
> > > #define A_PHYS_LDT_SPECIAL_MATCH_BITS   (0x00F8000000)
> > > #define A_PHYS_LDTPCI_IO_MATCH_BITS     (0x00FC000000)
> > > #define A_PHYS_LDTPCI_CFG_MATCH_BITS    (0x00FE000000)
> > >=20
> > > #define A_PHYS_PCI_FULLACCESS_BYTES     (0xF000000000)
> > > #define A_PHYS_PCI_FULLACCESS_BITS      (0xF100000000)
> > > #define A_PHYS_RESERVED                 (0xF200000000)
> > > #define A_PHYS_RESERVED_SPECIAL_LDT     (0xFD00000000)
> > >=20
> > > but no f8.0000.0000 ... except that it's inside A_PHYS_RESERVED
> >=20
> >=20
> >=20
> > You may be able to get some of these other regions to work, but I
> > didn't want to mess with anything that also mapped to HT (aka LDT).
> > I think this requires some additional setup inside the part.  Plus,
> > since we're specifying 32-bit PCI addresses, I'm not sure the 32-bit
> > regions listed above (eg A_PHYS_LDTPCI_IO_MATCH_BYTES_32)=20
> can be used
> > with the map that we've defined in PROM.  For example, to get to the
> > BM-FPGA, would we use an address of 0x5f40.0000?  Does bit 30 get
> > stripped off before going out to PCI or does each PCI device have to
> > be mapped at 0x4000.0000 and above?  Too many questions that require
> > work to figure out the answers!
>=20
> I admit that at the moment I don't get this myself.  the _IO part of
> the macro name would suggest it's IO space, but the code that=20
> registers
> the pci controller clearly sets the MEM space resource to start at
> physical 0x40000000, (and the IO space at 0??) so I'm quite confused
> about that AND about what you said.  Crap.
>=20

I was at first confused by the IO and MEM nomenclature as well.  I've
since figured out that IO is just a generic term for transferring data
over PCI bus.  The MEM refers to the type of cycle, which dictates what
is put out during the command phase.

As for what I said, I'll draw a picture sometime about the accesses.  My
belief with the 32-bit spaces is that everything on the bus has to then
offset from 0x4000.0000, which reduces the amount of useable space on
the PCI bus since the region is only 512MB.





> Unfortunately it's locking up solid when trying to scan the PCI bus on
> this machine, so I can't see anything that it's doing or numbers it
> [would be] is getting for the bars and such.

=20



> > The f8.0000.0000 is dedicated to just the PCI bus, so that's why we
> > used that region in PROM.  Also, the f8 doesn't come out the 32-bit
> > PCI bus, so the entire 32-bit space can be used.
>=20
> You lost me with this one.  I need an education here.

Yep, no problem.  The basic idea here is that the 40-bit address allows
full use of the 32 bits for addressing on PCI.  For example, the BM-FPGA
is addressed through this region via 0xf8.1f40.0000 and the TXRX via
0xf8.1a00.0000.  Not sure how (or if) this would work by using the above
0x4000.0000 space.  It comes down to having 4GB of space versus 512MB.



>=20
> > > > > 0081.0000	 64KB	 CF Controller Socket
> > > > > A / ExCA base	 Socket A memory accesses; ExCA mapped at
> > > > > 0x800 offset; accessed at base of 0xf8.0000.0000
> > > >=20
> > > > ditto on the nellie
> > > >=20
> > > > > 0082.0000	 64KB	 CF Controller Socket B / ExCA
> > > > > base	 Socket B memory accesses; ExCA mapped at 0x800
> > > > > offset; accessed at base of 0xf8.0000.0000=09
> > > >=20
> > > > > 1a00.0000	 16MB	 TXRX Shared Memory	 TXRX
> > > > > accesses this locally at 0x9000.0000=09
> > > >=20
> > > > i'm with ya here.
> > > >=20
> > > > > 1b00.0000	 16MB	 FP Shared Memory	 FP
> > > > > accesses this locally at 0x9000.0000=09
> > > >=20
> > > > ok...
> > > >=20
> > > > > 1f40.0000	 64KB	 BM FPGA	  =09
> > > >=20
> > > > hokee-dokee
> > > >=20
> > > > > 6000.0000	 16MB	 SSC Shared Memory	 SSC
> > > > > accesses this locally at 0x8300.0000=09
> > > >=20
> > > > 0x8300.0000?  you may have pasted the wrong number in here:
> > > >=20
> > > > 0x8000.0000 physical is 2nd 256MB of local mem.
> > > > 0xffff.ffff.8000.0000 virtual is start of 1st 256MB of local mem
>=20
> OK, I see what you're saying.  But then I need to set this memory as
> reserved and it's quite inconvenient to set 16MB in the middle of the
> memory aside -- uh, why didn't you guys put it at the beginning or the
> end of the memory?  That's where I'm gonna need it to be. =20
> Preferably at
> the end, so at 0x8f00.0000.
>=20
> Actually, why can't we take an 8MB chunk out of the memory=20
> for the TXRX
> and FP respectively?  Why does it have to come out of MCPU memory?  I
> sure hope it's not for some obvious reason that I am failing to grok.

We have 16MB chunks in SSC, TXRX, and FP memory.  It doesn't necessarily
need to be 16MB, and it can be relocated anywhere we like in physical
space.  This shared memory is used for mgmt bus stuff.



>=20
> As for Linux re-mapping everything, the sibyte code actually expects
> the CFE to configure everything and doesn't try to mess with=20
> it. Ironic.

That's funny.  To be honest, I actually prefer this.=20



> Cheers,
>=20
> a
>=20
