AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20090619132750.2d2eadd9@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:mail.onstor.net
NSV:
SSH:
R:<patrick.haverty@onstor.com>,<brian.stark@onstor.com>,<larry.scheer@onstor.com>,<dl-qa@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@exch1.onstor.net/INBOX	0	102AB4F33EBBDB4C91915B145C8E9FB31370ABBFD2@exch1.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Fri, 19 Jun 2009 13:28:17 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: Patrick Haverty <patrick.haverty@onstor.com>
Cc: Brian Stark <brian.stark@onstor.com>, Larry Scheer
 <larry.scheer@onstor.com>, dl-QA <dl-qa@onstor.com>
Subject: Re: Having trouble with getting Jumbo Frames to work in 1.1.9
Message-ID: <20090619132817.2a4e0ef4@ripper.onstor.net>
In-Reply-To: <102AB4F33EBBDB4C91915B145C8E9FB31370ABBFD2@exch1.onstor.net>
References: <102AB4F33EBBDB4C91915B145C8E9FB31370ABBFD2@exch1.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

On Fri, 19 Jun 2009 11:47:21 -0700 Patrick Haverty
<patrick.haverty@onstor.com> wrote:

> Okay, it works if you actually put the right driver in the right
> place.
> 
> nmc@panterals2110:/$ setup network interface
> Option ?  bnx1
> Option ?  static
> bnx1 IP address     : 10.11.1.181
> bnx1 netmask        : 255.255.0.0
> bnx1 mtu            : 9000
> Warning: changing mtu MAY require driver re-load! Network
> Interface(s) [bnx0] could be re-initialized. Change MTU from the
> current  to 9000 ?  Yes Enabling bnx1 as 10.11.1.181/255.255.0.0 mtu
> 9000 ... OK. Name Server #1      : 10.3.0.7
> Name Server #2      : 10.0.0.229
> Name Server #3      : 10.1.1.4
> Gateway IP address  : 10.11.0.1
> 
> nmc@panterals2110:/$ !bash
> Warning: using low-level UNIX commands is not recommended! Execute?
> Yes root@panterals2110:/volumes# ifconfig -a
> lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu
> 8232 index 1 inet 127.0.0.1 netmask ff000000
> bnx0:
> flags=1201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS,FIXEDMTU>
> mtu 1500 index 2 inet 10.11.1.180 netmask ffff0000 broadcast
> 10.11.255.255 ether 0:22:19:2b:60:7b bnx1:
> flags=1201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS,FIXEDMTU>
> mtu 9000 index 4 inet 10.11.1.181 netmask ffff0000 broadcast
> 10.11.255.255 ether 0:22:19:2b:60:7d lo0:
> flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252
> index 1 inet6 ::1/128
> 
> Now, how do we test that having mtu 9000 actually makes a difference
> or otherwise works?

Well, you're on your own as to whether it makes any difference ~:^)  If
it doesn't work, you won't get any traffic through the interface.  It
will be like that cable isn't plugged in.

Personally, I'm thinking typical leopard customers won't even know of
such a thing as jumbo frames.  But what do I know.  Burzon might want
it straight away, however.

> -----Original Message-----
> From: Patrick Haverty
> Sent: Friday, June 19, 2009 11:30 AM
> To: Brian Stark; Andy Sharp
> Cc: Larry Scheer; dl-QA
> Subject: RE: Having trouble with getting Jumbo Frames to work in 1.1.9
> 
> I can look at the file size and tell they are different.
> 
> -----Original Message-----
> From: Brian Stark
> Sent: Friday, June 19, 2009 11:22 AM
> To: Patrick Haverty; Andy Sharp
> Cc: Larry Scheer; dl-QA
> Subject: RE: Having trouble with getting Jumbo Frames to work in 1.1.9
> 
> No, I'm not positive about the bmc driver, but you can diff them to
> see if they're the same...
> 
> 
> -----Original Message-----
> From: Patrick Haverty
> Sent: Friday, June 19, 2009 11:18 AM
> To: Brian Stark; Andy Sharp
> Cc: Larry Scheer; dl-QA
> Subject: RE: Having trouble with getting Jumbo Frames to work in 1.1.9
> 
> The drivers in the tar file from Broadcom are different in the two
> locations you mentioned, but I can copy the one from the /amd64
> directory and try it in /kernel/drv.  I don't think they are the same
> driver in both locations although they may have the same name.  Are
> you sure we're placing the same exact bmc driver in both locations?
> 
> I don't have a clue what GLDv2 is other than they separate it out
> from GLDv3 versions on the Broadcom website.
> 
> I did try unplumbing the ports before applying the mtu change.  I
> think the driver IS enforcing a restriction, but it's enforcing an
> mtu setting of 1500.
> 
> -----Original Message-----
> From: Brian Stark
> Sent: Friday, June 19, 2009 10:34 AM
> To: Andy Sharp; Patrick Haverty
> Cc: Larry Scheer; dl-QA
> Subject: RE: Having trouble with getting Jumbo Frames to work in 1.1.9
> 
> Andy makes a good point about the driver.  In the case of the bmc
> driver for hardware notifications, the following directories get the
> same driver:
> 
> /kernel/drv/amd64
> /kernel/drv
> 
> This suggests that the driver is independent of the processor
> architecture.
> 
> Have we tried using the bnx driver in the amd64 directory on the
> 2950 / 2110 that uses Intel processors?
> 
> 
> 
> -----Original Message-----
> From: Andy Sharp
> Sent: Friday, June 19, 2009 10:03 AM
> To: Patrick Haverty
> Cc: Larry Scheer; dl-QA; Brian Stark
> Subject: Re: Having trouble with getting Jumbo Frames to work in 1.1.9
> 
> What is "GLDv2"?
> 
> The amd64 arch type should work on either processor, as the intel is
> a fully compatible x86_64.  The version should be the only thing what
> makes a difference.  I couldn't keep straight every thing you were
> saying, but it looks like there are 3, maybe 4, versions in play:
> 
> 3.5.6.1
> 4.4.5.1
> 4.6.q.z
> GLDv2 (??)
> 
> We seem to know that the 4.4 series doesn't work, the 3.5 series
> does, and I couldn't figure out what you were saying about the 4.6
> series. It seems some of the interfaces are at 9000, and some aren't.
> BTW, I'm gonna make a wild guess and say it's possible that the
> driver is enforcing a restriction that if the interface is up, then
> you can't change the MTU on it.  Just a guess.
> 
> On Fri, 19 Jun 2009 09:07:55 -0700 Patrick Haverty
> <patrick.haverty@onstor.com> wrote:
> 
> > So the i86 version of the Solaris GLDv2 bnx driver does NOT seem to
> > work.  Since there's some other weirdness (long boot times) going on
> > with the 2110 since I started playing around with the Ethernet
> > settings I'm going to do a fresh install (hopefully from the latest
> > ISO) and then try the driver again.
> >
> > nmc@panterals2110:/$ setup network interface Option ?  bnx1 Option ?
> > static
> > bnx1 IP address     : 10.11.1.181
> > bnx1 netmask        : 255.255.0.0
> > bnx1 mtu            : 9000
> > Warning: changing mtu MAY require driver re-load! Network
> > Interface(s) [bnx0] could be re-initialized. Change MTU from the
> > current 1500 to 9000 ?  Yes Enabling bnx1 as 10.11.1.181/255.255.0.0
> > mtu 9000 ... Failed. com.nexenta.nms.SystemCallError: failed to
> > configure bnx1 with ip 10.11.1.181 netmask 255.255.0.0 mtu 9000
> > broadcast + up: ifconfig: setifmtu: SIOCSLIFMTU: bnx1: Invalid
> > argument
> >
> > -----Original Message-----
> > From: Patrick Haverty
> > Sent: Thursday, June 18, 2009 9:16 PM
> > To: Patrick Haverty; Larry Scheer
> > Cc: dl-QA; Brian Stark; Andy Sharp
> > Subject: RE: Having trouble with getting Jumbo Frames to work in
> > 1.1.9
> >
> > nmc@Leopard-4:/$ setup network interface Option ?  bnx1 Option ?
> > static bnx1 IP address     : 10.11.1.219
> > bnx1 netmask        : 255.255.0.0
> > bnx1 mtu            : 9000
> > Warning: changing mtu MAY require driver re-load! Network
> > Interface(s) [bnx0] could be re-initialized. Change MTU from the
> > current  to 9000 ?  Yes Enabling bnx1 as 10.11.1.219/255.255.0.0 mtu
> > 9000 ... OK. Name Server #1      : 10.3.0.7 Name Server #2      :
> > 10.0.0.229 Name Server #3      : 10.1.1.4
> > Gateway IP address  : 10.11.0.1
> >
> > nmc@Leopard-4:/$ setup network interface Option ?  show ====
> > Interfaces ==== lo0:
> > flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu
> > 8232 index 1 inet 127.0.0.1 netmask ff000000 bnx0:
> > flags=1201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS,FIXEDMTU>
> > mtu 1500 index 2 inet 10.11.1.216 netmask ffff0000 broadcast
> > 10.11.255.255 ether 0:22:19:d2:9f:be bnx1:
> > flags=1201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS,FIXEDMTU>
> > mtu 9000 index 4 inet 10.11.1.219 netmask ffff0000 broadcast
> > 10.11.255.255 ether 0:22:19:d2:9f:c0 bnx2:
> > flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 9000 index
> > 5 inet 0.0.0.0 netmask 0 ether 0:22:19:d2:89:3b
> > bnx3: flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 9000
> > index 3 inet 0.0.0.0 netmask 0
> >         ether 0:22:19:d2:89:3d
> > lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL>
> > mtu 8252 index 1 inet6 ::1/128
> >
> > -----Original Message-----
> > From: Patrick Haverty
> > Sent: Thursday, June 18, 2009 9:05 PM
> > To: Patrick Haverty; Larry Scheer
> > Cc: dl-QA; Brian Stark; Andy Sharp
> > Subject: RE: Having trouble with getting Jumbo Frames to work in
> > 1.1.9
> >
> > Look what happened when I tried the Solaris GLDv2 version of the bnx
> > driver (and the bnx.conf edit of course).  Now to see if similar
> > magic happens with the 2110.
> >
> > root@Leopard-4:/volumes# ifconfig -a
> > lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL>
> > mtu 8232 index 1 inet 127.0.0.1 netmask ff000000
> > bnx0:
> > flags=1201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS,FIXEDMTU>
> > mtu 1500 index 2 inet 10.11.1.216 netmask ffff0000 broadcast
> > 10.11.255.255 ether 0:22:19:d2:9f:be bnx1:
> > flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 9000 index
> > 4 inet 0.0.0.0 netmask 0 ether 0:22:19:d2:9f:c0
> > bnx2: flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 9000
> > index 5 inet 0.0.0.0 netmask 0
> >         ether 0:22:19:d2:89:3b
> > bnx3: flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 9000
> > index 3 inet 0.0.0.0 netmask 0
> >         ether 0:22:19:d2:89:3d
> > lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL>
> > mtu 8252 index 1 inet6 ::1/128
> >
> >
> >
> > -----Original Message-----
> > From: Patrick Haverty
> > Sent: Thursday, June 18, 2009 8:27 PM
> > To: Larry Scheer
> > Cc: dl-QA; Brian Stark; Andy Sharp
> > Subject: RE: Having trouble with getting Jumbo Frames to work in
> > 1.1.9
> >
> > It looks like he may have "built/compiled/made" something for the
> > amd64, but the other driver is the one we'd use with the Intel
> > processors in the 2950 systems.  I should of steered him to the 2110
> > and then maybe he would have "built/compiled/made" that one.  It
> > looks like it didn't change.
> >
> > root@Leopard-4:/kernel/drv# ls -l bn* (Note: this system does
> > have AMD processors) -rwxr-xr-x 1 root sys 498720 Dec  7  2008 bnx
> > -rw-r--r-- 1 root sys  17273 Dec  7  2008 bnx.conf
> >
> > root@Leopard-2:/kernel/drv# ls -l bn*
> > -rwxr-xr-x 1 root sys 498720 Dec  7  2008 bnx
> > -rw-r--r-- 1 root sys  17272 Mar 20 15:59 bnx.conf
> >
> >
> > root@panterals2110:/kernel/drv# ls -l bn*
> > -rwxr-xr-x 1 root root 488884 Jun 18 15:49 bnx   <----- This is a
> > Solaris 10 driver I was trying -rw-r--r-- 1 root sys   17290 Jun 18
> > 17:50 bnx.conf -rwxr-xr-x 1 root sys  498720 Dec  7  2008
> > bnx.drv.old <----- This is the current i86 driver
> >
> >
> > root@Leopard-4:/kernel/drv# strings amd64/bnx | fgrep Broad
> > BFD: amd64/bnx: warning: sh_link not set for section `.eh_frame'
> > Broadcom NetXtreme II Gigabit Ethernet Driver v4.4.5.1 Broadcom NXII
> > GbE v4.4.5.1
> >
> >
> > root@Leopard-2:/kernel/drv# strings amd64/bnx | fgrep Broad
> > BFD: amd64/bnx: warning: sh_link not set for section `.eh_frame'
> > Broadcom NetXtreme II Gigabit Ethernet Driver v4.4.5.1 Broadcom NXII
> > GbE v4.4.5.1
> >
> > root@panterals2110:/kernel/drv# strings amd64/bnx | fgrep Broad
> > Broadcom NetXtreme II Gigabit Ethernet Driver v4.6.2 Broadcom NXII
> > GbE v4.6.2
> >
> >
> > -----Original Message-----
> > From: Larry Scheer
> > Sent: Thursday, June 18, 2009 6:20 PM
> > To: Larry Scheer; Patrick Haverty
> > Cc: dl-QA; Brian Stark; Andy Sharp
> > Subject: RE: Having trouble with getting Jumbo Frames to work in
> > 1.1.9
> >
> > Oh, one more thing I checked the version of the drivers using
> > strings | grep Broadcom and this is what I see on leopard-1:
> >
> > strings amd64/bnx | fgrep Broad
> > Broadcom NetXtreme II Gigabit Ethernet Driver v3.5.6.1 Broadcom NXII
> > GbE v3.5.6.1 Broadcom NetXtreme II Ethernet Driver ver #v3.5.6.1
> >
> > strings bnx | fgrep Broad
> > Broadcom NetXtreme II Gigabit Ethernet Driver v4.4.5.1 Broadcom NXII
> > GbE v4.4.5.1
> >
> > I also tar'd up Dmitry's changes and copied the tar file to the
> > build system.
> >
> > JFYI,
> >
> > Larry
> >
> > -----Original Message-----
> > From: Larry Scheer
> > Sent: Thursday, June 18, 2009 6:10 PM
> > To: Patrick Haverty
> > Cc: dl-QA; Brian Stark; Andy Sharp
> > Subject: RE: Having trouble with getting Jumbo Frames to work in
> > 1.1.9
> >
> > Pat, I suspect you could pull this version from Broadcom. He placed
> > a copy on 10.11.1.209 so we have it.
> >
> > I see in /kernel/drv the following changes:
> >
> > -rwxr-xr-x 1 root sys 498720 Dec  7  2008 bnx
> > -rw-r--r-- 1 root sys  17285 Jun 18 17:27 bnx.conf
> >
> > The driver we use is in the amd64 directory:
> >
> > -rwxr-xr-x 1 root root  566416 Jun 18 17:21 bnx.copy
> > -rwxr-xr-x 1 root sys   328872 Jun 18 17:22 bnx
> >
> > We can verify these against other binaries using their md5sum:
> >
> > 365da2195890f3461e0cea43e76c25ce  bnx
> > 46c96e5384ad1609c1c5cf39bf699266  amd64/bnx
> > 767f094c97b5f908e9e207d6ac444810  amd64/bnx.copy
> >
> > -----Original Message-----
> > From: Patrick Haverty
> > Sent: Thursday, June 18, 2009 5:49 PM
> > To: Larry Scheer; dl-QA
> > Cc: Brian Stark
> > Subject: RE: Having trouble with getting Jumbo Frames to work in
> > 1.1.9
> >
> > And of course he did say where we could get our grubbies on that
> > binary to play with.  Then again, I don't know how to incorporate
> > it. Being an older binary, what would it be missing that the newer
> > one has?
> >
> > -----Original Message-----
> > From: Larry Scheer
> > Sent: Thursday, June 18, 2009 5:43 PM
> > To: dl-QA; Patrick Haverty
> > Subject: FW: Having trouble with getting Jumbo Frames to work in
> > 1.1.9
> >
> >
> >
> > -----Original Message-----
> > From: Dmitry Yusupov [mailto:dmitry@nexenta.com]
> > Sent: Thursday, June 18, 2009 5:33 PM
> > To: Larry Scheer
> > Cc: 'Alex Aizman'; Brian Stark
> > Subject: Re: Having trouble with getting Jumbo Frames to work in
> > 1.1.9
> >
> > Larry,
> >
> > I logged in and tried various things.. However, the bnx driver we
> > are using currently is missing MTU support at all This is what Sun
> > eng is saying:
> >
> > """The bnx driver does not support jumbo frame for the time being.
> > There is plan to implement the feature in the near future.
> >
> > The man page is written according to the conf file from BroadCom.
> > There is discrepancy with the real implementation. :( Sorry for the
> > confusion."""
> >
> > However, he didn't say that it was working in previous releases of
> > the driver (namely in b90) driver v3.5.6.1. So, I decided to try
> > that binary and it worked. I currently configured it manually by
> > editing /kernel/drv/bnx.conf - added "Jumbo=9000;".
> >
> > I also noticed that NMS version you are running not including our
> > fixes. Please make sure it has fixes for NetworkInterface.pm. You
> > need to rebuild it if you want to use NMC.
> >
> > Please run some test on it, make sure it is stable.
> >
> > Thanks!
> >
> > Larry Scheer wrote:
> > > Dmitry,
> > >     We are unaware that we can have a choice of drivers for the
> > > chipset that is on the motherboard. We are assuming the bnx
> > > drivers that are automatically installed on the system are all
> > > that will work with the Broadcom chipset that we have with our
> > > hardware configuration.
> > >
> > > You can have access to our systems via ssh.
> > >
> > > Your account is still active on 66.201.51.69.
> > > Just ssh dmitry@66.201.51.69
> > > Pass is Xped!te
> > >
> > >>From there you can ssh into 10.11.1.209 as admin.
> > > Pass is onstor
> > >
> > > Thanks,
> > >
> > > Larry
> > >
> > > -----Original Message-----
> > > From: Dmitry Yusupov [mailto:dmitry@nexenta.com]
> > > Sent: Thursday, June 18, 2009 1:01 PM
> > > To: Larry Scheer
> > > Cc: 'Alex Aizman'; Brian Stark
> > > Subject: Re: Having trouble with getting Jumbo Frames to work in
> > > 1.1.9
> > >
> > > Hi Larry,
> > >
> > > there was a .conf parsing error which were causing wrong .conf
> > > generated. Take a look on the log for NZA/NetworkInterface.pm file
> > > changes to find out more details on what exactly changed.
> > >
> > > bnx driver known to be working with Jumbo (from forums, not
> > > verified by us). However, we highly recommend you to use Intel or
> > > Broadcom (bge ones) chipsets and keep bnx for mgmt interfaces
> > > only. Somehow, opensolaris community doesn't like bnx and all the
> > > efforts done towards intel and bge drivers.
> > >
> > > If you still insists on bnx driver as a primary networking, we
> > > could help you to track it down. Could you provide SSH access for
> > > us please? (or remind/verify us the old path)
> > >
> > > Thanks!
> > >
> > > Larry Scheer wrote:
> > >> Alex, Dmitry,
> > >>     According to the defect reports jumbo frames is supposed to
> > >> be fixed in release 1.1.9 however we have not been able to get
> > >> it to work with our hardware configuration. We even tried
> > >> OpenSolaris 2009.6 and it didn't work with our hardware
> > >> configuration. We suspect it is a problem with the bnx drivers
> > >> we use.
> > >>
> > >> Reviewing OnStorDev defect #314 I see the following comment from
> > >> 5/2/2009:
> > >> Comment:
> > >>
> > >> Fixed in 1.1.8
> > >>
> > >> Note that the testing was done with Intel's e1000g.
> > >>
> > >> I have some questions that would help us with our testing and
> > >> configurations.
> > >>
> > >>    1. What NIC(s) were used in your testing?
> > >>    2. What driver and driver versions did you use to verify
> > >> correct operation of jumbo frames?
> > >>    3. What change list incorporates the fix? (I would like to
> > >> verify the fixes are in my build.) If it is a Solaris package
> > >> that has the fix, what is the package's version?
> > >>
> > >>
> > >> Thanks in advance for taking the time to answer these questions
> > >> it will help us greatly with our testing and system
> > >> configurations.
> > >>
> > >> Larry
> > >>
> > >
> > >
