AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20090619100306.61d6df16@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:mail.onstor.net
NSV:
SSH:
R:<patrick.haverty@onstor.com>,<larry.scheer@onstor.com>,<dl-qa@onstor.com>,<brian.stark@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@exch1.onstor.net/INBOX	0	102AB4F33EBBDB4C91915B145C8E9FB31370ABBF89@exch1.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Fri, 19 Jun 2009 10:03:12 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: Patrick Haverty <patrick.haverty@onstor.com>
Cc: Larry Scheer <larry.scheer@onstor.com>, dl-QA <dl-qa@onstor.com>, Brian
 Stark <brian.stark@onstor.com>
Subject: Re: Having trouble with getting Jumbo Frames to work in 1.1.9
Message-ID: <20090619100312.5e4a7a00@ripper.onstor.net>
In-Reply-To: <102AB4F33EBBDB4C91915B145C8E9FB31370ABBF89@exch1.onstor.net>
References: <102AB4F33EBBDB4C91915B145C8E9FB31370ABBF73@exch1.onstor.net>
	<102AB4F33EBBDB4C91915B145C8E9FB31370ABBF89@exch1.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

What is "GLDv2"?

The amd64 arch type should work on either processor, as the
intel is a fully compatible x86_64.  The version should be the only
thing what makes a difference.  I couldn't keep straight every
thing you were saying, but it looks like there are 3, maybe 4, versions
in play:

3.5.6.1
4.4.5.1
4.6.q.z
GLDv2 (??)

We seem to know that the 4.4 series doesn't work, the 3.5 series does,
and I couldn't figure out what you were saying about the 4.6 series.
It seems some of the interfaces are at 9000, and some aren't. BTW, I'm
gonna make a wild guess and say it's possible that the driver is
enforcing a restriction that if the interface is up, then you can't
change the MTU on it.  Just a guess.

On Fri, 19 Jun 2009 09:07:55 -0700 Patrick Haverty
<patrick.haverty@onstor.com> wrote:

> So the i86 version of the Solaris GLDv2 bnx driver does NOT seem to
> work.  Since there's some other weirdness (long boot times) going on
> with the 2110 since I started playing around with the Ethernet
> settings I'm going to do a fresh install (hopefully from the latest
> ISO) and then try the driver again.
> 
> nmc@panterals2110:/$ setup network interface
> Option ?  bnx1
> Option ?  static
> bnx1 IP address     : 10.11.1.181
> bnx1 netmask        : 255.255.0.0
> bnx1 mtu            : 9000
> Warning: changing mtu MAY require driver re-load! Network
> Interface(s) [bnx0] could be re-initialized. Change MTU from the
> current 1500 to 9000 ?  Yes Enabling bnx1 as 10.11.1.181/255.255.0.0
> mtu 9000 ... Failed. com.nexenta.nms.SystemCallError: failed to
> configure bnx1 with ip 10.11.1.181 netmask 255.255.0.0 mtu 9000
> broadcast + up: ifconfig: setifmtu: SIOCSLIFMTU: bnx1: Invalid
> argument
> 
> -----Original Message-----
> From: Patrick Haverty 
> Sent: Thursday, June 18, 2009 9:16 PM
> To: Patrick Haverty; Larry Scheer
> Cc: dl-QA; Brian Stark; Andy Sharp
> Subject: RE: Having trouble with getting Jumbo Frames to work in 1.1.9
> 
> nmc@Leopard-4:/$ setup network interface Option ?  bnx1 Option ?
> static bnx1 IP address     : 10.11.1.219
> bnx1 netmask        : 255.255.0.0
> bnx1 mtu            : 9000
> Warning: changing mtu MAY require driver re-load! Network
> Interface(s) [bnx0] could be re-initialized. Change MTU from the
> current  to 9000 ?  Yes Enabling bnx1 as 10.11.1.219/255.255.0.0 mtu
> 9000 ... OK. Name Server #1      : 10.3.0.7 Name Server #2      :
> 10.0.0.229 Name Server #3      : 10.1.1.4
> Gateway IP address  : 10.11.0.1
> 
> nmc@Leopard-4:/$ setup network interface Option ?  show ====
> Interfaces ==== lo0:
> flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232
> index 1 inet 127.0.0.1 netmask ff000000 bnx0:
> flags=1201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS,FIXEDMTU>
> mtu 1500 index 2 inet 10.11.1.216 netmask ffff0000 broadcast
> 10.11.255.255 ether 0:22:19:d2:9f:be bnx1:
> flags=1201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS,FIXEDMTU>
> mtu 9000 index 4 inet 10.11.1.219 netmask ffff0000 broadcast
> 10.11.255.255 ether 0:22:19:d2:9f:c0 bnx2:
> flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 9000 index
> 5 inet 0.0.0.0 netmask 0 ether 0:22:19:d2:89:3b
> bnx3: flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 9000
> index 3 inet 0.0.0.0 netmask 0
>         ether 0:22:19:d2:89:3d
> lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu
> 8252 index 1 inet6 ::1/128 
> 
> -----Original Message-----
> From: Patrick Haverty
> Sent: Thursday, June 18, 2009 9:05 PM
> To: Patrick Haverty; Larry Scheer
> Cc: dl-QA; Brian Stark; Andy Sharp
> Subject: RE: Having trouble with getting Jumbo Frames to work in 1.1.9
> 
> Look what happened when I tried the Solaris GLDv2 version of the bnx
> driver (and the bnx.conf edit of course).  Now to see if similar
> magic happens with the 2110.
> 
> root@Leopard-4:/volumes# ifconfig -a
> lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu
> 8232 index 1 inet 127.0.0.1 netmask ff000000
> bnx0:
> flags=1201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS,FIXEDMTU>
> mtu 1500 index 2 inet 10.11.1.216 netmask ffff0000 broadcast
> 10.11.255.255 ether 0:22:19:d2:9f:be bnx1:
> flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 9000 index
> 4 inet 0.0.0.0 netmask 0 ether 0:22:19:d2:9f:c0
> bnx2: flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 9000
> index 5 inet 0.0.0.0 netmask 0
>         ether 0:22:19:d2:89:3b
> bnx3: flags=201000842<BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 9000
> index 3 inet 0.0.0.0 netmask 0
>         ether 0:22:19:d2:89:3d
> lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu
> 8252 index 1 inet6 ::1/128
> 
>  
> 
> -----Original Message-----
> From: Patrick Haverty
> Sent: Thursday, June 18, 2009 8:27 PM
> To: Larry Scheer
> Cc: dl-QA; Brian Stark; Andy Sharp
> Subject: RE: Having trouble with getting Jumbo Frames to work in 1.1.9
> 
> It looks like he may have "built/compiled/made" something for the
> amd64, but the other driver is the one we'd use with the Intel
> processors in the 2950 systems.  I should of steered him to the 2110
> and then maybe he would have "built/compiled/made" that one.  It
> looks like it didn't change.
> 
> root@Leopard-4:/kernel/drv# ls -l bn*	(Note: this system does
> have AMD processors) -rwxr-xr-x 1 root sys 498720 Dec  7  2008 bnx
> -rw-r--r-- 1 root sys  17273 Dec  7  2008 bnx.conf
> 
> root@Leopard-2:/kernel/drv# ls -l bn*
> -rwxr-xr-x 1 root sys 498720 Dec  7  2008 bnx
> -rw-r--r-- 1 root sys  17272 Mar 20 15:59 bnx.conf
> 
> 
> root@panterals2110:/kernel/drv# ls -l bn*
> -rwxr-xr-x 1 root root 488884 Jun 18 15:49 bnx   <----- This is a
> Solaris 10 driver I was trying -rw-r--r-- 1 root sys   17290 Jun 18
> 17:50 bnx.conf -rwxr-xr-x 1 root sys  498720 Dec  7  2008
> bnx.drv.old  <----- This is the current i86 driver
> 
> 
> root@Leopard-4:/kernel/drv# strings amd64/bnx | fgrep Broad
> BFD: amd64/bnx: warning: sh_link not set for section `.eh_frame'
> Broadcom NetXtreme II Gigabit Ethernet Driver v4.4.5.1 Broadcom NXII
> GbE v4.4.5.1
> 
> 
> root@Leopard-2:/kernel/drv# strings amd64/bnx | fgrep Broad
> BFD: amd64/bnx: warning: sh_link not set for section `.eh_frame'
> Broadcom NetXtreme II Gigabit Ethernet Driver v4.4.5.1 Broadcom NXII
> GbE v4.4.5.1
> 
> root@panterals2110:/kernel/drv# strings amd64/bnx | fgrep Broad
> Broadcom NetXtreme II Gigabit Ethernet Driver v4.6.2 Broadcom NXII
> GbE v4.6.2
> 
> 
> -----Original Message-----
> From: Larry Scheer
> Sent: Thursday, June 18, 2009 6:20 PM
> To: Larry Scheer; Patrick Haverty
> Cc: dl-QA; Brian Stark; Andy Sharp
> Subject: RE: Having trouble with getting Jumbo Frames to work in 1.1.9
> 
> Oh, one more thing I checked the version of the drivers using strings
> | grep Broadcom and this is what I see on leopard-1:
> 
> strings amd64/bnx | fgrep Broad
> Broadcom NetXtreme II Gigabit Ethernet Driver v3.5.6.1 Broadcom NXII
> GbE v3.5.6.1 Broadcom NetXtreme II Ethernet Driver ver #v3.5.6.1
> 
> strings bnx | fgrep Broad   
> Broadcom NetXtreme II Gigabit Ethernet Driver v4.4.5.1 Broadcom NXII
> GbE v4.4.5.1
> 
> I also tar'd up Dmitry's changes and copied the tar file to the build
> system.
> 
> JFYI,
> 
> Larry
> 
> -----Original Message-----
> From: Larry Scheer
> Sent: Thursday, June 18, 2009 6:10 PM
> To: Patrick Haverty
> Cc: dl-QA; Brian Stark; Andy Sharp
> Subject: RE: Having trouble with getting Jumbo Frames to work in 1.1.9
> 
> Pat, I suspect you could pull this version from Broadcom. He placed a
> copy on 10.11.1.209 so we have it. 
> 
> I see in /kernel/drv the following changes:
> 
> -rwxr-xr-x 1 root sys 498720 Dec  7  2008 bnx
> -rw-r--r-- 1 root sys  17285 Jun 18 17:27 bnx.conf
> 
> The driver we use is in the amd64 directory:
> 
> -rwxr-xr-x 1 root root  566416 Jun 18 17:21 bnx.copy
> -rwxr-xr-x 1 root sys   328872 Jun 18 17:22 bnx
> 
> We can verify these against other binaries using their md5sum:
> 
> 365da2195890f3461e0cea43e76c25ce  bnx
> 46c96e5384ad1609c1c5cf39bf699266  amd64/bnx
> 767f094c97b5f908e9e207d6ac444810  amd64/bnx.copy
> 
> -----Original Message-----
> From: Patrick Haverty
> Sent: Thursday, June 18, 2009 5:49 PM
> To: Larry Scheer; dl-QA
> Cc: Brian Stark
> Subject: RE: Having trouble with getting Jumbo Frames to work in 1.1.9
> 
> And of course he did say where we could get our grubbies on that
> binary to play with.  Then again, I don't know how to incorporate
> it.  Being an older binary, what would it be missing that the newer
> one has?
> 
> -----Original Message-----
> From: Larry Scheer
> Sent: Thursday, June 18, 2009 5:43 PM
> To: dl-QA; Patrick Haverty
> Subject: FW: Having trouble with getting Jumbo Frames to work in 1.1.9
> 
> 
> 
> -----Original Message-----
> From: Dmitry Yusupov [mailto:dmitry@nexenta.com]
> Sent: Thursday, June 18, 2009 5:33 PM
> To: Larry Scheer
> Cc: 'Alex Aizman'; Brian Stark
> Subject: Re: Having trouble with getting Jumbo Frames to work in 1.1.9
> 
> Larry,
> 
> I logged in and tried various things.. However, the bnx driver we are
> using currently is missing MTU support at all This is what Sun eng is
> saying:
> 
> """The bnx driver does not support jumbo frame for the time being.
> There is plan to implement the feature in the near future.
> 
> The man page is written according to the conf file from BroadCom.
> There is discrepancy with the real implementation. :( Sorry for the
> confusion."""
> 
> However, he didn't say that it was working in previous releases of
> the driver (namely in b90) driver v3.5.6.1. So, I decided to try that
> binary and it worked. I currently configured it manually by
> editing /kernel/drv/bnx.conf - added "Jumbo=9000;".
> 
> I also noticed that NMS version you are running not including our
> fixes. Please make sure it has fixes for NetworkInterface.pm. You
> need to rebuild it if you want to use NMC.
> 
> Please run some test on it, make sure it is stable.
> 
> Thanks!
> 
> Larry Scheer wrote:
> > Dmitry,
> >     We are unaware that we can have a choice of drivers for the
> > chipset that is on the motherboard. We are assuming the bnx drivers
> > that are automatically installed on the system are all that will
> > work with the Broadcom chipset that we have with our hardware
> > configuration.
> > 
> > You can have access to our systems via ssh.
> > 
> > Your account is still active on 66.201.51.69.
> > Just ssh dmitry@66.201.51.69
> > Pass is Xped!te
> > 
> >>From there you can ssh into 10.11.1.209 as admin.
> > Pass is onstor
> > 
> > Thanks,
> > 
> > Larry
> > 
> > -----Original Message-----
> > From: Dmitry Yusupov [mailto:dmitry@nexenta.com]
> > Sent: Thursday, June 18, 2009 1:01 PM
> > To: Larry Scheer
> > Cc: 'Alex Aizman'; Brian Stark
> > Subject: Re: Having trouble with getting Jumbo Frames to work in
> > 1.1.9
> > 
> > Hi Larry,
> > 
> > there was a .conf parsing error which were causing wrong .conf 
> > generated. Take a look on the log for NZA/NetworkInterface.pm file 
> > changes to find out more details on what exactly changed.
> > 
> > bnx driver known to be working with Jumbo (from forums, not
> > verified by us). However, we highly recommend you to use Intel or
> > Broadcom (bge ones) chipsets and keep bnx for mgmt interfaces only.
> > Somehow, opensolaris community doesn't like bnx and all the efforts
> > done towards intel and bge drivers.
> > 
> > If you still insists on bnx driver as a primary networking, we
> > could help you to track it down. Could you provide SSH access for
> > us please? (or remind/verify us the old path)
> > 
> > Thanks!
> > 
> > Larry Scheer wrote:
> >> Alex, Dmitry,
> >>     According to the defect reports jumbo frames is supposed to be 
> >> fixed in release 1.1.9 however we have not been able to get it to 
> >> work with our hardware configuration. We even tried OpenSolaris
> >> 2009.6 and it didn't work with our hardware configuration. We
> >> suspect it is a problem with the bnx drivers we use.
> >>  
> >> Reviewing OnStorDev defect #314 I see the following comment from
> >> 5/2/2009: 
> >> Comment:
> >>  
> >> Fixed in 1.1.8
> >>  
> >> Note that the testing was done with Intel's e1000g.
> >>  
> >> I have some questions that would help us with our testing and 
> >> configurations.
> >>
> >>    1. What NIC(s) were used in your testing?
> >>    2. What driver and driver versions did you use to verify correct
> >>       operation of jumbo frames?
> >>    3. What change list incorporates the fix? (I would like to
> >> verify the fixes are in my build.) If it is a Solaris package that
> >> has the fix, what is the package's version?
> >>
> >>  
> >> Thanks in advance for taking the time to answer these questions it 
> >> will help us greatly with our testing and system configurations.
> >>  
> >> Larry
> >>  
> > 
> > 
