X-Sylpheed-Account-Id:1
S:andy.sharp@onstor.com
SCF:#mh/Mailbox/sent
X-Sylpheed-Sign:0
X-Sylpheed-Encrypt:0
X-Sylpheed-Privacy-System:
X-Sylpheed-End-Special-Headers: 1
Date: Thu, 23 Apr 2009 12:55:02 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: Brian Stark <brian.stark@onstor.com>
Cc: Patrick Haverty <patrick.haverty@onstor.com>
Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
Message-ID: <20090423125502.3158ed6c@ripper.onstor.net>
References: <20090415163709.4fdba5be@ripper.onstor.net>
 <102AB4F33EBBDB4C91915B145C8E9FB31284F9B544@exch1.onstor.net>
 <6041AEBD8A3E264785ABDA2F60B52DB4FEE6E024@exch1.onstor.net>
 <102AB4F33EBBDB4C91915B145C8E9FB31284F9B5AA@exch1.onstor.net>
 <20090416142630.0927386a@ripper.onstor.net>
 <102AB4F33EBBDB4C91915B145C8E9FB31284F9B5AE@exch1.onstor.net>
 <20090416144613.1b7e676b@ripper.onstor.net>
 <102AB4F33EBBDB4C91915B145C8E9FB31284F9B5C3@exch1.onstor.net>
Organization: Onstor
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

OK.  Programs are re-installed.  I'll put a copy of them in

/n/software/slowaris_x86/


On Thu, 16 Apr 2009 15:21:57 -0700 Brian Stark <brian.stark@onstor.com>
wrote:

> ssh root@10.11.1.216
> 
> Password is 'onstor'
> 
> 
> 
> 
> -----Original Message-----
> From: Andy Sharp 
> Sent: Thursday, April 16, 2009 2:46 PM
> To: Brian Stark
> Cc: Patrick Haverty; Rendell Fong; John Keiffer
> Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> 
> On Thu, 16 Apr 2009 14:31:30 -0700 Brian Stark <brian.stark@onstor.com>
> wrote:
> 
> > Agree with what you outlined with SNMP and email.
> > 
> > On the last part, we still don't know if your stuff worked.  When we tried the OpenSolaris install on its own, we got the same failure as what we saw in Nexenta with your stuff.  I then found a workaround on the Sun site that then made the ipmitool stuff work in OpenSolaris.  
> > 
> > We are now going back to the Nexenta install and will try your stuff again along with the workaround.  Pat will need your help in getting your stuff back on the Leopard once the Nexenta install is done.
> > 
> > 
> 
> HOK.  What was this mysterious "work around?"  Are we talking a
> recompile type thing or ?
> 
> 
> 
> > 
> > -----Original Message-----
> > From: Andy Sharp 
> > Sent: Thursday, April 16, 2009 2:27 PM
> > To: Brian Stark
> > Cc: Patrick Haverty; Rendell Fong; John Keiffer
> > Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> > 
> > On Thu, 16 Apr 2009 14:01:32 -0700 Brian Stark <brian.stark@onstor.com>
> > wrote:
> > 
> > > Good news, we got ipmitool working today and can now see hardware events on the server.  The events are logged to /var/adm/messages.  This is the same place that the JBOD events are going.
> > > 
> > > This means that we have a single place where all hardware events are being sent.  I've asked Nexenta if there's a way to either generate an email or SNMP alert based on the entries in the messages log.
> > > 
> > > If not, then it's seem reasonable to come up with a script that combs through that log looking for various events and then reporting them to the user.  The whitepaper I sent out yesterday shows how this can be done via email or SNMP.  While not perfect, it's a solution that we could use as part of the first release.
> > > 
> > 
> > This approach has two (one each) prerequisites that would still have to
> > be addressed.  I'm not saying they're un-solvable or even terribly
> > difficult, just noting them.  SNMP will need the snmptrap program
> > which, like ipmitool, currently is not part of the build.  Email will
> > need some way to get the email address from the nexenta configuration,
> > and so way to handle it when the user changes that email address.
> > 
> > > We did all of this work on top of OpenSolaris since we couldn't get the stuff Andy did yesterday on top of Nexenta to work.  It turns out that the same problem existed with OpenSolaris until we did another workaround.  We're going to try this again on top of a Nexenta install.
> > > 
> > 
> > I didn't follow that.  You mean the ipmitool I built didn't work, and
> > you had to get it from somewhere else?
> > 
> > > 
> > > 
> > > -----Original Message-----
> > > From: Brian Stark 
> > > Sent: Wednesday, April 15, 2009 9:38 PM
> > > To: Patrick Haverty; Andy Sharp
> > > Cc: Rendell Fong; John Keiffer
> > > Subject: RE: Dell Openmanage IT Assistant and SNMP Messages
> > > 
> > > Pat and I are still pursuing ipmitool.  It turns out it's not as easy as we thought because OpenSolaris does not contain the necessary drivers for the BMC.  I found some posts on Sun forums that indicate how to get around this (at least in theory), so we'll give this a shot tomorrow.  
> > > 
> > > Also, Nexenta has reversed course on FMA and has now indicated that it can't be used for the hardware events we're trying to catch.  
> > > 
> > > 
> > > Brian
> > > 
> > > 
> > > -----Original Message-----
> > > From: Patrick Haverty 
> > > Sent: Wednesday, April 15, 2009 4:46 PM
> > > To: Andy Sharp
> > > Cc: Brian Stark; Rendell Fong; John Keiffer
> > > Subject: RE: Dell Openmanage IT Assistant and SNMP Messages
> > > 
> > > Okay, it's not connecting to the BMC so I'm going to give reboot a shot. 
> > > 
> > > -----Original Message-----
> > > From: Andy Sharp 
> > > Sent: Wednesday, April 15, 2009 4:37 PM
> > > To: Patrick Haverty
> > > Cc: Brian Stark; Rendell Fong; John Keiffer
> > > Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> > > 
> > > Ouch, ok, phew, I earned my coffee today.  I can't build it as a proper package because most of the build-dependecies are not installed, and, heck, much of the plain old regular runtime dependencies don't exist either, but I hacked and I hacked, and I have installed ipmitool and ipmievd in /usr/local/bin and /usr/local/sbin respectively.
> > > 
> > > It supports only lan and bmc "plugins" because those are the only two I could get to compile, and no crypto of any kind as far as I can tell.
> > > Certainly not MD5 or MD2, which I had to hack out because the headers required to compile that stuff aren't there.
> > > 
> > > I leave it up to you to read the man pages ~:^)
> > > 
> > > It was supposed to make them static, but if they squawk and die because they can't find some library, let me know.  I won't know what to do, but at least misery will have company.
> > > 
> > > 
> > > On Wed, 15 Apr 2009 15:43:57 -0700 Patrick Haverty <patrick.haverty@onstor.com> wrote:
> > > 
> > > > Um, yeah I was kinda hoping you could look in that directory and figure out what to put on the line after install-sh.  I'm not sure what it's looking for. Consider me < clueless in this regard.
> > > > 
> > > > -----Original Message-----
> > > > From: Andy Sharp
> > > > Sent: Wednesday, April 15, 2009 3:40 PM
> > > > To: Patrick Haverty
> > > > Cc: Brian Stark; Rendell Fong; John Keiffer
> > > > Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> > > > 
> > > > On Wed, 15 Apr 2009 15:31:59 -0700 Patrick Haverty <patrick.haverty@onstor.com> wrote:
> > > > 
> > > > > The untarred ipmitool stuff is in:
> > > > > 
> > > > > root@Leopard-4:~/ipmitool/ipmitool-1.8.11#
> > > > > 
> > > > > If someone can make it install.
> > > > 
> > > > And by someone, I think you mean <clueless> me. ~:^)  Yes, I can track down the packages and eventually figure out how to make it all work.
> > > > Will take time; hopefully Dmitry can give us some useful clues.
> > > > 
> > > > > -----Original Message-----
> > > > > From: Brian Stark
> > > > > Sent: Wednesday, April 15, 2009 3:29 PM
> > > > > To: Andy Sharp
> > > > > Cc: Patrick Haverty; Rendell Fong; John Keiffer
> > > > > Subject: RE: Dell Openmanage IT Assistant and SNMP Messages
> > > > > 
> > > > > I assume that the packages can be installed on top of what we have.  The email would be sent to whatever the user wants.  Much like we do autosupport today.
> > > > > 
> > > > > I'm also not clear on how FMA is supposed to work, particularly from the configuration standpoint.  I'm going to ping Dmitry on that.
> > > > > 
> > > > > 
> > > > > -----Original Message-----
> > > > > From: Andy Sharp
> > > > > Sent: Wednesday, April 15, 2009 3:26 PM
> > > > > To: Brian Stark
> > > > > Cc: Patrick Haverty; Rendell Fong; John Keiffer
> > > > > Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> > > > > 
> > > > > Sorry, but nothing in here is news.  If we only had {ipmitool,snmptrap,*} ....
> > > > > 
> > > > > Yes, we could query the log file, but then where would we send the email messages to?  Possibly there is a way to extract or scrape that information from the nexenta world; might be worth looking into.
> > > > > 
> > > > > The fma stuff might be a way to get fma/nexenta to do that for us, however I haven't had a chance to look into that far enough to tell.
> > > > > 
> > > > > 
> > > > > 
> > > > > On Wed, 15 Apr 2009 15:08:54 -0700 Brian Stark 
> > > > > <brian.stark@onstor.com>
> > > > > wrote:
> > > > > 
> > > > > > I came across the attached whitepaper on the Dell site.
> > > > > > 
> > > > > > This seems to provide a very good framework for doing what we want with respect to event notification.
> > > > > > 
> > > > > > Pat tried doing the 'ipmitool' command, but it doesn't appear to be there in our version of Nexenta / Solaris.  If we can get support for that command, we can then use that to query the BMC for server events.  
> > > > > > 
> > > > > > Andy or Rendell, can you help with this?  It may involve installing ipmitool from Solaris 10 into our version.
> > > > > > 
> > > > > > For the JBOD, we could periodically query the syslog and then generate an SNMP alert or email.  The .pdf shows how we'd do the alert or email.
> > > > > > 
> > > > > > The .pdf also shows an example script that could be run periodically to check for status.  
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > -----Original Message-----
> > > > > > From: Andy Sharp
> > > > > > Sent: Wednesday, April 15, 2009 12:24 PM
> > > > > > To: Patrick Haverty
> > > > > > Cc: Rendell Fong; Brian Stark
> > > > > > Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> > > > > > 
> > > > > > On Wed, 15 Apr 2009 12:20:06 -0700 Patrick Haverty 
> > > > > > <patrick.haverty@onstor.com> wrote:
> > > > > > 
> > > > > > > While not ideal to have the Leopard be a SNMP trap client, it could save us the cost of an optional RAC (Remote Access Controller) by allowing us to use the Nexenta SNMP to emailer route. 
> > > > > > > 
> > > > > > 
> > > > > > Hm, OK.  You must know something I don't. ~:^)  BTW, could you 
> > > > > > please configure your email client (Outlick?) to autowrap at some 
> > > > > > reasonable right margin, like 72 or whatever the default is?  Thankyouverymuch.
> > > > > > 
> > > > > > 
> > > > > > > -----Original Message-----
> > > > > > > From: Andy Sharp
> > > > > > > Sent: Wednesday, April 15, 2009 12:14 PM
> > > > > > > To: Rendell Fong
> > > > > > > Cc: Patrick Haverty; Brian Stark
> > > > > > > Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> > > > > > > 
> > > > > > > Well, there are two programs, snmptrapd and snmpd on nexenta/solaris.
> > > > > > > I assume that if you configure/run snmptrapd, that it will listen for traps.  Not exactly sure what that would accomplish for us.  Normally in a production environment I would not want a leopard also configured to be a SNMP trap host client.  Would I?  Yes, I'm very proud of myself for using the proper SNMP terminology.  One day when I'm older I'll actually know what it means.  Oops, I'm already older.  Sigh.
> > > > > > > 
> > > > > > > 
> > > > > > > On Wed, 15 Apr 2009 12:02:50 -0700 Rendell Fong <rendell.fong@onstor.com> wrote:
> > > > > > > 
> > > > > > > > If OpenSolaris has a command like netstat you can check to see if any process is listening on udp port 162.  This port is normally used for receiving SNMP trap msgs.
> > > > > > > > 
> > > > > > > > ________________________________
> > > > > > > > From: Patrick Haverty
> > > > > > > > Sent: Wednesday, April 15, 2009 11:41 AM
> > > > > > > > To: Andy Sharp; Rendell Fong
> > > > > > > > Cc: Brian Stark
> > > > > > > > Subject: Dell Openmanage IT Assistant and SNMP Messages
> > > > > > > > 
> > > > > > > > I disabled the IPMI over LAN function of the system (which 
> > > > > > > > would not
> > > > > > > work in "teamed" or aggregated port configuration) and then verified that hardware event messages were still being sent from the PowerEdge server and received by IT assistant running on a remote computer.
> > > > > > > SNMP_Message1.JPG shows the alert as it is received (bold
> > > > > > > highlighting) and SNMP_Message2.JPG just shows the complete set 
> > > > > > > of messages (upper
> > > > > > > six) that come through when an AC cord is unplugged and then re-plugged.
> > > > > > > Note that the IP address 10.11.1.220 is the one setup in the BMC BIOS settings (mostly for IPMI use), so there is not any response to an SNMP query.  I think that means that if there is not any active monitoring when the event messages are sent, then they are lost.  I was unable to figure out how to get the BMC to send the alert to a destination that the Nexenta/OpenSolaris system noticed.  There may be an OpenSolaris service which could receive an SNMP alert, but when I tried setting up the appliances IP address as the destination address for alerts, I could not see any reflection in the /var/adm/messages file or NMS log file that would indicate the system saw the SNMP alert.
> > > > > > > > 
> > > > > > > > Pat
