AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:mail.onstor.net
NSV:
SSH:
R:<patrick.haverty@onstor.com>,<brian.stark@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@exch1.onstor.net/INBOX	0	102AB4F33EBBDB4C91915B145C8E9FB31284F9B912@exch1.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Thu, 23 Apr 2009 08:43:42 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: Patrick Haverty <patrick.haverty@onstor.com>
Cc: Brian Stark <brian.stark@onstor.com>
Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
Message-ID: <20090423084342.1333835f@ripper.onstor.net>
In-Reply-To: <102AB4F33EBBDB4C91915B145C8E9FB31284F9B912@exch1.onstor.net>
References: <20090422182134.56db3ff4@ripper.onstor.net>
	<102AB4F33EBBDB4C91915B145C8E9FB31284F9B912@exch1.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

OK.  Please send me all the files for these BMC drivers.  Is all that
is needed is to copy them to the right place?


On Wed, 22 Apr 2009 20:49:42 -0700 Patrick Haverty
<patrick.haverty@onstor.com> wrote:

> Yeah, nothing fancy, I was thinking just a separate CD that
> Abdallah...., I mean Ops, can take pop in while in !bash mode and run to
> copy the files.  We need to enter that mode for the stmsboot (multipath)
> setup anyway, and maybe it could be done at the same time.  That would
> be cool.  I used a USB drive to copy the BMC drivers, so that might be
> an option too.
> 
> Thanks!
> 
> 
> -----Original Message-----
> From: Andy Sharp
> Sent: Wednesday, April 22, 2009 6:22 PM
> To: Patrick Haverty
> Cc: Brian Stark
> Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> 
> Ah-haha, you guys don't ask for much, do you?  ~:^)
> 
> I'll try to think of a way to throw that on the CD.  Or were you talking about a separate CD?  A way to do it w/o involving the CD should be pretty easy, so hopefully that's acceptable.
> 
> On Wed, 22 Apr 2009 17:35:51 -0700 Patrick Haverty <patrick.haverty@onstor.com> wrote:
> 
> > And if you could come up with a script, that can be run from a CD with
> > all the necessary files, which will copy all the files into the
> > correct places then Ops will love it.  I've attached my doc on what
> > goes where, in case this is something you might do.
> >
> > Thanks
> >
> > -----Original Message-----
> > From: Brian Stark
> > Sent: Wednesday, April 22, 2009 5:28 PM
> > To: Andy Sharp
> > Cc: Patrick Haverty
> > Subject: RE: Dell Openmanage IT Assistant and SNMP Messages
> >
> > Agree on the near-term stuff.  We'll document a process for Ops.
> >
> > As for starting ipmievd, a quick and dirty solution sounds good.  You can play with the following machine:
> >
> > 10.0.20.180
> >
> > root/onstor
> >
> > We need to start '/usr/local/sbin/ipmievd sel' on boot-up.  Let me know what you do, and we'll get that documented as well.
> >
> >
> > -----Original Message-----
> > From: Andy Sharp
> > Sent: Wednesday, April 22, 2009 5:08 PM
> > To: Brian Stark
> > Cc: Patrick Haverty
> > Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> >
> > On Wed, 22 Apr 2009 16:57:16 -0700 Brian Stark
> > <brian.stark@onstor.com>
> > wrote:
> >
> > > Andy,
> > >
> > > Is it possible to include these programs as part of the ISO when we
> > > create CDs?  We also have some other files in /kernel/drv that need
> > > to be included.
> >
> > I'll have to look into it.  It's one thing to throw it on the CD, it's quite another to get it installed.  I doesn't work like our uber-sophisticated "cp -r *" EverOn installation process.
> >
> > > If this can't be done easily (and the key word is easily), then we
> > > can document a process for Ops on how to pull these in after an
> > > install is done.  This is easy enough and is fine for now.
> >
> > That's what we'll have to do for the near term.
> >
> > > On another note, we need a way of automatically starting 'ipmievd'
> > > on a reboot.  How can we do this?  Right now, we are starting it
> > > manually from a !bash prompt, .e.g. ipmievd sel, but this doesn't stick after a reboot.
> >
> > Looks like we can just put a thing in /etc/init.d/ as it's standard System V format, just like Linux.  Ideally we would want to actually build the package and add it to the repository, which would take care of all that, but I'm not sure the best way to do that.  It doesn't want to build now because it wants several other source or dev packages to be installed on the build machine first.  And those will want several, and so on, and so on.  This also came up in my discussions with Larry.
> >
> > I can do something quick and dirty right now on the machine in question if that helps at all.
> >
> > Cheers,
> >
> > a
> >
> > >
> > > Brian
> > >
> > >
> > > -----Original Message-----
> > > From: Andy Sharp
> > > Sent: Thursday, April 16, 2009 3:28 PM
> > > To: Brian Stark
> > > Cc: Patrick Haverty
> > > Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> > >
> > > OK.  Programs are re-installed.  I'll put a copy of them in
> > >
> > > /n/software/slowaris_x86/
> > >
> > >
> > > On Thu, 16 Apr 2009 15:21:57 -0700 Brian Stark
> > > <brian.stark@onstor.com>
> > > wrote:
> > >
> > > > ssh root@10.11.1.216
> > > >
> > > > Password is 'onstor'
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Andy Sharp
> > > > Sent: Thursday, April 16, 2009 2:46 PM
> > > > To: Brian Stark
> > > > Cc: Patrick Haverty; Rendell Fong; John Keiffer
> > > > Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> > > >
> > > > On Thu, 16 Apr 2009 14:31:30 -0700 Brian Stark
> > > > <brian.stark@onstor.com>
> > > > wrote:
> > > >
> > > > > Agree with what you outlined with SNMP and email.
> > > > >
> > > > > On the last part, we still don't know if your stuff worked.  When we tried the OpenSolaris install on its own, we got the same failure as what we saw in Nexenta with your stuff.  I then found a workaround on the Sun site that then made the ipmitool stuff work in OpenSolaris.
> > > > >
> > > > > We are now going back to the Nexenta install and will try your stuff again along with the workaround.  Pat will need your help in getting your stuff back on the Leopard once the Nexenta install is done.
> > > > >
> > > > >
> > > >
> > > > HOK.  What was this mysterious "work around?"  Are we talking a
> > > > recompile type thing or ?
> > > >
> > > >
> > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Andy Sharp
> > > > > Sent: Thursday, April 16, 2009 2:27 PM
> > > > > To: Brian Stark
> > > > > Cc: Patrick Haverty; Rendell Fong; John Keiffer
> > > > > Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> > > > >
> > > > > On Thu, 16 Apr 2009 14:01:32 -0700 Brian Stark
> > > > > <brian.stark@onstor.com>
> > > > > wrote:
> > > > >
> > > > > > Good news, we got ipmitool working today and can now see hardware events on the server.  The events are logged to /var/adm/messages.  This is the same place that the JBOD events are going.
> > > > > >
> > > > > > This means that we have a single place where all hardware events are being sent.  I've asked Nexenta if there's a way to either generate an email or SNMP alert based on the entries in the messages log.
> > > > > >
> > > > > > If not, then it's seem reasonable to come up with a script that combs through that log looking for various events and then reporting them to the user.  The whitepaper I sent out yesterday shows how this can be done via email or SNMP.  While not perfect, it's a solution that we could use as part of the first release.
> > > > > >
> > > > >
> > > > > This approach has two (one each) prerequisites that would still
> > > > > have to be addressed.  I'm not saying they're un-solvable or
> > > > > even terribly difficult, just noting them.  SNMP will need the
> > > > > snmptrap program which, like ipmitool, currently is not part of the build.
> > > > > Email will need some way to get the email address from the
> > > > > nexenta configuration, and so way to handle it when the user changes that email address.
> > > > >
> > > > > > We did all of this work on top of OpenSolaris since we couldn't get the stuff Andy did yesterday on top of Nexenta to work.  It turns out that the same problem existed with OpenSolaris until we did another workaround.  We're going to try this again on top of a Nexenta install.
> > > > > >
> > > > >
> > > > > I didn't follow that.  You mean the ipmitool I built didn't
> > > > > work, and you had to get it from somewhere else?
> > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Brian Stark
> > > > > > Sent: Wednesday, April 15, 2009 9:38 PM
> > > > > > To: Patrick Haverty; Andy Sharp
> > > > > > Cc: Rendell Fong; John Keiffer
> > > > > > Subject: RE: Dell Openmanage IT Assistant and SNMP Messages
> > > > > >
> > > > > > Pat and I are still pursuing ipmitool.  It turns out it's not as easy as we thought because OpenSolaris does not contain the necessary drivers for the BMC.  I found some posts on Sun forums that indicate how to get around this (at least in theory), so we'll give this a shot tomorrow.
> > > > > >
> > > > > > Also, Nexenta has reversed course on FMA and has now indicated that it can't be used for the hardware events we're trying to catch.
> > > > > >
> > > > > >
> > > > > > Brian
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Patrick Haverty
> > > > > > Sent: Wednesday, April 15, 2009 4:46 PM
> > > > > > To: Andy Sharp
> > > > > > Cc: Brian Stark; Rendell Fong; John Keiffer
> > > > > > Subject: RE: Dell Openmanage IT Assistant and SNMP Messages
> > > > > >
> > > > > > Okay, it's not connecting to the BMC so I'm going to give reboot a shot.
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Andy Sharp
> > > > > > Sent: Wednesday, April 15, 2009 4:37 PM
> > > > > > To: Patrick Haverty
> > > > > > Cc: Brian Stark; Rendell Fong; John Keiffer
> > > > > > Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> > > > > >
> > > > > > Ouch, ok, phew, I earned my coffee today.  I can't build it as a proper package because most of the build-dependecies are not installed, and, heck, much of the plain old regular runtime dependencies don't exist either, but I hacked and I hacked, and I have installed ipmitool and ipmievd in /usr/local/bin and /usr/local/sbin respectively.
> > > > > >
> > > > > > It supports only lan and bmc "plugins" because those are the only two I could get to compile, and no crypto of any kind as far as I can tell.
> > > > > > Certainly not MD5 or MD2, which I had to hack out because the headers required to compile that stuff aren't there.
> > > > > >
> > > > > > I leave it up to you to read the man pages ~:^)
> > > > > >
> > > > > > It was supposed to make them static, but if they squawk and die because they can't find some library, let me know.  I won't know what to do, but at least misery will have company.
> > > > > >
> > > > > >
> > > > > > On Wed, 15 Apr 2009 15:43:57 -0700 Patrick Haverty <patrick.haverty@onstor.com> wrote:
> > > > > >
> > > > > > > Um, yeah I was kinda hoping you could look in that directory and figure out what to put on the line after install-sh.  I'm not sure what it's looking for. Consider me < clueless in this regard.
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Andy Sharp
> > > > > > > Sent: Wednesday, April 15, 2009 3:40 PM
> > > > > > > To: Patrick Haverty
> > > > > > > Cc: Brian Stark; Rendell Fong; John Keiffer
> > > > > > > Subject: Re: Dell Openmanage IT Assistant and SNMP Messages
> > > > > > >
> > > > > > > On Wed, 15 Apr 2009 15:31:59 -0700 Patrick Haverty <patrick.haverty@onstor.com> wrote:
> > > > > > >
> > > > > > > > The untarred ipmitool stuff is in:
> > > > > > > >
> > > > > > > > root@Leopard-4:~/ipmitool/ipmitool-1.8.11#
> > > > > > > >
> > > > > > > > If someone can make it install.
> > > > > > >
> > > > > > > And by someone, I think you mean <clueless> me. ~:^)  Yes, I can track down the packages and eventually figure out how to make it all work.
> > > > > > > Will take time; hopefully Dmitry can give us some useful clues.
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Brian Stark
> > > > > > > > Sent: Wednesday, April 15, 2009 3:29 PM
> > > > > > > > To: Andy Sharp
> > > > > > > > Cc: Patrick Haverty; Rendell Fong; John Keiffer
> > > > > > > > Subject: RE: Dell Openmanage IT Assistant and SNMP
> > > > > > > > Messages
> > > > > > > >
> > > > > > > > I assume that the packages can be installed on top of what we have.  The email would be sent to whatever the user wants.  Much like we do autosupport today.
> > > > > > > >
> > > > > > > > I'm also not clear on how FMA is supposed to work, particularly from the configuration standpoint.  I'm going to ping Dmitry on that.
> > > > > > > >
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Andy Sharp
> > > > > > > > Sent: Wednesday, April 15, 2009 3:26 PM
> > > > > > > > To: Brian Stark
> > > > > > > > Cc: Patrick Haverty; Rendell Fong; John Keiffer
> > > > > > > > Subject: Re: Dell Openmanage IT Assistant and SNMP
> > > > > > > > Messages
> > > > > > > >
> > > > > > > > Sorry, but nothing in here is news.  If we only had {ipmitool,snmptrap,*} ....
> > > > > > > >
> > > > > > > > Yes, we could query the log file, but then where would we send the email messages to?  Possibly there is a way to extract or scrape that information from the nexenta world; might be worth looking into.
> > > > > > > >
> > > > > > > > The fma stuff might be a way to get fma/nexenta to do that for us, however I haven't had a chance to look into that far enough to tell.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, 15 Apr 2009 15:08:54 -0700 Brian Stark
> > > > > > > > <brian.stark@onstor.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I came across the attached whitepaper on the Dell site.
> > > > > > > > >
> > > > > > > > > This seems to provide a very good framework for doing what we want with respect to event notification.
> > > > > > > > >
> > > > > > > > > Pat tried doing the 'ipmitool' command, but it doesn't appear to be there in our version of Nexenta / Solaris.  If we can get support for that command, we can then use that to query the BMC for server events.
> > > > > > > > >
> > > > > > > > > Andy or Rendell, can you help with this?  It may involve installing ipmitool from Solaris 10 into our version.
> > > > > > > > >
> > > > > > > > > For the JBOD, we could periodically query the syslog and then generate an SNMP alert or email.  The .pdf shows how we'd do the alert or email.
> > > > > > > > >
> > > > > > > > > The .pdf also shows an example script that could be run periodically to check for status.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Andy Sharp
> > > > > > > > > Sent: Wednesday, April 15, 2009 12:24 PM
> > > > > > > > > To: Patrick Haverty
> > > > > > > > > Cc: Rendell Fong; Brian Stark
> > > > > > > > > Subject: Re: Dell Openmanage IT Assistant and SNMP
> > > > > > > > > Messages
> > > > > > > > >
> > > > > > > > > On Wed, 15 Apr 2009 12:20:06 -0700 Patrick Haverty
> > > > > > > > > <patrick.haverty@onstor.com> wrote:
> > > > > > > > >
> > > > > > > > > > While not ideal to have the Leopard be a SNMP trap client, it could save us the cost of an optional RAC (Remote Access Controller) by allowing us to use the Nexenta SNMP to emailer route.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hm, OK.  You must know something I don't. ~:^)  BTW,
> > > > > > > > > could you please configure your email client (Outlick?)
> > > > > > > > > to autowrap at some reasonable right margin, like 72 or whatever the default is?  Thankyouverymuch.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Andy Sharp
> > > > > > > > > > Sent: Wednesday, April 15, 2009 12:14 PM
> > > > > > > > > > To: Rendell Fong
> > > > > > > > > > Cc: Patrick Haverty; Brian Stark
> > > > > > > > > > Subject: Re: Dell Openmanage IT Assistant and SNMP
> > > > > > > > > > Messages
> > > > > > > > > >
> > > > > > > > > > Well, there are two programs, snmptrapd and snmpd on nexenta/solaris.
> > > > > > > > > > I assume that if you configure/run snmptrapd, that it will listen for traps.  Not exactly sure what that would accomplish for us.  Normally in a production environment I would not want a leopard also configured to be a SNMP trap host client.  Would I?  Yes, I'm very proud of myself for using the proper SNMP terminology.  One day when I'm older I'll actually know what it means.  Oops, I'm already older.  Sigh.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, 15 Apr 2009 12:02:50 -0700 Rendell Fong <rendell.fong@onstor.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > If OpenSolaris has a command like netstat you can check to see if any process is listening on udp port 162.  This port is normally used for receiving SNMP trap msgs.
> > > > > > > > > > >
> > > > > > > > > > > ________________________________
> > > > > > > > > > > From: Patrick Haverty
> > > > > > > > > > > Sent: Wednesday, April 15, 2009 11:41 AM
> > > > > > > > > > > To: Andy Sharp; Rendell Fong
> > > > > > > > > > > Cc: Brian Stark
> > > > > > > > > > > Subject: Dell Openmanage IT Assistant and SNMP
> > > > > > > > > > > Messages
> > > > > > > > > > >
> > > > > > > > > > > I disabled the IPMI over LAN function of the system
> > > > > > > > > > > (which would not
> > > > > > > > > > work in "teamed" or aggregated port configuration) and then verified that hardware event messages were still being sent from the PowerEdge server and received by IT assistant running on a remote computer.
> > > > > > > > > > SNMP_Message1.JPG shows the alert as it is received
> > > > > > > > > > (bold
> > > > > > > > > > highlighting) and SNMP_Message2.JPG just shows the
> > > > > > > > > > complete set of messages (upper
> > > > > > > > > > six) that come through when an AC cord is unplugged and then re-plugged.
> > > > > > > > > > Note that the IP address 10.11.1.220 is the one setup in the BMC BIOS settings (mostly for IPMI use), so there is not any response to an SNMP query.  I think that means that if there is not any active monitoring when the event messages are sent, then they are lost.  I was unable to figure out how to get the BMC to send the alert to a destination that the Nexenta/OpenSolaris system noticed.  There may be an OpenSolaris service which could receive an SNMP alert, but when I tried setting up the appliances IP address as the destination address for alerts, I could not see any reflection in the /var/adm/messages file or NMS log file that would indicate the system saw the SNMP alert.
> > > > > > > > > > >
> > > > > > > > > > > Pat
