X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7942B.C7260E3F@onstor-exch02.onstor.net>; Fri, 11 May 2007 17:23:34 -0700
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
Subject: RE: Defect  TED00018513 bsd panic (out of space in kmem_map) when killing ssc daemons randomly on Bobcat Assigned
Date: Fri, 11 May 2007 17:23:34 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E030E342D@onstor-exch02.onstor.net>
In-Reply-To: <20070511172231.1848058a@ripper.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Defect  TED00018513 bsd panic (out of space in kmem_map) when killing ssc daemons randomly on Bobcat Assigned
Thread-Index: AceUK6GbrcCzXxYvRVm0X9p+Z5wR0QAABzIw
From: "Mike Lee" <mike.lee@onstor.com>
To: "Andy Sharp" <andy.sharp@onstor.com>

Rendell has got that change covered.
-Mike

-----Original Message-----
From: Andy Sharp=20
Sent: Friday, May 11, 2007 5:23 PM
To: Mike Lee
Cc: Rendell Fong; Brian DeForest; Sandrine Boulanger; James Kahn
Subject: Re: Defect TED00018513 bsd panic (out of space in kmem_map)
when killing ssc daemons randomly on Bobcat Assigned


On Fri, 11 May 2007 17:09:14 -0700 "Mike Lee" <mike.lee@onstor.com>
wrote:


> -----Original Message-----
> From: Maxim Kozlovsky=20
> Sent: Friday, May 11, 2007 10:49 AM
> To: Mike Lee; Rendell Fong; Sandrine Boulanger; Brian DeForest
> Cc: James Kahn
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
>=20
> It will be better if you fix this problem first before running the qa
> scripts. This is an obvious kernel memory leak, don't try to find
> another one without fixing the known problem.
>=20
> (The best way to go about fixing the ssccc is to move the mounting of
> the ramdisk to the rc.agile so it is done just once).


Please do this as the fix.  Not being able to find where this is mounted
[by searching the rc scripts] is a real nightmare for systems guys.

Thanks.


> -----Original Message-----
> From: Mike Lee=20
> Sent: Friday, May 11, 2007 10:45 AM
> To: Rendell Fong; Sandrine Boulanger; Brian DeForest
> Cc: James Kahn; Maxim Kozlovsky
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
>=20
> Randell:
>=20
> Thanks for the insight.=20
> I only killed ssccc once last night, and looking at my filer, your
> observation is correct, as I have two /tmp/ramdisk mount points. =20
>=20
> I plan to just use the QA scripts today.
> If time permits, I will also try your technique.
>=20
> -Mike
> -----Original Message-----
> From: Rendell Fong=20
> Sent: Friday, May 11, 2007 10:08 AM
> To: Mike Lee; Sandrine Boulanger; Brian DeForest
> Cc: James Kahn; Maxim Kozlovsky
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
>=20
>=20
> I don't know if this is related but I ran a test that constantly kills
> off the sscccc process and lets the pm restart it.  When it restarts
> it always creates a new mount_mfs process and creates a mounted fs for
> /tmp/ramdisk.  They seem to linger even after sscccc is killed off and
> have a cumulative effect.
>=20
> See enclosed.
>=20
> Rendell
>=20
>=20
> -----Original Message-----
> From: Mike Lee=20
> Sent: Wednesday, May 09, 2007 11:14 PM
> To: Sandrine Boulanger; Brian DeForest
> Cc: James Kahn; Maxim Kozlovsky; Rendell Fong
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
>=20
>=20
> Sandrine:
>=20
> Thank you for the clarification; I'll try again tomorrow, and perhaps
> I'll need to enlist your help.
>=20
> -Mike
>=20
> -----Original Message-----
> From: Sandrine Boulanger
> Sent: Wed 5/9/2007 7:26 PM
> To: Mike Lee; Brian DeForest
> Cc: James Kahn; Maxim Kozlovsky; Rendell Fong
> Subject: RE: Defect  TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
> =20
> I don't think killing a particular daemon is going to reproduce it. I
> think there could be a memory leak when we restart any daemon that
> will eventually lead to the bsd crashing.
>=20
> To reproduce is very simple, just follow the steps to reproduce and
> you're there, that's what I did again today.=20
>=20
> -----Original Message-----
> From: Mike Lee=20
> Sent: Wednesday, May 09, 2007 6:43 PM
> To: Brian DeForest
> Cc: James Kahn; Maxim Kozlovsky; Rendell Fong; Sandrine Boulanger
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
>=20
> Brian:
> It sure looks like that the kill must be done at an opportune time,
> based on the message entries you captured below.
> So, I guess we can easily ask Sandrine to try and kill crashsaved
> again. I will try on my filer as well tomorrow.
> THanks.
> -Mike
>=20
> -----Original Message-----
> From: Brian DeForest=20
> Sent: Wednesday, May 09, 2007 6:32 PM
> To: Mike Lee
> Cc: James Kahn; Maxim Kozlovsky; Rendell Fong
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
>=20
>=20
> Thanks Mike.   Considering you weren't able to reproduce originally,
> we probably need to look closely at how she reproduced it. Looks like
> killing crashsaved did it.
>=20
> Could also be that /var is 100% full, which could be tested easily.
>=20
> May  9 16:59:24 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:59:25 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/crashsaved (pid 32099) terminated with signal
> Killed=20
> May  9 16:59:25 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/crashsaved:
> finished initialization.=20
> panic: malloc: out of space in kmem_map
> dumpsys: cannot open crashdump device: 16
>=20
> dumping to dev 411, offset 8
> dump not yet implementedSystem restart.
>=20
> PMON CPU 00000000 Initializing. Standby...
>=20
> -----Original Message-----
> From: Mike Lee=20
> Sent: Wednesday, May 09, 2007 6:15 PM
> To: Brian DeForest
> Cc: Sandrine Boulanger; James Kahn
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
>=20
> Brian:
>=20
> I can try to repeat Sandrine's experiment on top of tree tomorrow, if
> you think it is worthwhile.
> Or perhaps we want to wait for Jim's check in?
>=20
> Pls let me know.
> Thnx.
>=20
> -Mike
>=20
> -----Original Message-----
> From: sandrine.boulanger@onstor.com
> [mailto:sandrine.boulanger@onstor.com]
> Sent: Wednesday, May 09, 2007 6:07 PM
> To: Mike Lee
> Cc: Sandrine Boulanger; Sandrine Boulanger; Brian DeForest
> Subject: Defect TED00018513 bsd panic (out of space in kmem_map) when
> killing ssc daemons randomly on Bobcat Assigned
>=20
>=20
> id: TED00018513
> Headline: bsd panic (out of space in kmem_map) when killing ssc
> daemons randomly on Bobcat
> Note_Entry: I reproduced it again in sub#15:
>=20
> Wed May  9 16:16:36 PDT 2007
>=20
> login: May  9 16:38:27 eng60 : 0:0:nfxsh:NOTICE: cmd[5]: elog display
> ena : status[0]
> May  9 16:38:30 eng60 : 0:0:nfxsh:NOTICE: cmd[6]: elog show config  :
> status[0]
> May  9 16:38:36 eng60 : 1:0:e_log:INFO: 15: e-log configure OK, cmd
> 9249 6 0 0
> May  9 16:38:36 eng60 : 2:0:e_log:INFO: 30: e-log configure OK, cmd
> 9249 6 0 0
> May  9 16:38:36 eng60 : 1:2:e_log:INFO: 22: e-log configure OK, cmd
> 9249 6 0 0
> May  9 16:38:36 eng60 : 2:0:e_log:INFO: 31: e-log configure OK, cmd
> 9249 6 0 0
> May  9 16:38:36 eng60 : 2:0:e_log:INFO: 32: e-log configure OK, cmd
> 9249 6 0 0
> May  9 16:38:36 eng60 : 2:0:e_log:INFO: 33: e-log configure OK, cmd
> 9249 6 0 0
> May  9 16:38:36 eng60 : 0:0:nfxsh:NOTICE: cmd[7]: elog level info :
> status[0]
> May  9 16:38:37 eng60 : 0:0:nfxsh:NOTICE: cmd[8]: elog show config  :
> status[0]
> May  9 16:39:30 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:39:32 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/vtmd (pid 8540) terminated with signal Killed=20
> May  9 16:39:33 eng60 : 0:0:cluster2:INFO: ClusterCtrl_releaseFiler:
> called by vtm=20
> May  9 16:39:33 eng60 : 0:0:vtm:INFO: vtm_sendCardStateMsg: Sending
> card UP to eng60=20
> May  9 16:39:33 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/vtmd:
> finished initialization.=20
> May  9 16:39:52 eng60 : 0:0:vtm:INFO: Vtm_ProcCardStateMsg: card state
> UP message from eng60=20
> May  9 16:39:57 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:39:58 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/ncmd (pid 27267) terminated with signal Killed=20
> May  9 16:39:59 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:40:00 eng60 : 0:0:cluster2:ERROR: main: rcv ncm msg =20
> May  9 16:40:00 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open failed
> May  9 16:40:00 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:40:00 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/ncmd:
> finished initialization.=20
> May  9 16:40:01 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open failed
> May  9 16:40:01 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:40:01 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open success
> May  9 16:40:04 eng60 : 0:0:ea:NOTICE: ea_closeSess : NCM session
> closed May  9 16:40:04 eng60 : 0:0:ea:NOTICE: ea_rcvRmcMsg : NCM
> session open success
> May  9 16:40:24 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:40:25 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/evm_cfgd (pid 18999) terminated with signal
> Killed May  9 16:40:25 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/evm_cfgd: finished initialization.=20
> May  9 16:40:27 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open failed
> May  9 16:40:27 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open failed
> May  9 16:40:27 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:40:27 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:40:27 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open success
> May  9 16:40:27 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open success
> May  9 16:40:50 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:40:52 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sdm_cfgd (pid 12313) terminated with signal
> Killed May  9 16:40:52 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sdm_cfgd: finished initialization.=20
> May  9 16:40:52 eng60 : 0:0:sdm:INFO: ONStor Storage Device Manager
> (c)2003: Started
> May  9 16:40:52 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:40:53 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open failed
> May  9 16:40:53 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:40:53 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open success
> May  9 16:40:53 eng60 : 0:0:sdm:INFO: sdm_rcvRmcMsg : RMC open
> successful (0)
> May  9 16:40:53 eng60 : 0:0:sdm:INFO: sdm_rcvRmcMsg : RMC open
> successful (0)
> May  9 16:40:53 eng60 : 0:0:sdm:INFO: sdm_rcvRmcMsg : RMC Open to FC
> Port successful {sdm:Agile2775}->{:scsi2.2.0}
> May  9 16:40:53 eng60 : 2:0:scsi:INFO: 34: scsi_openSessToSDM:RMC
> rc[-9] msg[0x0]
> May  9 16:40:53 eng60 : 2:0:scsi:INFO: 35: scsi_openSessToSDM: RMC
> session found open at [0x60e51010], issue close for session. tagVal =
=3D
> [40c] =20
> May  9 16:40:53 eng60 : 2:0:scsi:INFO: 36: scsi_openSessToSDM:
> scsiCtrl->sdmOpenSession =3D 0x60e51010, and &scsiCtrl->sdmOpenSession =
=3D
> 0x623a98bc =20
> May  9 16:40:54 eng60 : 0:0:tape-driver:ERROR: tape_rpc: rmc_rpc app
> sdm cpu 0 slot 0 failed - -3
> May  9 16:41:03 eng60 : 0:0:spm:WARNING: spm_sendNcmMsg: ncm_send()
> failed - -20
> May  9 16:41:03 eng60 : 0:0:evm:NOTICE: evm_getTxn : transaction with
> handle[0xe9020000] already freed
> May  9 16:41:03 eng60 : 0:0:evm:ERROR: evm_processMsg : Stale SPM
> response[2607]
> May  9 16:41:03 eng60 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
> before open completed
> May  9 16:41:03 eng60 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
> before open completed
> May  9 16:41:17 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:41:18 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/auth-agent (pid 5401) terminated with signal
> Killed
>=20
> May  9 16:41:20 eng60 : 0:0:auth_agent:NOTICE: main: auth-agent
> started May  9 16:41:20 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/auth-agent: finished
> initialization. May  9 16:41:20 eng60 : 0:0:ea:WARNING:
> nfxnis_createVsContext: Context for VS=3D1 already exists.=20
> May  9 16:41:20 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=3D2 already exists.=20
> May  9 16:41:24 eng60 : 0:0:sdm:INFO: Success reading configuration
> record from cluster db.
> May  9 16:41:39 eng60 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
> before open completed
> May  9 16:41:44 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:41:45 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/asd (pid 24872) terminated with signal Killed=20
> May  9 16:41:46 eng60 : 0:0:eventd:INFO: close_session: Lost
> connection. Close rmc session 0x100b7c00 for event session 0x1008b608
> app unused 0. May  9 16:41:46 eng60 : 0:0:asd:INFO: asd_main: auto
> support conf is TO , NOTETO , FROM eng60@onstor.com, server 0.0.0.0,
> enable 0 May  9 16:41:46 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/asd: finished initialization.=20
> May  9 16:41:57 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:41:57 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open success
> May  9 16:42:00 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol1', Id 0x000001b400000065, Event 'Create', State 'Down'
> May  9 16:42:01 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol2', Id 0x000001b400000066, Event 'Create', State 'Down'
> May  9 16:42:01 eng60 : 0:0:vsd:ERROR: vsd_sendMsg[9123] :
> rmc_send_msg_ex failed. rc=3D-3
> May  9 16:42:01 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol1)
> current state - Mounted-RW State.
> May  9 16:42:01 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol1)
> already mounted in desired state.
> May  9 16:42:01 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol2)
> current state - Mounted-RW State.
> May  9 16:42:01 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol2)
> already mounted in desired state.
> May  9 16:42:01 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol3', Id 0x000001b400000067, Event 'Create', State 'Down'
> May  9 16:42:01 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol4', Id 0x000001b400000068, Event 'Create', State 'Down'
> May  9 16:42:01 eng60 : 0:0:evm:ERROR: evm_putVolOnline : Volume[vol3]
> is already online
> May  9 16:42:01 eng60 : 0:0:ea:INFO: ea_efsMountVol : Volume (vol3)
> already mounted in the correct state.
> May  9 16:42:01 eng60 : 0:0:evm:ERROR: evm_putVolOnline : Volume[vol4]
> is already online
> May  9 16:42:01 eng60 : 0:0:ea:INFO: ea_efsMountVol : Volume (vol4)
> already mounted in the correct state.
> May  9 16:42:10 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:42:11 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 18145) terminated with signal
> Killed May  9 16:42:13 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization.=20
> May  9 16:42:14 eng60 : 0:0:sscccc:NOTICE: New client 0: addr
> 10.0.0.135, port 3036; Assigning fd(12)
> May  9 16:42:16 eng60 : 0:0:nfxsh:NOTICE: cmd[0]: cluster show
> summary : status[0]
> May  9 16:42:16 eng60 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show group :
> status[0]
> May  9 16:42:16 eng60 : 0:0:nfxsh:NOTICE: cmd[2]: vsvr show all -v :
> status[0]
> May  9 16:42:16 eng60 : 0:0:nfxsh:NOTICE: cmd[3]: volume show :
> status[0]
> May  9 16:42:18 eng60 : 0:0:nfxsh:NOTICE: cmd[0]: cluster show
> summary : status[0]
> May  9 16:42:20 eng60 : 0:0:nfxsh:NOTICE: cmd[0]: volume show -r :
> status[0]
> May  9 16:42:20 eng60 : 0:0:sscccc:INFO: bad read, bytes=3D0 (client 0
> socket 12)
> May  9 16:42:20 eng60 : 0:0:sscccc:NOTICE: closing session for client
> 0 May  9 16:42:20 eng60 : 0:0:sscccc:NOTICE: New client 0: addr
> 10.0.0.135, port 3040; Assigning fd(12)
> May  9 16:42:20 eng60 : 0:0:sscccc:INFO: bad read, bytes=3D-1 (client =
0
> socket 12)
> May  9 16:42:20 eng60 : 0:0:sscccc:NOTICE: closing session for client
> 0 May  9 16:42:37 eng60 : 0:0:ssh:INFO: 'root' logged in through
> remote host(10.0.0.94)
> May  9 16:42:38 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/ncmd (pid 11932) terminated with signal Killed=20
> May  9 16:42:39 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:42:40 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/ncmd:
> finished initialization.=20
> May  9 16:42:40 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open failed
> May  9 16:42:40 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:42:40 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open success
> May  9 16:42:44 eng60 : 0:0:ea:NOTICE: ea_closeSess : NCM session
> closed May  9 16:42:44 eng60 : 0:0:ea:NOTICE: ea_rcvRmcMsg : NCM
> session open success
> May  9 16:43:03 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:43:05 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/chassisd (pid 32145) terminated with signal
> Killed May  9 16:43:05 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/chassisd: finished initialization.=20
> May  9 16:43:05 eng60 : 0:0:cm:NOTICE: CHASSISD: Chassis Manager:
> Started
> May  9 16:43:05 eng60 : 0:0:cm:NOTICE: CM: opened kseg1
> May  9 16:43:05 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 0, State Up
> May  9 16:43:05 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 0=20
> May  9 16:43:05 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 1, State Up
> May  9 16:43:05 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 2, State Up
> May  9 16:43:05 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 3, State Up
> May  9 16:43:05 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 2,
> CPU 0, State Up
> May  9 16:43:05 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 1=20
> May  9 16:43:05 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 2=20
> May  9 16:43:05 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 3=20
> May  9 16:43:05 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 2, cpu 0=20
> May  9 16:43:30 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:43:31 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/auth-agent (pid 21156) terminated with signal
> Killed=20
> May  9 16:43:33 eng60 : 0:0:auth_agent:NOTICE: main: auth-agent
> started May  9 16:43:33 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/auth-agent: finished
> initialization. May  9 16:43:33 eng60 : 0:0:ea:WARNING:
> nfxnis_createVsContext: Context for VS=3D1 already exists.=20
> May  9 16:43:33 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=3D2 already exists.=20
> May  9 16:43:57 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:43:58 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/asd (pid 17258) terminated with signal Killed=20
> May  9 16:43:59 eng60 : 0:0:eventd:INFO: close_session: Lost
> connection. Close rmc session 0x100b7c00 for event session 0x1008b608
> app unused 0. May  9 16:43:59 eng60 : 0:0:asd:INFO: asd_main: auto
> support conf is TO , NOTETO , FROM eng60@onstor.com, server 0.0.0.0,
> enable 0 May  9 16:43:59 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/asd: finished initialization.=20
> May  9 16:44:23 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:44:25 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/ndmp_cfgd (pid 8912) terminated with signal
> Killed May  9 16:44:26 eng60 : 0:0:ssc_ndmp:INFO: ndmp_GetFPState:
> fp card is UP=20
> May  9 16:44:26 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/ndmp_cfgd:
> finished initialization.=20
> May  9 16:44:50 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:44:51 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/auth-agent (pid 29019) terminated with signal
> Killed=20
> May  9 16:44:53 eng60 : 0:0:auth_agent:NOTICE: main: auth-agent
> started May  9 16:44:53 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/auth-agent: finished
> initialization. May  9 16:44:53 eng60 : 0:0:ea:WARNING:
> nfxnis_createVsContext: Context for VS=3D1 already exists.=20
> May  9 16:44:53 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=3D2 already exists.=20
> May  9 16:45:17 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:45:18 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/snmpd (pid 19673) terminated with signal Killed=20
> May  9 16:45:18 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/snmpd:
> finished initialization.=20
> May  9 16:45:20 eng60 : 0:0:snmpd:INFO: UCD-SNMP version 4.2.2=20
> May  9 16:45:43 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:45:45 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 15549) terminated with signal
> Killed May  9 16:45:46 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization.=20
> May  9 16:46:10 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:46:11 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/ipmd (pid 6227) terminated with signal Killed=20
> May  9 16:46:12 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/ipmd:
> finished initialization.=20
> May  9 16:46:37 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:46:38 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sanmd (pid 25855) terminated with signal Killed=20
> May  9 16:46:39 eng60 : 1:2:sanm_ag:WARNING: 23: sanm_doRMCLocalClose:
> reopening session to sanmd=20
> May  9 16:46:40 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/sanmd:
> finished initialization.=20
> May  9 16:46:40 eng60 : 0:0:sanm:NOTICE: SANM: ONStor Data Mirror
> (c)2006: Started
> May  9 16:47:03 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:47:05 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/vtmd (pid 17793) terminated with signal Killed=20
> May  9 16:47:06 eng60 : 0:0:cluster2:INFO: ClusterCtrl_releaseFiler:
> called by vtm=20
> May  9 16:47:06 eng60 : 0:0:vtm:INFO: vtm_sendCardStateMsg: Sending
> card UP to eng60=20
> May  9 16:47:06 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/vtmd:
> finished initialization.=20
> May  9 16:47:07 eng60 : 0:0:vtm:INFO: Vtm_ProcCardStateMsg: card state
> UP message from eng60=20
> May  9 16:47:30 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:47:31 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/auth-agent (pid 11683) terminated with signal
> Killed=20
> May  9 16:47:33 eng60 : 0:0:auth_agent:NOTICE: main: auth-agent
> started May  9 16:47:33 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/auth-agent: finished
> initialization. May  9 16:47:33 eng60 : 0:0:ea:WARNING:
> nfxnis_createVsContext: Context for VS=3D1 already exists.=20
> May  9 16:47:33 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=3D2 already exists.=20
> May  9 16:47:57 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:47:58 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/vsd (pid 3150) terminated with signal Killed=20
> May  9 16:47:59 eng60 : 0:0:ea:INFO: ea_closeAcptSess: close from
> {vsd:Agile1358.0.0}
> May  9 16:48:00 eng60 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1223] : vs=3D1, file=3D71, #masterRec=3D0
> May  9 16:48:00 eng60 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1223] : vs=3D1, file=3D72, #masterRec=3D0
> May  9 16:48:00 eng60 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1223] : vs=3D1, file=3D74, #masterRec=3D0
> May  9 16:48:00 eng60 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1223] : vs=3D2, file=3D71, #masterRec=3D0
> May  9 16:48:00 eng60 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1223] : vs=3D2, file=3D72, #masterRec=3D0
> May  9 16:48:00 eng60 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1223] : vs=3D2, file=3D74, #masterRec=3D0
> May  9 16:48:00 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/vsd:
> finished initialization.=20
> May  9 16:48:03 eng60 : 0:0:ea:INFO: ea_newAcptSess: creating new
> session
> May  9 16:48:03 eng60 : 0:0:ea:INFO: ea_openSess: open for
> {vsd:Agile3651.0.0}
> May  9 16:48:03 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=3D1 already exists.=20
> May  9 16:48:03 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol1)
> current state - Mounted-RW State.
> May  9 16:48:03 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol1)
> already mounted in desired state.
> May  9 16:48:03 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol2)
> current state - Mounted-RW State.
> May  9 16:48:03 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol2)
> already mounted in desired state.
> May  9 16:48:03 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol3)
> current state - Mounted-RW State.
> May  9 16:48:03 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol3)
> already mounted in desired state.
> May  9 16:48:03 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol4)
> current state - Mounted-RW State.
> May  9 16:48:03 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol4)
> already mounted in desired state.
> May  9 16:48:04 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=3D2 already exists.=20
> May  9 16:48:24 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:48:25 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/cluster_relay (pid 24390) terminated with signal
> Killed=20
> May  9 16:48:25 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/cluster_relay: finished
> initialization. May  9 16:48:50 eng60 : 0:0:ssh:INFO: 'root' logged
> in through remote host(10.0.0.94)
> May  9 16:48:52 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sscccc (pid 18116) terminated with signal Killed=20
> May  9 16:48:52 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/sscccc:
> finished initialization.=20
> May  9 16:48:52 eng60 : 0:0:unused:INFO: sscccc no
> /usr/local/agile/etc/sscccc_hosts_deny file
> May  9 16:49:17 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:49:18 eng60 pm: pm_sig_handler: /usr/local/agile/bin/elog
> (pid 11657) terminated with signal Killed=20
> May  9 16:49:19 eng60 pm: /usr/local/agile/bin/elog: finished
> initialization.=20
> May  9 16:49:44 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:49:45 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/chassisd (pid 330) terminated with signal Killed=20
> May  9 16:49:45 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/chassisd:
> finished initialization.=20
> May  9 16:49:45 eng60 : 0:0:cm:NOTICE: CHASSISD: Chassis Manager:
> Started
> May  9 16:49:45 eng60 : 0:0:cm:NOTICE: CM: opened kseg1
> May  9 16:49:45 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 0, State Up
> May  9 16:49:45 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 0=20
> May  9 16:49:45 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 1, State Up
> May  9 16:49:45 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 2, State Up
> May  9 16:49:45 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 3, State Up
> May  9 16:49:45 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 2,
> CPU 0, State Up
> May  9 16:49:45 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 1=20
> May  9 16:49:45 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 2=20
> May  9 16:49:45 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 3=20
> May  9 16:49:45 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 2, cpu 0=20
> May  9 16:50:10 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:50:12 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/vtmd (pid 22618) terminated with signal Killed=20
> May  9 16:50:13 eng60 : 0:0:cluster2:INFO: ClusterCtrl_releaseFiler:
> called by vtm=20
> May  9 16:50:13 eng60 : 0:0:vtm:INFO: vtm_sendCardStateMsg: Sending
> card UP to eng60=20
> May  9 16:50:13 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/vtmd:
> finished initialization.=20
> May  9 16:50:31 eng60 : 0:0:vtm:INFO: Vtm_ProcCardStateMsg: card state
> UP message from eng60=20
> May  9 16:50:37 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:50:59 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:51:00 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 8085) terminated with signal
> Killed May  9 16:51:02 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization.=20
> May  9 16:51:26 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:51:27 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/tape-driver (pid 23456) terminated with signal
> Killed=20
> May  9 16:51:27 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/tape-driver:
> finished initialization.=20
> May  9 16:51:29 eng60 : 0:0:tape-driver:NOTICE: Adding new tape
> device: id 00000000f91f8ee8, WWN 500308c13d680003, LUN 0, vendor
> 'IBM', model 'ULT3580-TD2'=20
> May  9 16:51:29 eng60 : 0:0:tape-driver:NOTICE: Adding new
> media-changer: id 00000001796dd4fb, WWN 500308c13d680003, LUN 1,
> vendor 'IBM', model 'ULT3582-TL'=20
> May  9 16:51:52 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:52:14 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:52:15 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 8481) terminated with signal
> Killed May  9 16:52:17 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization.=20
> May  9 16:52:20 eng60 : 0:0:sscccc:NOTICE: New client 0: addr
> 10.0.0.135, port 3116; Assigning fd(12)
> May  9 16:52:22 eng60 : 0:0:nfxsh:NOTICE: cmd[0]: cluster show
> summary : status[0]
> May  9 16:52:22 eng60 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show group :
> status[0]
> May  9 16:52:22 eng60 : 0:0:nfxsh:NOTICE: cmd[2]: vsvr show all -v :
> status[0]
> May  9 16:52:22 eng60 : 0:0:nfxsh:NOTICE: cmd[3]: volume show :
> status[0]
> May  9 16:52:24 eng60 : 0:0:nfxsh:NOTICE: cmd[0]: cluster show
> summary : status[0]
> May  9 16:52:26 eng60 : 0:0:nfxsh:NOTICE: cmd[0]: volume show -r :
> status[0]
> May  9 16:52:26 eng60 : 0:0:sscccc:INFO: bad read, bytes=3D0 (client 0
> socket 12)
> May  9 16:52:26 eng60 : 0:0:sscccc:NOTICE: closing session for client
> 0 May  9 16:52:26 eng60 : 0:0:sscccc:NOTICE: New client 0: addr
> 10.0.0.135, port 3119; Assigning fd(12)
> May  9 16:52:26 eng60 : 0:0:sscccc:INFO: bad read, bytes=3D-1 (client =
0
> socket 12)
> May  9 16:52:26 eng60 : 0:0:sscccc:NOTICE: closing session for client
> 0 May  9 16:52:41 eng60 : 0:0:ssh:INFO: 'root' logged in through
> remote host(10.0.0.94)
> May  9 16:52:42 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/tape-driver (pid 27313) terminated with signal
> Killed=20
> May  9 16:52:42 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/tape-driver:
> finished initialization.=20
> May  9 16:52:44 eng60 : 0:0:tape-driver:NOTICE: Adding new tape
> device: id 00000000f91f8ee8, WWN 500308c13d680003, LUN 0, vendor
> 'IBM', model 'ULT3580-TD2'=20
> May  9 16:52:44 eng60 : 0:0:tape-driver:NOTICE: Adding new
> media-changer: id 00000001796dd4fb, WWN 500308c13d680003, LUN 1,
> vendor 'IBM', model 'ULT3582-TL'=20
> May  9 16:53:08 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:53:09 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sscccc (pid 19184) terminated with signal Killed=20
> May  9 16:53:09 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/sscccc:
> finished initialization.=20
> May  9 16:53:09 eng60 : 0:0:unused:INFO: sscccc no
> /usr/local/agile/etc/sscccc_hosts_deny file
> May  9 16:53:34 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:53:36 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/eventd (pid 14473) terminated with signal Killed=20
> May  9 16:53:36 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/eventd:
> finished initialization.=20
> May  9 16:53:36 eng60 : 0:0:asd:ERROR: asd_main: Lost connection to
> eventd. Reinitializing conn =20
> May  9 16:53:36 eng60 : 0:0:asd:NOTICE: asd_main: Connected to
> eventd May  9 16:54:01 eng60 : 0:0:ssh:INFO: 'root' logged in through
> remote host(10.0.0.94)
> May  9 16:54:02 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sdm_cfgd (pid 8039) terminated with signal
> Killed May  9 16:54:02 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sdm_cfgd: finished initialization.=20
> May  9 16:54:02 eng60 : 0:0:sdm:INFO: ONStor Storage Device Manager
> (c)2003: Started
> May  9 16:54:03 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:54:03 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open success
> May  9 16:54:04 eng60 : 0:0:sdm:INFO: sdm_rcvRmcMsg : RMC open
> successful (0)
> May  9 16:54:04 eng60 : 0:0:sdm:INFO: sdm_rcvRmcMsg : RMC open
> successful (0)
> May  9 16:54:04 eng60 : 0:0:sdm:INFO: sdm_rcvRmcMsg : RMC Open to FC
> Port successful {sdm:Agile3974}->{:scsi2.2.0}
> May  9 16:54:04 eng60 : 2:0:scsi:INFO: 37: scsi_openSessToSDM:RMC
> rc[-9] msg[0x0]
> May  9 16:54:04 eng60 : 2:0:scsi:INFO: 38: scsi_openSessToSDM: RMC
> session found open at [0x60e51010], issue close for session. tagVal =
=3D
> [40c] =20
> May  9 16:54:04 eng60 : 2:0:scsi:INFO: 39: scsi_openSessToSDM:
> scsiCtrl->sdmOpenSession =3D 0x60e51010, and &scsiCtrl->sdmOpenSession =
=3D
> 0x623a98bc =20
> May  9 16:54:08 eng60 : 0:0:tape-driver:ERROR: tape_rpc: rmc_rpc app
> sdm cpu 0 slot 0 failed - -3
> May  9 16:54:28 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:54:29 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/auth-agent (pid 24682) terminated with signal
> Killed=20
> May  9 16:54:31 eng60 : 0:0:auth_agent:NOTICE: main: auth-agent
> started May  9 16:54:31 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/auth-agent: finished
> initialization. May  9 16:54:31 eng60 : 0:0:ea:WARNING:
> nfxnis_createVsContext: Context for VS=3D1 already exists.=20
> May  9 16:54:31 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=3D2 already exists.=20
> May  9 16:54:36 eng60 : 0:0:sdm:INFO: Success reading configuration
> record from cluster db.
> May  9 16:54:55 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:54:56 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/ea (pid 18322) terminated with signal Killed=20
> May  9 16:54:56 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/ea: finished
> initialization.=20
> May  9 16:54:58 eng60 : 0:0:ea:NOTICE: ea_rcvRmcMsg : NCM session open
> failed
> May  9 16:55:03 eng60 : 0:0:ea:NOTICE: ea_closeSess : NCM session
> closed May  9 16:55:03 eng60 : 0:0:ea:NOTICE: ea_rcvRmcMsg : NCM
> session open success
> May  9 16:55:22 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:55:23 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/snmpd (pid 12658) terminated with signal Killed=20
> May  9 16:55:23 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/snmpd:
> finished initialization.=20
> May  9 16:55:25 eng60 : 0:0:snmpd:INFO: UCD-SNMP version 4.2.2=20
> May  9 16:55:48 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:55:50 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/cluster_relay (pid 5022) terminated with signal
> Killed=20
> May  9 16:55:50 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/cluster_relay: finished
> initialization. May  9 16:56:15 eng60 : 0:0:ssh:INFO: 'root' logged
> in through remote host(10.0.0.94)
> May  9 16:56:17 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/spm (pid 21333) terminated with signal Killed=20
> May  9 16:56:18 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/spm:
> finished initialization.=20
> May  9 16:56:23 eng60 : 0:0:evm:ERROR: evm_readLunLabelReq : SDM
> error[-1] handle[0x1210000] for read label
> LUN[MYLEX_20000080e5128ae5_3] May  9 16:56:23 eng60 : 0:0:evm:ERROR:
> evm_verifyLunReqProc : Cannot read label for
> LUN[MYLEX_20000080e5128ae5_3] May  9 16:56:23 eng60 : 0:0:evm:ERROR:
> evm_verifyLunReqProc: Missing label on LUN[MYLEX_20000080e5128ae5_3]!
> May  9 16:56:23 eng60 : 0:0:evm:ERROR: evm_verifyLunReqProc: removing
> vol[] LUN[MYLEX_20000080e5128ae5_3] from cluster[eng60]
> May  9 16:56:23 eng60 : 0:0:evm:ERROR: evm_readLunLabelReq : SDM
> error[-1] handle[0x1210001] for read label
> LUN[MYLEX_20000080e5128ae5_16]
> May  9 16:56:23 eng60 : 0:0:evm:ERROR: evm_verifyLunReqProc : Cannot
> read label for LUN[MYLEX_20000080e5128ae5_16]
> May  9 16:56:23 eng60 : 0:0:evm:ERROR: evm_verifyLunReqProc: Missing
> label on LUN[MYLX_20000080e5128ae5_16]!
> May  9 16:56:23 eng60 : 0:0:evm:ERROR:2         0x1b400000066 - write
> - buf - Volume Signal Exception: frame[5] =3D 0xffffffff83062830=20
> May  9 16:56:23 eng60 : 1:2:efs:WARNING: 32: FS: vol2
> 0x1b400000066 - write - buf - Volume Signal Exception: frame[6] =3D
> 0xffffffff83056190=20
> May  9 16:56:24 eng60 : 1:2:efs:WARNING: 33: FS: vol2
> 0x1b400000066 - write - buf - Volume Signal Exception: frame[7] =3D
> 0xffffffff830563f0=20
> May  9 16:56:24 eng60 : 1:2:efs:WARNING: 34: FS: vol2
> 0x1b400000066 - write - buf - Volume Signal Exception: frame[8] =3D
> 0xffffffff830cf768=20
> May  9 16:56:24 eng60 : 1:2:efs:WARNING: 35: FS: vol2
> 0x1b400000066 - write - buf - Volume Signal Exception: frame[9] =3D
> 0xffffffff830e3a60=20
> May  9 16:56:24 eng60 : 1:3:evm:ERROR: 36: evm_verifyReq: cannot find
> volume[0x1b400000068]
> May  9 16:56:24 eng60 : 1:2:evm:ERROR: 37: evm_verifyReq: cannot find
> volume[0x1b400000067]
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 38: FILESYSTEM WARNING on
> volId 0x1b400000068: volume write, filesystem error on volId
> 0x1b400000068 (vol4), aborting volume =20
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 39: FS: vol4
> 0x1b400000068 - write - buf - Volume Signal Exception: frame[0] =3D
> 0xffffffff8304ee40=20
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 40: FS: vol4
> 0x1b400000068 - write - buf - Volume Signal Exception: frame[1] =3D
> 0xffffffff830abf74=20
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 48: FS: vol4
> 0x1b400000068 - write - buf - Volume Signal Exception: frame[9] =3D
> 0xffffffff830d0ba4=20
> May  9 16:56:24 eng60 : 1:2:efs:WARNING: 49: FILESYSTEM WARNING on
> volId 0x1b400000067: volume write, filesystem error on volId
> 0x1b400000067 (vol3), aborting volume =20
> May  9 16:56:24 eng60 : 1:2:efs:WARNING: 50: FS: vol3
> 0x1b400000067 - write - buf - Volume Signal Exception: frame[0] =3D
> 0xffffffff8304ee40=20
> May  9 16:56:24 eng60 : 1:2:efs:WARNING: 51: FS: vol3
> 0x1b400000067 - write - buf - Volume Signal Exception: frame[1] =3D
> 0xffffffff830abf74=20
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 61: FILESYSTEM WARNING on
> volId 0x1b400000065: volume write, filesystem error on volId
> 0x1b400000065 (vol1), aborting volume =20
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 62: FS: vol1
> 0x1b400000065 - remove - buf - Volume Signal Exception: frame[0] =3D
> 0xffffffff8304ee40=20
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 63: FS: vol1
> 0x1b400000065 - remove - buf - Volume Signal Exception: frame[1] =3D
> 0xffffffff830abf74=20
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 64: FS: vol1
> 0x1b400000065 - remove - buf - Volume Signal Exception: frame[2] =3D
> 0xffffffff83103508=20
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 65: FS: vol1
> 0x1b400000065 - remove - buf - Volume Signal Exception: frame[3] =3D
> 0xffffffff83105614=20
> May  9 16:56:27 eng60 : 1:2:efs:INFO: 72: FS: vol4
> 0x1b400000068
> - sm 0x0 - req - Forcibly unmounting volume
> May  9 16:56:27 eng60 : 0:0:eventd:INFO: close_session: Lost
> connection. Close rmc session 0x1008fc00 for event session 0x1008b608
> app unused 0. May  9 16:56:27 eng60 : 1:2:efs:INFO: 73: FS:
> vol3         0x1b400000067
> - sm 0x0 - req - Forcibly unmounting volume
> May  9 16:56:27 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol4', Id 0x000001b400000068, Event 'File System Abort', State 'Down'
> May  9 16:56:27 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:56:28 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol3', Id 0x000001b400000067, Event 'File System Abort', State 'Down'
> May  9 16:56:28 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:56:28 eng60 : 0:0:vsd:ERROR: vsd_mountVolProc : Abort mount
> operation for VS[2], req[4] > rsp[0]
> May  9 16:56:28 eng60 : 1:3:efs:INFO: 74: FS: vol2
> 0x1b400000066
> - sm 0x0 - req - Forcibly unmounting volume
> May  9 16:56:28 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol2', Id 0x000001b400000066, Event 'File System Abort', State 'Down'
> May  9 16:56:28 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:56:28 eng60 : 0:0:vsd:ERROR: vsd_getVolInfoReqProc : Failed
> to get status for volume 0x1b400000066
> May  9 16:56:28 eng60 last message repeated 2 times
> May  9 16:56:28 eng60 : 0:0:vsd:ERROR: vsd_getVolInfoReqProc : Failed
> to get status for volume 0x1b400000067
> May  9 16:56:28 eng60 last message repeated 3 times
> May  9 16:56:29 eng60 : 1:2:efs:INFO: 75: FS: vol1
> 0x1b400000065
> - sm 0x0 - req - Forcibly unmounting volume
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol1', Id 0x000001b400000065, Event 'File System Abort', State 'Down'
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol1', Id 0x000001b400000065, Event 'Create', State 'Down'
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol2', Id 0x000001b400000066, Event 'Create', State 'Down'
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol4', Id 0x000001b400000068, Event 'Create', State 'Down'
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol3', Id 0x000001b400000067, Event 'Create', State 'Down'
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:56:38 eng60 : 0:0:vsd:ERROR: vsd_sendMsg[9123] :
> rmc_send_msg_ex failed. rc=3D-3
> May  9 16:56:38 eng60 : 0:0:ea:INFO: ea_newAcptSess: creating new
> session
> May  9 16:56:38 eng60 : 0:0:ea:INFO: ea_openSess: open for
> {vsd:Agile1202.0.0}
> May  9 16:56:38 eng60 : 1:3:efs:INFO: 76: FS: vol1
> 0x1b400000065
> - mount - req - Volume is dirty, will check for log (devtype 2 logsize
> 62906816)
> May  9 16:56:38 eng60 : 1:3:efs:INFO: 77: FS: vol2
> 0x1b400000066
> - mount - req - Volume is dirty, will check for log (devtype 2 logsize
> 62906816)
> May  9 16:56:38 eng60 : 1:2:efs:INFO: 78: FS: vol3
> 0x1b400000067
> - mount - req - Volume is dirty, will check for log (devtype 2 logsize
> 62906816)
> May  9 16:56:38 eng60 : 1:2:efs:INFO: 79: FS: vol4
> 0x1b400000068
> - mount - req - Volume is dirty, will check for log (devtype 2 logsize
> 62906816)
> May  9 16:56:39 eng60 : 1:3:efs:INFO: 80: FS: vol4
> 0x1b400000068
> - mount - log - filesystem queued for replay
> May  9 16:56:39 eng60 : 1:2:efs:INFO: 81: FS: vol3
> 0x1b400000067
> - mount - log - filesystem queued for replay
> May  9 16:56:40 eng60 : 1:2:efs:INFO: 82: FS: vol1
> 0x1b400000065
> - mount - log - replay successfull head 25556048 tail 47947232 len
> 22391184 commits 64532
> May  9 16:56:40 eng60 : 1:2:efs:NOTICE: 83: FS: vol1
> 0x1b400000065 - mount - log - replay: syncing user data after log
> replay May  9 16:56:40 eng60 : 1:3:efs:INFO: 84: FS: vol2
> 0x1b400000066
> - mount - log - replay successfull head 28504776 tail 55285384 len
> 26780608 commits 97023
> May  9 16:56:40 eng60 : 1:3:efs:NOTICE: 85: FS: vol2
> 0x1b400000066 - mount - log - replay: syncing user data after log
> replay May  9 16:56:41 eng60 : 1:2:efs:INFO: 86: FS: vol4
> 0x1b400000068
> - mount - log - filesystem dequeued, starting replay
> May  9 16:56:41 eng60 : 1:2:efs:INFO: 87: FS: vol1
> 0x1b400000065
> - mount - req - zero-fill begin
> May  9 16:56:41 eng60 : 1:2:efs:INFO: 88: FS: vol3
> 0x1b400000067
> - mount - log - filesystem dequeued, starting replay
> May  9 16:56:41 eng60 : 1:3:efs:INFO: 89: FS: vol2
> 0x1b400000066
> - mount - req - zero-fill begin
> May  9 16:56:41 eng60 : 1:3:efs:INFO: 90: FS: vol1
> 0x1b400000065
> - mount - req - Volume online   version 28
> May  9 16:56:41 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol1', Id 0x000001b400000065, Event 'Online', was offline for roughly
> 24 sec.
> May  9 16:56:41 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:56:41 eng60 : 1:3:efs:INFO: 91: FS: vol2
> 0x1b400000066
> - mount - req - Volume online   version 28
> May  9 16:56:41 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol2', Id 0x000001b400000066, Event 'Online', was offline for roughly
> 25 sec.
> May  9 16:56:41 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:56:42 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:56:42 eng60 : 1:3:efs:INFO: 92: FS: vol4
> 0x1b400000068
> - mount - log - replay successfull head 54060624 tail 10011648 len
> 18857840 commits 98955
> May  9 16:56:42 eng60 : 1:3:efs:NOTICE: 93: FS: vol4
> 0x1b400000068 - mount - log - replay: syncing user data after log
> replay May  9 16:56:43 eng60 : 1:3:efs:INFO: 94: FS: vol4
> 0x1b400000068
> - mount - req - zero-fill begin
> May  9 16:56:43 eng60 : 1:3:efs:INFO: 95: FS: vol4
> 0x1b400000068
> - mount - req - Volume online   version 28
> May  9 16:56:43 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol4', Id 0x000001b400000068, Event 'Online', was offline for roughly
> 23 sec.
> May  9 16:56:43 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:56:43 eng60 : 1:3:efs:INFO: 96: FS: vol3
> 0x1b400000067
> - mount - log - replay successfull head 10812112 tail 35568600 len
> 24756488 commits 84685
> May  9 16:56:43 eng60 : 1:3:efs:NOTICE: 97: FS: vol3
> 0x1b400000067 - mount - log - replay: syncing user data after log
> replay May  9 16:56:43 eng60 : 1:2:efs:INFO: 98: FS: vol3
> 0x1b400000067
> - mount - req - zero-fill begin
> May  9 16:56:43 eng60 : 1:2:efs:INFO: 99: FS: vol3
> 0x1b400000067
> - mount - req - Volume online   version 28
> May  9 16:56:43 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol3', Id 0x000001b400000067, Event 'Online', was offline for roughly
> 24 sec.
> May  9 16:56:43 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:56:44 eng60 : 1:1:nfs:INFO: 16: Finished failover of NSM
> file after volume event on virtual server 2=20
> May  9 16:56:44 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 18229) terminated with signal
> Killed May  9 16:56:46 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization.=20
> May  9 16:57:09 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:57:11 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/evm_cfgd (pid 10777) terminated with signal
> Killed May  9 16:57:11 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/evm_cfgd: finished initialization.=20
> May  9 16:57:12 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open failed
> May  9 16:57:12 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open failed
> May  9 16:57:12 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:57:12 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:57:12 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open success
> May  9 16:57:12 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open success
> May  9 16:57:36 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:57:38 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/evm_cfgd (pid 718) terminated with signal Killed=20
> May  9 16:57:38 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/evm_cfgd:
> finished initialization.=20
> May  9 16:57:39 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open failed
> May  9 16:57:39 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open failed
> May  9 16:57:39 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:57:39 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:57:39 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open success
> May  9 16:57:39 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open success
> May  9 16:58:03 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:58:04 eng60 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
> before open completed
> May  9 16:58:04 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 25995) terminated with signal
> Killed May  9 16:58:06 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization.=20
> May  9 16:58:27 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol1', Id 0x000001b400000065, Event 'Create', State 'Down'
> May  9 16:58:27 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:58:27 eng60 : 0:0:vsd:ERROR: vsd_sendMsg[9123] :
> rmc_send_msg_ex failed. rc=3D-3
> May  9 16:58:27 eng60 : 0:0:vsd:ERROR: vsd_mountVolProc : Abort mount
> operation for VS[2], req[4] > rsp[1]
> May  9 16:58:30 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:58:31 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sscccc (pid 16877) terminated with signal Killed=20
> May  9 16:58:31 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/sscccc:
> finished initialization.=20
> May  9 16:58:31 eng60 : 0:0:unused:INFO: sscccc no
> /usr/local/agile/etc/sscccc_hosts_deny file
> May  9 16:58:37 eng60 : 0:0:vsd:ERROR: vsd_mountVolProc : Abort mount
> operation for VS[2], req[4] > rsp[1]
> May  9 16:58:57 eng60 last message repeated 2 times
> May  9 16:58:57 eng60 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
> before open completed
> May  9 16:58:57 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:58:58 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 10969) terminated with signal
> Killed May  9 16:59:00 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization.=20
> May  9 16:59:02 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol2', Id 0x000001b400000066, Event 'Create', State 'Down'
> May  9 16:59:02 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:59:02 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol3', Id 0x000001b400000067, Event 'Create', State 'Down'
> May  9 16:59:02 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:59:02 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol4', Id 0x000001b400000068, Event 'Create', State 'Down'
> May  9 16:59:02 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3=20
> May  9 16:59:07 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol1)
> current state - Mounted-RW State.
> May  9 16:59:07 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol1)
> already mounted in desired state.
> May  9 16:59:07 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol2)
> current state - Mounted-RW State.
> May  9 16:59:07 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol2)
> already mounted in desired state.
> May  9 16:59:07 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol3)
> current state - Mounted-RW State.
> May  9 16:59:07 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol3)
> already mounted in desired state.
> May  9 16:59:07 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol4)
> current state - Mounted-RW State.
> May  9 16:59:07 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol4)
> already mounted in desired state.
> May  9 16:59:24 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:59:25 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/crashsaved (pid 32099) terminated with signal
> Killed=20
> May  9 16:59:25 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/crashsaved:
> finished initialization.=20
> panic: malloc: out of space in kmem_map
> dumpsys: cannot open crashdump device: 16
>=20
> dumping to dev 411, offset 8
> dump not yet implementedSystem restart.
>=20
> PMON CPU 00000000 Initializing. Standby...
> EXCEPTIONPC=3D00000000 CONFIG=3D005264b0 STATUS=3D00400000
> Setting up SDRAM controllers
> R9K MEMORY SIZE=3D40000000
> SDM  REG=3D00000047
> SDT  REG=3D283f8482
> SDRAM Memory setup done.
> Clearing cache size memory...
> Cache init done.
> Initializing ECC memory.
> Decompression bss start.
> Init bss.
>=20
>=20
>=20
> PowerOn Self Test........OK
>=20
> Initializing System......please wait
>=20
> Initializing Shared Memory (1024 MB) for ECC
> Taking PCI out of reset.
> Waiting for FP/TXRX out of reset......
>=20
>=20
> pid=3D20929, daemon=3Dchassisd
> pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D22618, daemon=3Dvtmd
> pid=3D23456, daemon=3Dtape-driver
> pid=3D24682, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.96 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  2063 q9  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=3D28664, daemon=3Dtimekeeper
> pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D18, length=3D24, name=3Dvtmd
> kill -9 22618
> # Killing vtmd pid 22618
> ps ax | grep vtmd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 25766 q9  R+      0:00.00 grep vtmd (sh)
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:50:10 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.01 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=3D5022, daemon=3Dcluster_relay
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8085 ??  I       0:00.08 /usr/local/agile/bin/sendmail -Q30m -r
> pid=3D8039, daemon=3Dsdm_cfgd
> pid=3D8085, daemon=3Dsendmail
>  8202 ??  S       0:00.04 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.29 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> pid=3D8202, daemon=3Delog
> pid=3D8699, daemon=3Dsanmd
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D12658, daemon=3Dsnmpd
> pid=3D13027, daemon=3Dipmd
> pid=3D14473, daemon=3Deventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.07 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.47 /usr/local/agile/bin/sshd
> 18322 ??  S       0:00.95 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.32 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.09 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.11 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.18 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 23456 ??  S       0:00.14 /usr/local/agile/bin/tape-driver
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> pid=3D18322, daemon=3Dea
> pid=3D18816, daemon=3Dpm
> pid=3D19184, daemon=3Dsscccc
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D23456, daemon=3Dtape-driver
> pid=3D24682, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.96 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 18960 qa  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=3D28664, daemon=3Dtimekeeper
> pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D11, length=3D24, name=3Dsshd
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:50:37 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.01 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=3D5022, daemon=3Dcluster_relay
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8085 ??  I       0:00.08 /usr/local/agile/bin/sendmail -Q30m -r
> pid=3D8039, daemon=3Dsdm_cfgd
> pid=3D8085, daemon=3Dsendmail
>  8202 ??  S       0:00.04 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.29 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> pid=3D8202, daemon=3Delog
> pid=3D8699, daemon=3Dsanmd
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D12658, daemon=3Dsnmpd
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.07 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.47 /usr/local/agile/bin/sshd
> 18322 ??  S       0:00.95 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.32 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.09 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.11 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.18 /usr/local/agile/bin/chassisd -r
> pid=3D13027, daemon=3Dipmd
> pid=3D14473, daemon=3Deventd
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> pid=3D18322, daemon=3Dea
> pid=3D18816, daemon=3Dpm
> pid=3D19184, daemon=3Dsscccc
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> pid=3D21333, daemon=3Dspm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 23456 ??  S       0:00.14 /usr/local/agile/bin/tape-driver
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.96 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 28732 qb  S+      0:00.00 grep /usr/local/agile/bin
> # pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D23456, daemon=3Dtape-driver
> pid=3D24682, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D2, length=3D24, name=3Dsendmail
> kill -9 8085
> # Killing sendmail pid 8085
> ps ax | grep sendmail
>  8481 ??  S       0:00.08 /usr/local/agile/bin/sendmail -Q30m -r
> 16237 qb  R+      0:00.00 grep sendmail (sh)
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:50:59 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.02 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=3D5022, daemon=3Dcluster_relay
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8202 ??  S       0:00.04 /usr/local/agile/bin/elog -r
> pid=3D8039, daemon=3Dsdm_cfgd
> pid=3D8202, daemon=3Delog
>  8481 ??  I       0:00.08 /usr/local/agile/bin/sendmail -Q30m -r
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.29 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> pid=3D8481, daemon=3Dsendmail
> pid=3D8699, daemon=3Dsanmd
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D12658, daemon=3Dsnmpd
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.07 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.47 /usr/local/agile/bin/sshd
> 18322 ??  S       0:00.95 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.34 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.09 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.11 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.18 /usr/local/agile/bin/chassisd -r
> pid=3D13027, daemon=3Dipmd
> pid=3D14473, daemon=3Deventd
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> pid=3D18322, daemon=3Dea
> pid=3D18816, daemon=3Dpm
> pid=3D19184, daemon=3Dsscccc
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 23456 ??  S       0:00.14 /usr/local/agile/bin/tape-driver
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> # pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D23456, daemon=3Dtape-driver
> pid=3D24682, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D19, length=3D24, name=3Dtape-driver
> kill -9 23456
> # Killing tape-driver pid 23456
> ps ax | grep tape-driver
> 27313 ??  S       0:00.04 /usr/local/agile/bin/tape-driver -r
> 11863 qc  S+      0:00.00 grep tape-driver
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:51:26 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.02 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=3D5022, daemon=3Dcluster_relay
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8202 ??  S       0:00.07 /usr/local/agile/bin/elog -r
>  8481 ??  I       0:00.08 /usr/local/agile/bin/sendmail -Q30m -r
> pid=3D8039, daemon=3Dsdm_cfgd
> pid=3D8202, daemon=3Delog
> pid=3D8481, daemon=3Dsendmail
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.29 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.08 /usr/local/agile/bin/ncmd -r
> pid=3D8699, daemon=3Dsanmd
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D12658, daemon=3Dsnmpd
> pid=3D13027, daemon=3Dipmd
> pid=3D14473, daemon=3Deventd
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> 16585 ??  Ss      0:00.47 /usr/local/agile/bin/sshd
> 18322 ??  S       0:00.95 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.35 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.09 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.11 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> pid=3D16585, daemon=3Dsshd
> pid=3D18322, daemon=3Dea
> pid=3D18816, daemon=3Dpm
> pid=3D19184, daemon=3Dsscccc
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> 27313 ??  S       0:00.04 /usr/local/agile/bin/tape-driver -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 17481 qd  S+      0:00.00 grep /usr/local/agile/bin
> pid=3D24682, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> pid=3D27313, daemon=3Dtape-driver
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D32099, daemon=3Dcrashsaved
> #
> num=3D13, length=3D24, name=3Dpm
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:51:52 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.02 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=3D5022, daemon=3Dcluster_relay
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8202 ??  S       0:00.07 /usr/local/agile/bin/elog -r
> pid=3D8039, daemon=3Dsdm_cfgd
> pid=3D8202, daemon=3Delog
>  8481 ??  I       0:00.08 /usr/local/agile/bin/sendmail -Q30m -r
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.29 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> pid=3D8481, daemon=3Dsendmail
> pid=3D8699, daemon=3Dsanmd
> pid=3D10777, daemon=3Devm_cfgd
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.08 /usr/local/agile/bin/ncmd -r
> pid=3D12658, daemon=3Dsnmpd
> pid=3D13027, daemon=3Dipmd
> pid=3D14473, daemon=3Deventd
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> 16585 ??  Ss      0:00.48 /usr/local/agile/bin/sshd
> 18322 ??  S       0:00.95 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.36 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.09 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.11 /usr/local/agile/bin/vsd -r
> pid=3D16585, daemon=3Dsshd
> pid=3D18322, daemon=3Dea
> pid=3D18816, daemon=3Dpm
> pid=3D19184, daemon=3Dsscccc
> pid=3D20443, daemon=3Dvsd
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> 27313 ??  S       0:00.04 /usr/local/agile/bin/tape-driver -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 11499 qe  R+      0:00.00 grep /usr/local/agile/bin (sh)
> pid=3D20929, daemon=3Dchassisd
> pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24682, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> pid=3D27313, daemon=3Dtape-driver
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D32099, daemon=3Dcrashsaved
> #
> num=3D3, length=3D24, name=3Dsendmail
> kill -9 8481
> # Killing sendmail pid 8481
> ps ax | grep sendmail
> 18229 ??  S       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
>  1890 qe  S+      0:00.00 grep sendmail
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:52:14 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.02 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=3D5022, daemon=3Dcluster_relay
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8202 ??  S       0:00.12 /usr/local/agile/bin/elog -r
> pid=3D8039, daemon=3Dsdm_cfgd
> pid=3D8202, daemon=3Delog
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.29 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> pid=3D8699, daemon=3Dsanmd
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D12658, daemon=3Dsnmpd
> pid=3D13027, daemon=3Dipmd
> pid=3D14473, daemon=3Deventd
> pid=3D15806, daemon=3Dvtmd
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18322 ??  S       0:01.03 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.43 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.17 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.12 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> pid=3D18229, daemon=3Dsendmail
> pid=3D18322, daemon=3Dea
> pid=3D18816, daemon=3Dpm
> pid=3D19184, daemon=3Dsscccc
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24682, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> 27313 ??  S       0:00.04 /usr/local/agile/bin/tape-driver -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 10362 qf  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=3D27313, daemon=3Dtape-driver
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D21, length=3D24, name=3Dtape-driver
> kill -9 27313
> # Killing tape-driver pid 27313
> ps ax | grep tape-driver
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  6714 qf  R+      0:00.00 grep tape-driver (sh)
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:52:41 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.02 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=3D5022, daemon=3Dcluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8202 ??  S       0:00.12 /usr/local/agile/bin/elog -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8039, daemon=3Dsdm_cfgd
> pid=3D8202, daemon=3Delog
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> pid=3D8699, daemon=3Dsanmd
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D12658, daemon=3Dsnmpd
> pid=3D13027, daemon=3Dipmd
> pid=3D14473, daemon=3Deventd
> pid=3D15806, daemon=3Dvtmd
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18322 ??  S       0:01.03 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.45 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.17 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.12 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> pid=3D18229, daemon=3Dsendmail
> pid=3D18322, daemon=3Dea
> pid=3D18816, daemon=3Dpm
> pid=3D19184, daemon=3Dsscccc
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24682, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 29675 r0  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D15, length=3D24, name=3Dsscccc
> kill -9 19184
> # Killing sscccc pid 19184
> ps ax | grep sscccc
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:53:08 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.02 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=3D5022, daemon=3Dcluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8039, daemon=3Dsdm_cfgd
>  8202 ??  S       0:00.12 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.20 /usr/local/agile/bin/ipmd -r
> pid=3D8202, daemon=3Delog
> pid=3D8699, daemon=3Dsanmd
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D12658, daemon=3Dsnmpd
> pid=3D13027, daemon=3Dipmd
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> pid=3D14473, daemon=3Deventd
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> pid=3D16877, daemon=3Dsscccc
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18322 ??  S       0:01.03 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.49 /usr/local/agile/bin/pm
> 20443 ??  S       0:00.12 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> pid=3D18229, daemon=3Dsendmail
> pid=3D18322, daemon=3Dea
> pid=3D18816, daemon=3Dpm
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>   177 r1  R+      0:00.00 grep /usr/local/agile/bin
> pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24682, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D32099, daemon=3Dcrashsaved
> #
> num=3D8, length=3D24, name=3Deventd
> kill -9 14473
> # Killing eventd pid 14473
> ps ax | grep eventd
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> 27221 r1  S+      0:00.00 grep eventd
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:53:34 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.03 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=3D5022, daemon=3Dcluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8039, daemon=3Dsdm_cfgd
>  8202 ??  S       0:00.13 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> pid=3D8202, daemon=3Delog
> pid=3D8699, daemon=3Dsanmd
> pid=3D9368, daemon=3Deventd
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D12658, daemon=3Dsnmpd
> 13027 ??  S       0:00.23 /usr/local/agile/bin/ipmd -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> pid=3D13027, daemon=3Dipmd
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> pid=3D16877, daemon=3Dsscccc
> pid=3D18229, daemon=3Dsendmail
> 18322 ??  S       0:01.03 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.55 /usr/local/agile/bin/pm
> 20443 ??  S       0:00.12 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.06 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 30872 r2  R+      0:00.00 grep /usr/local/agile/bin (sh)
> pid=3D18322, daemon=3Dea
> pid=3D18816, daemon=3Dpm
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24682, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D32099, daemon=3Dcrashsaved
> #
> num=3D2, length=3D24, name=3Dsdm_cfgd
> kill -9 8039
> # Killing sdm_cfgd pid 8039
> ps ax | grep sdm_cfgd
>  1343 ??  S       0:00.12 /usr/local/agile/bin/sdm_cfgd -r
> 30194 r2  R+      0:00.00 grep sdm_cfgd (sh)
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:54:01 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.12 /usr/local/agile/bin/sdm_cfgd -r
> pid=3D1343, daemon=3Dsdm_cfgd
>  2604 ??  S<      0:01.03 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=3D5022, daemon=3Dcluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.15 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8202, daemon=3Delog
> pid=3D8699, daemon=3Dsanmd
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.26 /usr/local/agile/bin/ipmd -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18322 ??  S       0:01.03 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.58 /usr/local/agile/bin/pm
> 20443 ??  S       0:00.12 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> pid=3D9368, daemon=3Deventd
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D12658, daemon=3Dsnmpd
> pid=3D13027, daemon=3Dipmd
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> pid=3D16877, daemon=3Dsscccc
> pid=3D18229, daemon=3Dsendmail
> pid=3D18322, daemon=3Dea
> pid=3D18816, daemon=3Dpm
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> 24682 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.06 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.98 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  9766 r3  S+      0:00.00 grep /usr/local/agile/bin
> # pid=3D24682, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D20, length=3D24, name=3Dauth-agent
> kill -9 24682
> # Killing auth-agent pid 24682
> ps ax | grep auth-agent
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
>  9278 r3  R+      0:00.00 grep auth-agent (sh)
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:54:28 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.13 /usr/local/agile/bin/sdm_cfgd -r
> pid=3D1343, daemon=3Dsdm_cfgd
>  2604 ??  S<      0:01.03 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=3D5022, daemon=3Dcluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.16 /usr/local/agile/bin/elog -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8202, daemon=3Delog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.28 /usr/local/agile/bin/ipmd -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> pid=3D8699, daemon=3Dsanmd
> pid=3D9368, daemon=3Deventd
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D12658, daemon=3Dsnmpd
> pid=3D13027, daemon=3Dipmd
> pid=3D15806, daemon=3Dvtmd
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18322 ??  S       0:01.03 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.64 /usr/local/agile/bin/pm
> 20443 ??  S       0:00.12 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.18 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> pid=3D16877, daemon=3Dsscccc
> pid=3D18229, daemon=3Dsendmail
> pid=3D18322, daemon=3Dea
> pid=3D18816, daemon=3Dpm
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24910, daemon=3Dauth-agent
> 26301 ??  S       0:00.06 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> 28773 ??  S<      0:00.99 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  9784 r4  R       0:00.00 grep /usr/local/agile/bin (sh)
> # pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D14, length=3D24, name=3Dea
> kill -9 18322
> # Killing ea pid 18322
> ps ax | grep ea
> 14882 ??  S       0:00.07 /usr/local/agile/bin/ea -r
> 10271 r4  R+      0:00.00 grep ea (sh)
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:54:55 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.13 /usr/local/agile/bin/sdm_cfgd -r
> pid=3D1343, daemon=3Dsdm_cfgd
>  2604 ??  S<      0:01.04 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=3D5022, daemon=3Dcluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.17 /usr/local/agile/bin/elog -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8202, daemon=3Delog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> pid=3D8699, daemon=3Dsanmd
> pid=3D9368, daemon=3Deventd
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D12658, daemon=3Dsnmpd
> 13027 ??  S       0:00.31 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.07 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18816 ??  Ss      0:03.69 /usr/local/agile/bin/pm
> 20443 ??  S       0:00.13 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> pid=3D13027, daemon=3Dipmd
> pid=3D14882, daemon=3Dea
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> pid=3D16877, daemon=3Dsscccc
> pid=3D18229, daemon=3Dsendmail
> pid=3D18816, daemon=3Dpm
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> 21333 ??  S       0:01.18 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.06 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.99 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 16918 r5  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24910, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D7, length=3D24, name=3Dsnmpd
> kill -9 12658
> # Killing snmpd pid 12658
> ps ax | grep snmpd
> 19586 ??  S       0:00.10 /usr/local/agile/bin/snmpd -f -r
>  5402 r5  S+      0:00.00 grep snmpd
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:55:22 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.13 /usr/local/agile/bin/sdm_cfgd -r
> pid=3D1343, daemon=3Dsdm_cfgd
>  2604 ??  S<      0:01.04 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=3D5022, daemon=3Dcluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.17 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8202, daemon=3Delog
> pid=3D8699, daemon=3Dsanmd
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> pid=3D9368, daemon=3Deventd
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 13027 ??  S       0:00.34 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.07 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D13027, daemon=3Dipmd
> pid=3D14882, daemon=3Dea
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18816 ??  Ss      0:03.73 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.10 /usr/local/agile/bin/snmpd -f -r
> pid=3D16585, daemon=3Dsshd
> pid=3D16877, daemon=3Dsscccc
> pid=3D18229, daemon=3Dsendmail
> pid=3D18816, daemon=3Dpm
> pid=3D19586, daemon=3Dsnmpd
> 20443 ??  S       0:00.13 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.18 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.06 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24910, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> 28773 ??  S<      0:00.99 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  1967 r6  S+      0:00.00 grep /usr/local/agile/bin
> # pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D1, length=3D24, name=3Dcluster_relay
> kill -9 5022
> # Killing cluster_relay pid 5022
> ps ax | grep cluster_relay
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
> 18374 r6  S+      0:00.00 grep cluster_relay
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:55:48 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.13 /usr/local/agile/bin/sdm_cfgd -r
> pid=3D1343, daemon=3Dsdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
>  2604 ??  S<      0:01.04 /usr/local/agile/bin/cluster_server
> pid=3D2071, daemon=3Dcluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.17 /usr/local/agile/bin/elog -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8202, daemon=3Delog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 13027 ??  S       0:00.36 /usr/local/agile/bin/ipmd -r
> pid=3D8699, daemon=3Dsanmd
> pid=3D9368, daemon=3Deventd
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D13027, daemon=3Dipmd
> 14882 ??  S       0:00.07 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> pid=3D14882, daemon=3Dea
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> pid=3D16877, daemon=3Dsscccc
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18816 ??  Ss      0:03.75 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.10 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:00.14 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.19 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.06 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.00 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 19561 r7  S+      0:00.00 grep /usr/local/agile/bin
> # pid=3D18229, daemon=3Dsendmail
> pid=3D18816, daemon=3Dpm
> pid=3D19586, daemon=3Dsnmpd
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21333, daemon=3Dspm
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24910, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D18, length=3D24, name=3Dspm
> kill -9 21333
> # Killing spm pid 21333
> ps ax | grep spm
> 31173 ??  S       0:00.06 /usr/local/agile/bin/spm -r
>  4901 r7  R+      0:00.00 grep spm (sh)
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:56:15 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.43 /usr/local/agile/bin/sdm_cfgd -r
> pid=3D1343, daemon=3Dsdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
>  2604 ??  D<      0:01.06 /usr/local/agile/bin/cluster_server
> pid=3D2071, daemon=3Dcluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.50 /usr/local/agile/bin/elog -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8202, daemon=3Delog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.19 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.49 /usr/local/agile/bin/evm_cfgd -r
> pid=3D8699, daemon=3Dsanmd
> pid=3D9368, daemon=3Deventd
> pid=3D10777, daemon=3Devm_cfgd
> 13027 ??  S       0:00.39 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.41 /usr/local/agile/bin/ncmd -r
> pid=3D13027, daemon=3Dipmd
> pid=3D14882, daemon=3Dea
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18816 ??  Ss      0:03.79 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.16 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.91 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> pid=3D16585, daemon=3Dsshd
> pid=3D16877, daemon=3Dsscccc
> pid=3D18229, daemon=3Dsendmail
> pid=3D18816, daemon=3Dpm
> pid=3D19586, daemon=3Dsnmpd
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> 21489 ??  S       0:00.07 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.08 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.12 /usr/local/agile/bin/cluster_contrl
> 31173 ??  S       0:00.25 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  4052 r8  R+      0:00.01 grep /usr/local/agile/bin
> # pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24910, daemon=3Dauth-agent
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D31173, daemon=3Dspm
> pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D13, length=3D24, name=3Dsendmail
> kill -9 18229
> # Killing sendmail pid 18229
> ps ax | grep sendmail
> 25995 ??  S       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
>  3852 r8  R+      0:00.00 grep sendmail (sh)
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:56:42 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.43 /usr/local/agile/bin/sdm_cfgd -r
> pid=3D1343, daemon=3Dsdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
>  2604 ??  S<      0:01.07 /usr/local/agile/bin/cluster_server
> pid=3D2071, daemon=3Dcluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.52 /usr/local/agile/bin/elog -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8202, daemon=3Delog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.19 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.49 /usr/local/agile/bin/evm_cfgd -r
> 13027 ??  S       0:00.40 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> pid=3D8699, daemon=3Dsanmd
> pid=3D9368, daemon=3Deventd
> pid=3D10777, daemon=3Devm_cfgd
> pid=3D13027, daemon=3Dipmd
> pid=3D14882, daemon=3Dea
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.41 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18816 ??  Ss      0:03.84 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.16 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.95 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> pid=3D16877, daemon=3Dsscccc
> pid=3D18816, daemon=3Dpm
> pid=3D19586, daemon=3Dsnmpd
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> 21489 ??  S       0:00.07 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 25995 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.08 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24910, daemon=3Dauth-agent
> pid=3D25995, daemon=3Dsendmail
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> 28773 ??  S<      0:01.18 /usr/local/agile/bin/cluster_contrl
> 31173 ??  S       0:00.25 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  9307 r9  S+      0:00.00 grep /usr/local/agile/bin
> # pid=3D31173, daemon=3Dspm
> pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D6, length=3D24, name=3Devm_cfgd
> kill -9 10777
> # Killing evm_cfgd pid 10777
> ps ax | grep evm_cfgd
>   718 ??  S       0:00.07 /usr/local/agile/bin/evm_cfgd -r
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:57:09 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>   718 ??  S       0:00.07 /usr/local/agile/bin/evm_cfgd -r
> pid=3D718, daemon=3Devm_cfgd
>  1343 ??  S       0:00.43 /usr/local/agile/bin/sdm_cfgd -r
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
> pid=3D1343, daemon=3Dsdm_cfgd
> pid=3D2071, daemon=3Dcluster_relay
>  2604 ??  S<      0:01.07 /usr/local/agile/bin/cluster_server
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
> pid=3D6146, daemon=3Dtape-driver
>  8202 ??  S       0:00.56 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.19 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 13027 ??  S       0:00.40 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.07 /usr/local/agile/bin/vtmd -r
> pid=3D8202, daemon=3Delog
> pid=3D8699, daemon=3Dsanmd
> pid=3D9368, daemon=3Deventd
> pid=3D13027, daemon=3Dipmd
> pid=3D14882, daemon=3Dea
> pid=3D15806, daemon=3Dvtmd
> 15926 ??  S       0:00.49 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.50 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18816 ??  Ss      0:03.87 /usr/local/agile/bin/pm
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> pid=3D16877, daemon=3Dsscccc
> pid=3D18816, daemon=3Dpm
> 19586 ??  S       0:00.16 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.95 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21489 ??  S       0:00.07 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 25995 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.09 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.18 /usr/local/agile/bin/cluster_contrl
> 31173 ??  S       0:00.28 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  7912 ra  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=3D19586, daemon=3Dsnmpd
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24910, daemon=3Dauth-agent
> pid=3D25995, daemon=3Dsendmail
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D31173, daemon=3Dspm
> pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D0, length=3D24, name=3Devm_cfgd
> kill -9 718
> # Killing evm_cfgd pid 718
> ps ax | grep evm_cfgd
> 26078 ??  S       0:00.10 /usr/local/agile/bin/evm_cfgd -r
>  1763 ra  R+      0:00.00 grep evm_cfgd (sh)
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:57:36 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.43 /usr/local/agile/bin/sdm_cfgd -r
> pid=3D1343, daemon=3Dsdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
> pid=3D2071, daemon=3Dcluster_relay
>  2604 ??  S<      0:01.08 /usr/local/agile/bin/cluster_server
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.57 /usr/local/agile/bin/elog -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8202, daemon=3Delog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.19 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 13027 ??  S       0:00.40 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> pid=3D8699, daemon=3Dsanmd
> pid=3D9368, daemon=3Deventd
> pid=3D13027, daemon=3Dipmd
> pid=3D14882, daemon=3Dea
> 15806 ??  S       0:00.07 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.49 /usr/local/agile/bin/ncmd -r
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> 16585 ??  Ss      0:00.51 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18816 ??  Ss      0:03.93 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.16 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.95 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21489 ??  S       0:00.07 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 25995 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26078 ??  S       0:00.10 /usr/local/agile/bin/evm_cfgd -r
> pid=3D16585, daemon=3Dsshd
> pid=3D16877, daemon=3Dsscccc
> pid=3D18816, daemon=3Dpm
> pid=3D19586, daemon=3Dsnmpd
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24910, daemon=3Dauth-agent
> pid=3D25995, daemon=3Dsendmail
> pid=3D26078, daemon=3Devm_cfgd
> 26301 ??  S       0:00.09 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.18 /usr/local/agile/bin/cluster_contrl
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> 31173 ??  S       0:00.28 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 17773 rb  R+      0:00.01 grep /usr/local/agile/bin
> # pid=3D31173, daemon=3Dspm
> pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D18, length=3D24, name=3Dsendmail
> kill -9 25995
> # Killing sendmail pid 25995
> ps ax | grep sendmail
> 10969 ??  S       0:00.12 /usr/local/agile/bin/sendmail -Q30m -r
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:58:03 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.44 /usr/local/agile/bin/sdm_cfgd -r
> pid=3D1343, daemon=3Dsdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
>  2604 ??  S<      0:01.08 /usr/local/agile/bin/cluster_server
> pid=3D2071, daemon=3Dcluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.58 /usr/local/agile/bin/elog -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8202, daemon=3Delog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.20 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10969 ??  I       0:00.12 /usr/local/agile/bin/sendmail -Q30m -r
> 13027 ??  S       0:00.40 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> pid=3D8699, daemon=3Dsanmd
> pid=3D9368, daemon=3Deventd
> pid=3D10969, daemon=3Dsendmail
> pid=3D13027, daemon=3Dipmd
> pid=3D14882, daemon=3Dea
> 15806 ??  S       0:00.07 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.57 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.51 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18816 ??  Ss      0:03.98 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.17 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.95 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21489 ??  S       0:00.08 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26078 ??  S       0:00.12 /usr/local/agile/bin/evm_cfgd -r
> 26301 ??  S       0:00.09 /usr/local/agile/bin/asd -r
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> pid=3D16585, daemon=3Dsshd
> pid=3D16877, daemon=3Dsscccc
> pid=3D18816, daemon=3Dpm
> pid=3D19586, daemon=3Dsnmpd
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24910, daemon=3Dauth-agent
> pid=3D26078, daemon=3Devm_cfgd
> pid=3D26301, daemon=3Dasd
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.19 /usr/local/agile/bin/cluster_contrl
> 31173 ??  S       0:00.30 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  2061 rc  R+      0:00.00 grep /usr/local/agile/bin (sh)
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D31173, daemon=3Dspm
> pid=3D32099, daemon=3Dcrashsaved
> #
> num=3D12, length=3D24, name=3Dsscccc
> kill -9 16877
> # Killing sscccc pid 16877
> ps ax | grep sscccc
> 18080 ??  S       0:00.10 /usr/local/agile/bin/sscccc -r
> 20149 rc  R+      0:00.00 grep sscccc (sh)
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:58:30 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.44 /usr/local/agile/bin/sdm_cfgd -r
> pid=3D1343, daemon=3Dsdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
>  2604 ??  S<      0:01.08 /usr/local/agile/bin/cluster_server
> pid=3D2071, daemon=3Dcluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.59 /usr/local/agile/bin/elog -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8202, daemon=3Delog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.20 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10969 ??  I       0:00.12 /usr/local/agile/bin/sendmail -Q30m -r
> 13027 ??  S       0:00.40 /usr/local/agile/bin/ipmd -r
> pid=3D8699, daemon=3Dsanmd
> pid=3D9368, daemon=3Deventd
> pid=3D10969, daemon=3Dsendmail
> pid=3D13027, daemon=3Dipmd
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.07 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.57 /usr/local/agile/bin/ncmd -r
> pid=3D14882, daemon=3Dea
> pid=3D15806, daemon=3Dvtmd
> pid=3D15926, daemon=3Dncmd
> 16585 ??  Ss      0:00.52 /usr/local/agile/bin/sshd
> 18080 ??  S       0:00.10 /usr/local/agile/bin/sscccc -r
> 18816 ??  Ss      0:04.01 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.17 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.97 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21489 ??  S       0:00.08 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26078 ??  S       0:00.12 /usr/local/agile/bin/evm_cfgd -r
> 26301 ??  S       0:00.09 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.19 /usr/local/agile/bin/cluster_contrl
> 31173 ??  S       0:00.30 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>   205 rd  R+      0:00.00 grep /usr/local/agile/bin (sh)
> pid=3D16585, daemon=3Dsshd
> pid=3D18080, daemon=3Dsscccc
> pid=3D18816, daemon=3Dpm
> pid=3D19586, daemon=3Dsnmpd
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24910, daemon=3Dauth-agent
> pid=3D26078, daemon=3Devm_cfgd
> pid=3D26301, daemon=3Dasd
> pid=3D28664, daemon=3Dtimekeeper
> pid=3D31173, daemon=3Dspm
> pid=3D32099, daemon=3Dcrashsaved
> #
> num=3D6, length=3D24, name=3Dsendmail
> kill -9 10969
> # Killing sendmail pid 10969
> ps ax | grep sendmail
> 11605 ??  S       0:00.09 /usr/local/agile/bin/sendmail -Q30m -r
> ***** Result is true
>=20
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:58:57 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
>=20
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.76 /usr/local/agile/bin/sdm_cfgd -r
> pid=3D1343, daemon=3Dsdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
> pid=3D2071, daemon=3Dcluster_relay
>  2604 ??  S<      0:01.10 /usr/local/agile/bin/cluster_server
>  6146 ??  S       0:00.06 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.65 /usr/local/agile/bin/elog -r
> pid=3D6146, daemon=3Dtape-driver
> pid=3D8202, daemon=3Delog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.23 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> pid=3D8699, daemon=3Dsanmd
> pid=3D9368, daemon=3Deventd
> 11605 ??  I       0:00.09 /usr/local/agile/bin/sendmail -Q30m -r
> 13027 ??  S       0:00.40 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.07 /usr/local/agile/bin/vtmd -r
> pid=3D11605, daemon=3Dsendmail
> pid=3D13027, daemon=3Dipmd
> pid=3D14882, daemon=3Dea
> pid=3D15806, daemon=3Dvtmd
> 15926 ??  S       0:00.79 /usr/local/agile/bin/ncmd -r
> pid=3D15926, daemon=3Dncmd
> 16585 ??  Ss      0:00.52 /usr/local/agile/bin/sshd
> 18080 ??  S       0:00.10 /usr/local/agile/bin/sscccc -r
> 18816 ??  Ss      0:04.09 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.17 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.98 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21489 ??  S       0:00.09 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> pid=3D16585, daemon=3Dsshd
> pid=3D18080, daemon=3Dsscccc
> pid=3D18816, daemon=3Dpm
> pid=3D19586, daemon=3Dsnmpd
> pid=3D20443, daemon=3Dvsd
> pid=3D20929, daemon=3Dchassisd
> pid=3D21489, daemon=3Dndmp_cfgd
> pid=3D24910, daemon=3Dauth-agent
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26078 ??  S       0:00.42 /usr/local/agile/bin/evm_cfgd -r
> 26301 ??  S       0:00.09 /usr/local/agile/bin/asd -r
> pid=3D26078, daemon=3Devm_cfgd
> pid=3D26301, daemon=3Dasd
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.25 /usr/local/agile/bin/cluster_contrl
> 31173 ??  S       0:00.44 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 22881 re  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=3D28664, daemon=3Dtimekeeper
> pid=3D31173, daemon=3Dspm
> pid=3D32099, daemon=3Dcrashsaved
>=20
> num=3D23, length=3D24, name=3Dcrashsaved
> kill -9 32099
> # Killing crashsaved pid 32099
> ***** Result is false
>=20
> >>> Result is false
>=20
> [sandrineb@sandrineb cluster]$
>=20
>=20
>=20
> Release_Project: Delorean
>=20
>=20
