AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20070511172154.2ea1168f@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<mike.lee@onstor.com>,<rendell.fong@onstor.com>,<brian.deforest@onstor.com>,<sandrine.boulanger@onstor.com>,<james.kahn@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#mh/Mailbox/clearcrap	0	BB375AF679D4A34E9CA8DFA650E2B04E030E342C@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Fri, 11 May 2007 17:22:31 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: "Mike Lee" <mike.lee@onstor.com>
Cc: "Rendell Fong" <rendell.fong@onstor.com>, "Brian DeForest"
 <brian.deforest@onstor.com>, "Sandrine Boulanger"
 <sandrine.boulanger@onstor.com>, "James Kahn" <james.kahn@onstor.com>
Subject: Re: Defect  TED00018513 bsd panic (out of space in kmem_map) when
 killing ssc daemons randomly on Bobcat Assigned
Message-ID: <20070511172231.1848058a@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E030E342C@onstor-exch02.onstor.net>
References: <BB375AF679D4A34E9CA8DFA650E2B04E03B1BD88@onstor-exch02.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E030E342C@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

On Fri, 11 May 2007 17:09:14 -0700 "Mike Lee" <mike.lee@onstor.com>
wrote:


> -----Original Message-----
> From: Maxim Kozlovsky 
> Sent: Friday, May 11, 2007 10:49 AM
> To: Mike Lee; Rendell Fong; Sandrine Boulanger; Brian DeForest
> Cc: James Kahn
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
> 
> It will be better if you fix this problem first before running the qa
> scripts. This is an obvious kernel memory leak, don't try to find
> another one without fixing the known problem.
> 
> (The best way to go about fixing the ssccc is to move the mounting of
> the ramdisk to the rc.agile so it is done just once).


Please do this as the fix.  Not being able to find where this is mounted
[by searching the rc scripts] is a real nightmare for systems guys.

Thanks.


> -----Original Message-----
> From: Mike Lee 
> Sent: Friday, May 11, 2007 10:45 AM
> To: Rendell Fong; Sandrine Boulanger; Brian DeForest
> Cc: James Kahn; Maxim Kozlovsky
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
> 
> Randell:
> 
> Thanks for the insight. 
> I only killed ssccc once last night, and looking at my filer, your
> observation is correct, as I have two /tmp/ramdisk mount points.  
> 
> I plan to just use the QA scripts today.
> If time permits, I will also try your technique.
> 
> -Mike
> -----Original Message-----
> From: Rendell Fong 
> Sent: Friday, May 11, 2007 10:08 AM
> To: Mike Lee; Sandrine Boulanger; Brian DeForest
> Cc: James Kahn; Maxim Kozlovsky
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
> 
> 
> I don't know if this is related but I ran a test that constantly kills
> off the sscccc process and lets the pm restart it.  When it restarts
> it always creates a new mount_mfs process and creates a mounted fs for
> /tmp/ramdisk.  They seem to linger even after sscccc is killed off and
> have a cumulative effect.
> 
> See enclosed.
> 
> Rendell
> 
> 
> -----Original Message-----
> From: Mike Lee 
> Sent: Wednesday, May 09, 2007 11:14 PM
> To: Sandrine Boulanger; Brian DeForest
> Cc: James Kahn; Maxim Kozlovsky; Rendell Fong
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
> 
> 
> Sandrine:
> 
> Thank you for the clarification; I'll try again tomorrow, and perhaps
> I'll need to enlist your help.
> 
> -Mike
> 
> -----Original Message-----
> From: Sandrine Boulanger
> Sent: Wed 5/9/2007 7:26 PM
> To: Mike Lee; Brian DeForest
> Cc: James Kahn; Maxim Kozlovsky; Rendell Fong
> Subject: RE: Defect  TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
>  
> I don't think killing a particular daemon is going to reproduce it. I
> think there could be a memory leak when we restart any daemon that
> will eventually lead to the bsd crashing.
> 
> To reproduce is very simple, just follow the steps to reproduce and
> you're there, that's what I did again today. 
> 
> -----Original Message-----
> From: Mike Lee 
> Sent: Wednesday, May 09, 2007 6:43 PM
> To: Brian DeForest
> Cc: James Kahn; Maxim Kozlovsky; Rendell Fong; Sandrine Boulanger
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
> 
> Brian:
> It sure looks like that the kill must be done at an opportune time,
> based on the message entries you captured below.
> So, I guess we can easily ask Sandrine to try and kill crashsaved
> again. I will try on my filer as well tomorrow.
> THanks.
> -Mike
> 
> -----Original Message-----
> From: Brian DeForest 
> Sent: Wednesday, May 09, 2007 6:32 PM
> To: Mike Lee
> Cc: James Kahn; Maxim Kozlovsky; Rendell Fong
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
> 
> 
> Thanks Mike.   Considering you weren't able to reproduce originally,
> we probably need to look closely at how she reproduced it. Looks like
> killing crashsaved did it.
> 
> Could also be that /var is 100% full, which could be tested easily.
> 
> May  9 16:59:24 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:59:25 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/crashsaved (pid 32099) terminated with signal
> Killed 
> May  9 16:59:25 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/crashsaved:
> finished initialization. 
> panic: malloc: out of space in kmem_map
> dumpsys: cannot open crashdump device: 16
> 
> dumping to dev 411, offset 8
> dump not yet implementedSystem restart.
> 
> PMON CPU 00000000 Initializing. Standby...
> 
> -----Original Message-----
> From: Mike Lee 
> Sent: Wednesday, May 09, 2007 6:15 PM
> To: Brian DeForest
> Cc: Sandrine Boulanger; James Kahn
> Subject: RE: Defect TED00018513 bsd panic (out of space in kmem_map)
> when killing ssc daemons randomly on Bobcat Assigned
> 
> Brian:
> 
> I can try to repeat Sandrine's experiment on top of tree tomorrow, if
> you think it is worthwhile.
> Or perhaps we want to wait for Jim's check in?
> 
> Pls let me know.
> Thnx.
> 
> -Mike
> 
> -----Original Message-----
> From: sandrine.boulanger@onstor.com
> [mailto:sandrine.boulanger@onstor.com]
> Sent: Wednesday, May 09, 2007 6:07 PM
> To: Mike Lee
> Cc: Sandrine Boulanger; Sandrine Boulanger; Brian DeForest
> Subject: Defect TED00018513 bsd panic (out of space in kmem_map) when
> killing ssc daemons randomly on Bobcat Assigned
> 
> 
> id: TED00018513
> Headline: bsd panic (out of space in kmem_map) when killing ssc
> daemons randomly on Bobcat
> Note_Entry: I reproduced it again in sub#15:
> 
> Wed May  9 16:16:36 PDT 2007
> 
> login: May  9 16:38:27 eng60 : 0:0:nfxsh:NOTICE: cmd[5]: elog display
> ena : status[0]
> May  9 16:38:30 eng60 : 0:0:nfxsh:NOTICE: cmd[6]: elog show config  :
> status[0]
> May  9 16:38:36 eng60 : 1:0:e_log:INFO: 15: e-log configure OK, cmd
> 9249 6 0 0
> May  9 16:38:36 eng60 : 2:0:e_log:INFO: 30: e-log configure OK, cmd
> 9249 6 0 0
> May  9 16:38:36 eng60 : 1:2:e_log:INFO: 22: e-log configure OK, cmd
> 9249 6 0 0
> May  9 16:38:36 eng60 : 2:0:e_log:INFO: 31: e-log configure OK, cmd
> 9249 6 0 0
> May  9 16:38:36 eng60 : 2:0:e_log:INFO: 32: e-log configure OK, cmd
> 9249 6 0 0
> May  9 16:38:36 eng60 : 2:0:e_log:INFO: 33: e-log configure OK, cmd
> 9249 6 0 0
> May  9 16:38:36 eng60 : 0:0:nfxsh:NOTICE: cmd[7]: elog level info :
> status[0]
> May  9 16:38:37 eng60 : 0:0:nfxsh:NOTICE: cmd[8]: elog show config  :
> status[0]
> May  9 16:39:30 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:39:32 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/vtmd (pid 8540) terminated with signal Killed 
> May  9 16:39:33 eng60 : 0:0:cluster2:INFO: ClusterCtrl_releaseFiler:
> called by vtm 
> May  9 16:39:33 eng60 : 0:0:vtm:INFO: vtm_sendCardStateMsg: Sending
> card UP to eng60 
> May  9 16:39:33 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/vtmd:
> finished initialization. 
> May  9 16:39:52 eng60 : 0:0:vtm:INFO: Vtm_ProcCardStateMsg: card state
> UP message from eng60 
> May  9 16:39:57 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:39:58 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/ncmd (pid 27267) terminated with signal Killed 
> May  9 16:39:59 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:40:00 eng60 : 0:0:cluster2:ERROR: main: rcv ncm msg  
> May  9 16:40:00 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open failed
> May  9 16:40:00 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:40:00 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/ncmd:
> finished initialization. 
> May  9 16:40:01 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open failed
> May  9 16:40:01 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:40:01 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open success
> May  9 16:40:04 eng60 : 0:0:ea:NOTICE: ea_closeSess : NCM session
> closed May  9 16:40:04 eng60 : 0:0:ea:NOTICE: ea_rcvRmcMsg : NCM
> session open success
> May  9 16:40:24 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:40:25 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/evm_cfgd (pid 18999) terminated with signal
> Killed May  9 16:40:25 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/evm_cfgd: finished initialization. 
> May  9 16:40:27 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open failed
> May  9 16:40:27 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open failed
> May  9 16:40:27 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:40:27 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:40:27 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open success
> May  9 16:40:27 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open success
> May  9 16:40:50 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:40:52 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sdm_cfgd (pid 12313) terminated with signal
> Killed May  9 16:40:52 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sdm_cfgd: finished initialization. 
> May  9 16:40:52 eng60 : 0:0:sdm:INFO: ONStor Storage Device Manager
> (c)2003: Started
> May  9 16:40:52 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:40:53 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open failed
> May  9 16:40:53 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:40:53 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open success
> May  9 16:40:53 eng60 : 0:0:sdm:INFO: sdm_rcvRmcMsg : RMC open
> successful (0)
> May  9 16:40:53 eng60 : 0:0:sdm:INFO: sdm_rcvRmcMsg : RMC open
> successful (0)
> May  9 16:40:53 eng60 : 0:0:sdm:INFO: sdm_rcvRmcMsg : RMC Open to FC
> Port successful {sdm:Agile2775}->{:scsi2.2.0}
> May  9 16:40:53 eng60 : 2:0:scsi:INFO: 34: scsi_openSessToSDM:RMC
> rc[-9] msg[0x0]
> May  9 16:40:53 eng60 : 2:0:scsi:INFO: 35: scsi_openSessToSDM: RMC
> session found open at [0x60e51010], issue close for session. tagVal =
> [40c]  
> May  9 16:40:53 eng60 : 2:0:scsi:INFO: 36: scsi_openSessToSDM:
> scsiCtrl->sdmOpenSession = 0x60e51010, and &scsiCtrl->sdmOpenSession =
> 0x623a98bc  
> May  9 16:40:54 eng60 : 0:0:tape-driver:ERROR: tape_rpc: rmc_rpc app
> sdm cpu 0 slot 0 failed - -3
> May  9 16:41:03 eng60 : 0:0:spm:WARNING: spm_sendNcmMsg: ncm_send()
> failed - -20
> May  9 16:41:03 eng60 : 0:0:evm:NOTICE: evm_getTxn : transaction with
> handle[0xe9020000] already freed
> May  9 16:41:03 eng60 : 0:0:evm:ERROR: evm_processMsg : Stale SPM
> response[2607]
> May  9 16:41:03 eng60 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
> before open completed
> May  9 16:41:03 eng60 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
> before open completed
> May  9 16:41:17 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:41:18 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/auth-agent (pid 5401) terminated with signal
> Killed
> 
> May  9 16:41:20 eng60 : 0:0:auth_agent:NOTICE: main: auth-agent
> started May  9 16:41:20 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/auth-agent: finished
> initialization. May  9 16:41:20 eng60 : 0:0:ea:WARNING:
> nfxnis_createVsContext: Context for VS=1 already exists. 
> May  9 16:41:20 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=2 already exists. 
> May  9 16:41:24 eng60 : 0:0:sdm:INFO: Success reading configuration
> record from cluster db.
> May  9 16:41:39 eng60 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
> before open completed
> May  9 16:41:44 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:41:45 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/asd (pid 24872) terminated with signal Killed 
> May  9 16:41:46 eng60 : 0:0:eventd:INFO: close_session: Lost
> connection. Close rmc session 0x100b7c00 for event session 0x1008b608
> app unused 0. May  9 16:41:46 eng60 : 0:0:asd:INFO: asd_main: auto
> support conf is TO , NOTETO , FROM eng60@onstor.com, server 0.0.0.0,
> enable 0 May  9 16:41:46 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/asd: finished initialization. 
> May  9 16:41:57 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:41:57 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open success
> May  9 16:42:00 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol1', Id 0x000001b400000065, Event 'Create', State 'Down'
> May  9 16:42:01 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol2', Id 0x000001b400000066, Event 'Create', State 'Down'
> May  9 16:42:01 eng60 : 0:0:vsd:ERROR: vsd_sendMsg[9123] :
> rmc_send_msg_ex failed. rc=-3
> May  9 16:42:01 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol1)
> current state - Mounted-RW State.
> May  9 16:42:01 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol1)
> already mounted in desired state.
> May  9 16:42:01 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol2)
> current state - Mounted-RW State.
> May  9 16:42:01 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol2)
> already mounted in desired state.
> May  9 16:42:01 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol3', Id 0x000001b400000067, Event 'Create', State 'Down'
> May  9 16:42:01 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol4', Id 0x000001b400000068, Event 'Create', State 'Down'
> May  9 16:42:01 eng60 : 0:0:evm:ERROR: evm_putVolOnline : Volume[vol3]
> is already online
> May  9 16:42:01 eng60 : 0:0:ea:INFO: ea_efsMountVol : Volume (vol3)
> already mounted in the correct state.
> May  9 16:42:01 eng60 : 0:0:evm:ERROR: evm_putVolOnline : Volume[vol4]
> is already online
> May  9 16:42:01 eng60 : 0:0:ea:INFO: ea_efsMountVol : Volume (vol4)
> already mounted in the correct state.
> May  9 16:42:10 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:42:11 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 18145) terminated with signal
> Killed May  9 16:42:13 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization. 
> May  9 16:42:14 eng60 : 0:0:sscccc:NOTICE: New client 0: addr
> 10.0.0.135, port 3036; Assigning fd(12)
> May  9 16:42:16 eng60 : 0:0:nfxsh:NOTICE: cmd[0]: cluster show
> summary : status[0]
> May  9 16:42:16 eng60 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show group :
> status[0]
> May  9 16:42:16 eng60 : 0:0:nfxsh:NOTICE: cmd[2]: vsvr show all -v :
> status[0]
> May  9 16:42:16 eng60 : 0:0:nfxsh:NOTICE: cmd[3]: volume show :
> status[0]
> May  9 16:42:18 eng60 : 0:0:nfxsh:NOTICE: cmd[0]: cluster show
> summary : status[0]
> May  9 16:42:20 eng60 : 0:0:nfxsh:NOTICE: cmd[0]: volume show -r :
> status[0]
> May  9 16:42:20 eng60 : 0:0:sscccc:INFO: bad read, bytes=0 (client 0
> socket 12)
> May  9 16:42:20 eng60 : 0:0:sscccc:NOTICE: closing session for client
> 0 May  9 16:42:20 eng60 : 0:0:sscccc:NOTICE: New client 0: addr
> 10.0.0.135, port 3040; Assigning fd(12)
> May  9 16:42:20 eng60 : 0:0:sscccc:INFO: bad read, bytes=-1 (client 0
> socket 12)
> May  9 16:42:20 eng60 : 0:0:sscccc:NOTICE: closing session for client
> 0 May  9 16:42:37 eng60 : 0:0:ssh:INFO: 'root' logged in through
> remote host(10.0.0.94)
> May  9 16:42:38 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/ncmd (pid 11932) terminated with signal Killed 
> May  9 16:42:39 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:42:40 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/ncmd:
> finished initialization. 
> May  9 16:42:40 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open failed
> May  9 16:42:40 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:42:40 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open success
> May  9 16:42:44 eng60 : 0:0:ea:NOTICE: ea_closeSess : NCM session
> closed May  9 16:42:44 eng60 : 0:0:ea:NOTICE: ea_rcvRmcMsg : NCM
> session open success
> May  9 16:43:03 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:43:05 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/chassisd (pid 32145) terminated with signal
> Killed May  9 16:43:05 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/chassisd: finished initialization. 
> May  9 16:43:05 eng60 : 0:0:cm:NOTICE: CHASSISD: Chassis Manager:
> Started
> May  9 16:43:05 eng60 : 0:0:cm:NOTICE: CM: opened kseg1
> May  9 16:43:05 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 0, State Up
> May  9 16:43:05 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 0 
> May  9 16:43:05 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 1, State Up
> May  9 16:43:05 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 2, State Up
> May  9 16:43:05 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 3, State Up
> May  9 16:43:05 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 2,
> CPU 0, State Up
> May  9 16:43:05 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 1 
> May  9 16:43:05 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 2 
> May  9 16:43:05 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 3 
> May  9 16:43:05 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 2, cpu 0 
> May  9 16:43:30 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:43:31 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/auth-agent (pid 21156) terminated with signal
> Killed 
> May  9 16:43:33 eng60 : 0:0:auth_agent:NOTICE: main: auth-agent
> started May  9 16:43:33 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/auth-agent: finished
> initialization. May  9 16:43:33 eng60 : 0:0:ea:WARNING:
> nfxnis_createVsContext: Context for VS=1 already exists. 
> May  9 16:43:33 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=2 already exists. 
> May  9 16:43:57 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:43:58 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/asd (pid 17258) terminated with signal Killed 
> May  9 16:43:59 eng60 : 0:0:eventd:INFO: close_session: Lost
> connection. Close rmc session 0x100b7c00 for event session 0x1008b608
> app unused 0. May  9 16:43:59 eng60 : 0:0:asd:INFO: asd_main: auto
> support conf is TO , NOTETO , FROM eng60@onstor.com, server 0.0.0.0,
> enable 0 May  9 16:43:59 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/asd: finished initialization. 
> May  9 16:44:23 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:44:25 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/ndmp_cfgd (pid 8912) terminated with signal
> Killed May  9 16:44:26 eng60 : 0:0:ssc_ndmp:INFO: ndmp_GetFPState:
> fp card is UP 
> May  9 16:44:26 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/ndmp_cfgd:
> finished initialization. 
> May  9 16:44:50 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:44:51 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/auth-agent (pid 29019) terminated with signal
> Killed 
> May  9 16:44:53 eng60 : 0:0:auth_agent:NOTICE: main: auth-agent
> started May  9 16:44:53 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/auth-agent: finished
> initialization. May  9 16:44:53 eng60 : 0:0:ea:WARNING:
> nfxnis_createVsContext: Context for VS=1 already exists. 
> May  9 16:44:53 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=2 already exists. 
> May  9 16:45:17 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:45:18 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/snmpd (pid 19673) terminated with signal Killed 
> May  9 16:45:18 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/snmpd:
> finished initialization. 
> May  9 16:45:20 eng60 : 0:0:snmpd:INFO: UCD-SNMP version 4.2.2 
> May  9 16:45:43 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:45:45 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 15549) terminated with signal
> Killed May  9 16:45:46 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization. 
> May  9 16:46:10 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:46:11 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/ipmd (pid 6227) terminated with signal Killed 
> May  9 16:46:12 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/ipmd:
> finished initialization. 
> May  9 16:46:37 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:46:38 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sanmd (pid 25855) terminated with signal Killed 
> May  9 16:46:39 eng60 : 1:2:sanm_ag:WARNING: 23: sanm_doRMCLocalClose:
> reopening session to sanmd 
> May  9 16:46:40 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/sanmd:
> finished initialization. 
> May  9 16:46:40 eng60 : 0:0:sanm:NOTICE: SANM: ONStor Data Mirror
> (c)2006: Started
> May  9 16:47:03 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:47:05 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/vtmd (pid 17793) terminated with signal Killed 
> May  9 16:47:06 eng60 : 0:0:cluster2:INFO: ClusterCtrl_releaseFiler:
> called by vtm 
> May  9 16:47:06 eng60 : 0:0:vtm:INFO: vtm_sendCardStateMsg: Sending
> card UP to eng60 
> May  9 16:47:06 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/vtmd:
> finished initialization. 
> May  9 16:47:07 eng60 : 0:0:vtm:INFO: Vtm_ProcCardStateMsg: card state
> UP message from eng60 
> May  9 16:47:30 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:47:31 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/auth-agent (pid 11683) terminated with signal
> Killed 
> May  9 16:47:33 eng60 : 0:0:auth_agent:NOTICE: main: auth-agent
> started May  9 16:47:33 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/auth-agent: finished
> initialization. May  9 16:47:33 eng60 : 0:0:ea:WARNING:
> nfxnis_createVsContext: Context for VS=1 already exists. 
> May  9 16:47:33 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=2 already exists. 
> May  9 16:47:57 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:47:58 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/vsd (pid 3150) terminated with signal Killed 
> May  9 16:47:59 eng60 : 0:0:ea:INFO: ea_closeAcptSess: close from
> {vsd:Agile1358.0.0}
> May  9 16:48:00 eng60 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1223] : vs=1, file=71, #masterRec=0
> May  9 16:48:00 eng60 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1223] : vs=1, file=72, #masterRec=0
> May  9 16:48:00 eng60 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1223] : vs=1, file=74, #masterRec=0
> May  9 16:48:00 eng60 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1223] : vs=2, file=71, #masterRec=0
> May  9 16:48:00 eng60 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1223] : vs=2, file=72, #masterRec=0
> May  9 16:48:00 eng60 : 0:0:vsd:INFO:
> vsd_ensureNisFileCoherence[1223] : vs=2, file=74, #masterRec=0
> May  9 16:48:00 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/vsd:
> finished initialization. 
> May  9 16:48:03 eng60 : 0:0:ea:INFO: ea_newAcptSess: creating new
> session
> May  9 16:48:03 eng60 : 0:0:ea:INFO: ea_openSess: open for
> {vsd:Agile3651.0.0}
> May  9 16:48:03 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=1 already exists. 
> May  9 16:48:03 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol1)
> current state - Mounted-RW State.
> May  9 16:48:03 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol1)
> already mounted in desired state.
> May  9 16:48:03 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol2)
> current state - Mounted-RW State.
> May  9 16:48:03 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol2)
> already mounted in desired state.
> May  9 16:48:03 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol3)
> current state - Mounted-RW State.
> May  9 16:48:03 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol3)
> already mounted in desired state.
> May  9 16:48:03 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol4)
> current state - Mounted-RW State.
> May  9 16:48:03 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol4)
> already mounted in desired state.
> May  9 16:48:04 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=2 already exists. 
> May  9 16:48:24 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:48:25 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/cluster_relay (pid 24390) terminated with signal
> Killed 
> May  9 16:48:25 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/cluster_relay: finished
> initialization. May  9 16:48:50 eng60 : 0:0:ssh:INFO: 'root' logged
> in through remote host(10.0.0.94)
> May  9 16:48:52 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sscccc (pid 18116) terminated with signal Killed 
> May  9 16:48:52 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/sscccc:
> finished initialization. 
> May  9 16:48:52 eng60 : 0:0:unused:INFO: sscccc no
> /usr/local/agile/etc/sscccc_hosts_deny file
> May  9 16:49:17 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:49:18 eng60 pm: pm_sig_handler: /usr/local/agile/bin/elog
> (pid 11657) terminated with signal Killed 
> May  9 16:49:19 eng60 pm: /usr/local/agile/bin/elog: finished
> initialization. 
> May  9 16:49:44 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:49:45 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/chassisd (pid 330) terminated with signal Killed 
> May  9 16:49:45 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/chassisd:
> finished initialization. 
> May  9 16:49:45 eng60 : 0:0:cm:NOTICE: CHASSISD: Chassis Manager:
> Started
> May  9 16:49:45 eng60 : 0:0:cm:NOTICE: CM: opened kseg1
> May  9 16:49:45 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 0, State Up
> May  9 16:49:45 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 0 
> May  9 16:49:45 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 1, State Up
> May  9 16:49:45 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 2, State Up
> May  9 16:49:45 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
> CPU 3, State Up
> May  9 16:49:45 eng60 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 2,
> CPU 0, State Up
> May  9 16:49:45 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 1 
> May  9 16:49:45 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 2 
> May  9 16:49:45 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 1, cpu 3 
> May  9 16:49:45 eng60 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 1, slot 2, cpu 0 
> May  9 16:50:10 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:50:12 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/vtmd (pid 22618) terminated with signal Killed 
> May  9 16:50:13 eng60 : 0:0:cluster2:INFO: ClusterCtrl_releaseFiler:
> called by vtm 
> May  9 16:50:13 eng60 : 0:0:vtm:INFO: vtm_sendCardStateMsg: Sending
> card UP to eng60 
> May  9 16:50:13 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/vtmd:
> finished initialization. 
> May  9 16:50:31 eng60 : 0:0:vtm:INFO: Vtm_ProcCardStateMsg: card state
> UP message from eng60 
> May  9 16:50:37 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:50:59 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:51:00 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 8085) terminated with signal
> Killed May  9 16:51:02 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization. 
> May  9 16:51:26 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:51:27 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/tape-driver (pid 23456) terminated with signal
> Killed 
> May  9 16:51:27 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/tape-driver:
> finished initialization. 
> May  9 16:51:29 eng60 : 0:0:tape-driver:NOTICE: Adding new tape
> device: id 00000000f91f8ee8, WWN 500308c13d680003, LUN 0, vendor
> 'IBM', model 'ULT3580-TD2' 
> May  9 16:51:29 eng60 : 0:0:tape-driver:NOTICE: Adding new
> media-changer: id 00000001796dd4fb, WWN 500308c13d680003, LUN 1,
> vendor 'IBM', model 'ULT3582-TL' 
> May  9 16:51:52 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:52:14 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:52:15 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 8481) terminated with signal
> Killed May  9 16:52:17 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization. 
> May  9 16:52:20 eng60 : 0:0:sscccc:NOTICE: New client 0: addr
> 10.0.0.135, port 3116; Assigning fd(12)
> May  9 16:52:22 eng60 : 0:0:nfxsh:NOTICE: cmd[0]: cluster show
> summary : status[0]
> May  9 16:52:22 eng60 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show group :
> status[0]
> May  9 16:52:22 eng60 : 0:0:nfxsh:NOTICE: cmd[2]: vsvr show all -v :
> status[0]
> May  9 16:52:22 eng60 : 0:0:nfxsh:NOTICE: cmd[3]: volume show :
> status[0]
> May  9 16:52:24 eng60 : 0:0:nfxsh:NOTICE: cmd[0]: cluster show
> summary : status[0]
> May  9 16:52:26 eng60 : 0:0:nfxsh:NOTICE: cmd[0]: volume show -r :
> status[0]
> May  9 16:52:26 eng60 : 0:0:sscccc:INFO: bad read, bytes=0 (client 0
> socket 12)
> May  9 16:52:26 eng60 : 0:0:sscccc:NOTICE: closing session for client
> 0 May  9 16:52:26 eng60 : 0:0:sscccc:NOTICE: New client 0: addr
> 10.0.0.135, port 3119; Assigning fd(12)
> May  9 16:52:26 eng60 : 0:0:sscccc:INFO: bad read, bytes=-1 (client 0
> socket 12)
> May  9 16:52:26 eng60 : 0:0:sscccc:NOTICE: closing session for client
> 0 May  9 16:52:41 eng60 : 0:0:ssh:INFO: 'root' logged in through
> remote host(10.0.0.94)
> May  9 16:52:42 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/tape-driver (pid 27313) terminated with signal
> Killed 
> May  9 16:52:42 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/tape-driver:
> finished initialization. 
> May  9 16:52:44 eng60 : 0:0:tape-driver:NOTICE: Adding new tape
> device: id 00000000f91f8ee8, WWN 500308c13d680003, LUN 0, vendor
> 'IBM', model 'ULT3580-TD2' 
> May  9 16:52:44 eng60 : 0:0:tape-driver:NOTICE: Adding new
> media-changer: id 00000001796dd4fb, WWN 500308c13d680003, LUN 1,
> vendor 'IBM', model 'ULT3582-TL' 
> May  9 16:53:08 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:53:09 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sscccc (pid 19184) terminated with signal Killed 
> May  9 16:53:09 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/sscccc:
> finished initialization. 
> May  9 16:53:09 eng60 : 0:0:unused:INFO: sscccc no
> /usr/local/agile/etc/sscccc_hosts_deny file
> May  9 16:53:34 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:53:36 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/eventd (pid 14473) terminated with signal Killed 
> May  9 16:53:36 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/eventd:
> finished initialization. 
> May  9 16:53:36 eng60 : 0:0:asd:ERROR: asd_main: Lost connection to
> eventd. Reinitializing conn  
> May  9 16:53:36 eng60 : 0:0:asd:NOTICE: asd_main: Connected to
> eventd May  9 16:54:01 eng60 : 0:0:ssh:INFO: 'root' logged in through
> remote host(10.0.0.94)
> May  9 16:54:02 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sdm_cfgd (pid 8039) terminated with signal
> Killed May  9 16:54:02 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sdm_cfgd: finished initialization. 
> May  9 16:54:02 eng60 : 0:0:sdm:INFO: ONStor Storage Device Manager
> (c)2003: Started
> May  9 16:54:03 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:54:03 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open success
> May  9 16:54:04 eng60 : 0:0:sdm:INFO: sdm_rcvRmcMsg : RMC open
> successful (0)
> May  9 16:54:04 eng60 : 0:0:sdm:INFO: sdm_rcvRmcMsg : RMC open
> successful (0)
> May  9 16:54:04 eng60 : 0:0:sdm:INFO: sdm_rcvRmcMsg : RMC Open to FC
> Port successful {sdm:Agile3974}->{:scsi2.2.0}
> May  9 16:54:04 eng60 : 2:0:scsi:INFO: 37: scsi_openSessToSDM:RMC
> rc[-9] msg[0x0]
> May  9 16:54:04 eng60 : 2:0:scsi:INFO: 38: scsi_openSessToSDM: RMC
> session found open at [0x60e51010], issue close for session. tagVal =
> [40c]  
> May  9 16:54:04 eng60 : 2:0:scsi:INFO: 39: scsi_openSessToSDM:
> scsiCtrl->sdmOpenSession = 0x60e51010, and &scsiCtrl->sdmOpenSession =
> 0x623a98bc  
> May  9 16:54:08 eng60 : 0:0:tape-driver:ERROR: tape_rpc: rmc_rpc app
> sdm cpu 0 slot 0 failed - -3
> May  9 16:54:28 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:54:29 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/auth-agent (pid 24682) terminated with signal
> Killed 
> May  9 16:54:31 eng60 : 0:0:auth_agent:NOTICE: main: auth-agent
> started May  9 16:54:31 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/auth-agent: finished
> initialization. May  9 16:54:31 eng60 : 0:0:ea:WARNING:
> nfxnis_createVsContext: Context for VS=1 already exists. 
> May  9 16:54:31 eng60 : 0:0:ea:WARNING: nfxnis_createVsContext:
> Context for VS=2 already exists. 
> May  9 16:54:36 eng60 : 0:0:sdm:INFO: Success reading configuration
> record from cluster db.
> May  9 16:54:55 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:54:56 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/ea (pid 18322) terminated with signal Killed 
> May  9 16:54:56 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/ea: finished
> initialization. 
> May  9 16:54:58 eng60 : 0:0:ea:NOTICE: ea_rcvRmcMsg : NCM session open
> failed
> May  9 16:55:03 eng60 : 0:0:ea:NOTICE: ea_closeSess : NCM session
> closed May  9 16:55:03 eng60 : 0:0:ea:NOTICE: ea_rcvRmcMsg : NCM
> session open success
> May  9 16:55:22 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:55:23 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/snmpd (pid 12658) terminated with signal Killed 
> May  9 16:55:23 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/snmpd:
> finished initialization. 
> May  9 16:55:25 eng60 : 0:0:snmpd:INFO: UCD-SNMP version 4.2.2 
> May  9 16:55:48 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:55:50 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/cluster_relay (pid 5022) terminated with signal
> Killed 
> May  9 16:55:50 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/cluster_relay: finished
> initialization. May  9 16:56:15 eng60 : 0:0:ssh:INFO: 'root' logged
> in through remote host(10.0.0.94)
> May  9 16:56:17 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/spm (pid 21333) terminated with signal Killed 
> May  9 16:56:18 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/spm:
> finished initialization. 
> May  9 16:56:23 eng60 : 0:0:evm:ERROR: evm_readLunLabelReq : SDM
> error[-1] handle[0x1210000] for read label
> LUN[MYLEX_20000080e5128ae5_3] May  9 16:56:23 eng60 : 0:0:evm:ERROR:
> evm_verifyLunReqProc : Cannot read label for
> LUN[MYLEX_20000080e5128ae5_3] May  9 16:56:23 eng60 : 0:0:evm:ERROR:
> evm_verifyLunReqProc: Missing label on LUN[MYLEX_20000080e5128ae5_3]!
> May  9 16:56:23 eng60 : 0:0:evm:ERROR: evm_verifyLunReqProc: removing
> vol[] LUN[MYLEX_20000080e5128ae5_3] from cluster[eng60]
> May  9 16:56:23 eng60 : 0:0:evm:ERROR: evm_readLunLabelReq : SDM
> error[-1] handle[0x1210001] for read label
> LUN[MYLEX_20000080e5128ae5_16]
> May  9 16:56:23 eng60 : 0:0:evm:ERROR: evm_verifyLunReqProc : Cannot
> read label for LUN[MYLEX_20000080e5128ae5_16]
> May  9 16:56:23 eng60 : 0:0:evm:ERROR: evm_verifyLunReqProc: Missing
> label on LUN[MYLX_20000080e5128ae5_16]!
> May  9 16:56:23 eng60 : 0:0:evm:ERROR:2         0x1b400000066 - write
> - buf - Volume Signal Exception: frame[5] = 0xffffffff83062830 
> May  9 16:56:23 eng60 : 1:2:efs:WARNING: 32: FS: vol2
> 0x1b400000066 - write - buf - Volume Signal Exception: frame[6] =
> 0xffffffff83056190 
> May  9 16:56:24 eng60 : 1:2:efs:WARNING: 33: FS: vol2
> 0x1b400000066 - write - buf - Volume Signal Exception: frame[7] =
> 0xffffffff830563f0 
> May  9 16:56:24 eng60 : 1:2:efs:WARNING: 34: FS: vol2
> 0x1b400000066 - write - buf - Volume Signal Exception: frame[8] =
> 0xffffffff830cf768 
> May  9 16:56:24 eng60 : 1:2:efs:WARNING: 35: FS: vol2
> 0x1b400000066 - write - buf - Volume Signal Exception: frame[9] =
> 0xffffffff830e3a60 
> May  9 16:56:24 eng60 : 1:3:evm:ERROR: 36: evm_verifyReq: cannot find
> volume[0x1b400000068]
> May  9 16:56:24 eng60 : 1:2:evm:ERROR: 37: evm_verifyReq: cannot find
> volume[0x1b400000067]
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 38: FILESYSTEM WARNING on
> volId 0x1b400000068: volume write, filesystem error on volId
> 0x1b400000068 (vol4), aborting volume  
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 39: FS: vol4
> 0x1b400000068 - write - buf - Volume Signal Exception: frame[0] =
> 0xffffffff8304ee40 
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 40: FS: vol4
> 0x1b400000068 - write - buf - Volume Signal Exception: frame[1] =
> 0xffffffff830abf74 
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 48: FS: vol4
> 0x1b400000068 - write - buf - Volume Signal Exception: frame[9] =
> 0xffffffff830d0ba4 
> May  9 16:56:24 eng60 : 1:2:efs:WARNING: 49: FILESYSTEM WARNING on
> volId 0x1b400000067: volume write, filesystem error on volId
> 0x1b400000067 (vol3), aborting volume  
> May  9 16:56:24 eng60 : 1:2:efs:WARNING: 50: FS: vol3
> 0x1b400000067 - write - buf - Volume Signal Exception: frame[0] =
> 0xffffffff8304ee40 
> May  9 16:56:24 eng60 : 1:2:efs:WARNING: 51: FS: vol3
> 0x1b400000067 - write - buf - Volume Signal Exception: frame[1] =
> 0xffffffff830abf74 
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 61: FILESYSTEM WARNING on
> volId 0x1b400000065: volume write, filesystem error on volId
> 0x1b400000065 (vol1), aborting volume  
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 62: FS: vol1
> 0x1b400000065 - remove - buf - Volume Signal Exception: frame[0] =
> 0xffffffff8304ee40 
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 63: FS: vol1
> 0x1b400000065 - remove - buf - Volume Signal Exception: frame[1] =
> 0xffffffff830abf74 
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 64: FS: vol1
> 0x1b400000065 - remove - buf - Volume Signal Exception: frame[2] =
> 0xffffffff83103508 
> May  9 16:56:24 eng60 : 1:3:efs:WARNING: 65: FS: vol1
> 0x1b400000065 - remove - buf - Volume Signal Exception: frame[3] =
> 0xffffffff83105614 
> May  9 16:56:27 eng60 : 1:2:efs:INFO: 72: FS: vol4
> 0x1b400000068
> - sm 0x0 - req - Forcibly unmounting volume
> May  9 16:56:27 eng60 : 0:0:eventd:INFO: close_session: Lost
> connection. Close rmc session 0x1008fc00 for event session 0x1008b608
> app unused 0. May  9 16:56:27 eng60 : 1:2:efs:INFO: 73: FS:
> vol3         0x1b400000067
> - sm 0x0 - req - Forcibly unmounting volume
> May  9 16:56:27 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol4', Id 0x000001b400000068, Event 'File System Abort', State 'Down'
> May  9 16:56:27 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:56:28 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol3', Id 0x000001b400000067, Event 'File System Abort', State 'Down'
> May  9 16:56:28 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:56:28 eng60 : 0:0:vsd:ERROR: vsd_mountVolProc : Abort mount
> operation for VS[2], req[4] > rsp[0]
> May  9 16:56:28 eng60 : 1:3:efs:INFO: 74: FS: vol2
> 0x1b400000066
> - sm 0x0 - req - Forcibly unmounting volume
> May  9 16:56:28 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol2', Id 0x000001b400000066, Event 'File System Abort', State 'Down'
> May  9 16:56:28 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:56:28 eng60 : 0:0:vsd:ERROR: vsd_getVolInfoReqProc : Failed
> to get status for volume 0x1b400000066
> May  9 16:56:28 eng60 last message repeated 2 times
> May  9 16:56:28 eng60 : 0:0:vsd:ERROR: vsd_getVolInfoReqProc : Failed
> to get status for volume 0x1b400000067
> May  9 16:56:28 eng60 last message repeated 3 times
> May  9 16:56:29 eng60 : 1:2:efs:INFO: 75: FS: vol1
> 0x1b400000065
> - sm 0x0 - req - Forcibly unmounting volume
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol1', Id 0x000001b400000065, Event 'File System Abort', State 'Down'
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol1', Id 0x000001b400000065, Event 'Create', State 'Down'
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol2', Id 0x000001b400000066, Event 'Create', State 'Down'
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol4', Id 0x000001b400000068, Event 'Create', State 'Down'
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol3', Id 0x000001b400000067, Event 'Create', State 'Down'
> May  9 16:56:29 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:56:38 eng60 : 0:0:vsd:ERROR: vsd_sendMsg[9123] :
> rmc_send_msg_ex failed. rc=-3
> May  9 16:56:38 eng60 : 0:0:ea:INFO: ea_newAcptSess: creating new
> session
> May  9 16:56:38 eng60 : 0:0:ea:INFO: ea_openSess: open for
> {vsd:Agile1202.0.0}
> May  9 16:56:38 eng60 : 1:3:efs:INFO: 76: FS: vol1
> 0x1b400000065
> - mount - req - Volume is dirty, will check for log (devtype 2 logsize
> 62906816)
> May  9 16:56:38 eng60 : 1:3:efs:INFO: 77: FS: vol2
> 0x1b400000066
> - mount - req - Volume is dirty, will check for log (devtype 2 logsize
> 62906816)
> May  9 16:56:38 eng60 : 1:2:efs:INFO: 78: FS: vol3
> 0x1b400000067
> - mount - req - Volume is dirty, will check for log (devtype 2 logsize
> 62906816)
> May  9 16:56:38 eng60 : 1:2:efs:INFO: 79: FS: vol4
> 0x1b400000068
> - mount - req - Volume is dirty, will check for log (devtype 2 logsize
> 62906816)
> May  9 16:56:39 eng60 : 1:3:efs:INFO: 80: FS: vol4
> 0x1b400000068
> - mount - log - filesystem queued for replay
> May  9 16:56:39 eng60 : 1:2:efs:INFO: 81: FS: vol3
> 0x1b400000067
> - mount - log - filesystem queued for replay
> May  9 16:56:40 eng60 : 1:2:efs:INFO: 82: FS: vol1
> 0x1b400000065
> - mount - log - replay successfull head 25556048 tail 47947232 len
> 22391184 commits 64532
> May  9 16:56:40 eng60 : 1:2:efs:NOTICE: 83: FS: vol1
> 0x1b400000065 - mount - log - replay: syncing user data after log
> replay May  9 16:56:40 eng60 : 1:3:efs:INFO: 84: FS: vol2
> 0x1b400000066
> - mount - log - replay successfull head 28504776 tail 55285384 len
> 26780608 commits 97023
> May  9 16:56:40 eng60 : 1:3:efs:NOTICE: 85: FS: vol2
> 0x1b400000066 - mount - log - replay: syncing user data after log
> replay May  9 16:56:41 eng60 : 1:2:efs:INFO: 86: FS: vol4
> 0x1b400000068
> - mount - log - filesystem dequeued, starting replay
> May  9 16:56:41 eng60 : 1:2:efs:INFO: 87: FS: vol1
> 0x1b400000065
> - mount - req - zero-fill begin
> May  9 16:56:41 eng60 : 1:2:efs:INFO: 88: FS: vol3
> 0x1b400000067
> - mount - log - filesystem dequeued, starting replay
> May  9 16:56:41 eng60 : 1:3:efs:INFO: 89: FS: vol2
> 0x1b400000066
> - mount - req - zero-fill begin
> May  9 16:56:41 eng60 : 1:3:efs:INFO: 90: FS: vol1
> 0x1b400000065
> - mount - req - Volume online   version 28
> May  9 16:56:41 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol1', Id 0x000001b400000065, Event 'Online', was offline for roughly
> 24 sec.
> May  9 16:56:41 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:56:41 eng60 : 1:3:efs:INFO: 91: FS: vol2
> 0x1b400000066
> - mount - req - Volume online   version 28
> May  9 16:56:41 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol2', Id 0x000001b400000066, Event 'Online', was offline for roughly
> 25 sec.
> May  9 16:56:41 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:56:42 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:56:42 eng60 : 1:3:efs:INFO: 92: FS: vol4
> 0x1b400000068
> - mount - log - replay successfull head 54060624 tail 10011648 len
> 18857840 commits 98955
> May  9 16:56:42 eng60 : 1:3:efs:NOTICE: 93: FS: vol4
> 0x1b400000068 - mount - log - replay: syncing user data after log
> replay May  9 16:56:43 eng60 : 1:3:efs:INFO: 94: FS: vol4
> 0x1b400000068
> - mount - req - zero-fill begin
> May  9 16:56:43 eng60 : 1:3:efs:INFO: 95: FS: vol4
> 0x1b400000068
> - mount - req - Volume online   version 28
> May  9 16:56:43 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol4', Id 0x000001b400000068, Event 'Online', was offline for roughly
> 23 sec.
> May  9 16:56:43 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:56:43 eng60 : 1:3:efs:INFO: 96: FS: vol3
> 0x1b400000067
> - mount - log - replay successfull head 10812112 tail 35568600 len
> 24756488 commits 84685
> May  9 16:56:43 eng60 : 1:3:efs:NOTICE: 97: FS: vol3
> 0x1b400000067 - mount - log - replay: syncing user data after log
> replay May  9 16:56:43 eng60 : 1:2:efs:INFO: 98: FS: vol3
> 0x1b400000067
> - mount - req - zero-fill begin
> May  9 16:56:43 eng60 : 1:2:efs:INFO: 99: FS: vol3
> 0x1b400000067
> - mount - req - Volume online   version 28
> May  9 16:56:43 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol3', Id 0x000001b400000067, Event 'Online', was offline for roughly
> 24 sec.
> May  9 16:56:43 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:56:44 eng60 : 1:1:nfs:INFO: 16: Finished failover of NSM
> file after volume event on virtual server 2 
> May  9 16:56:44 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 18229) terminated with signal
> Killed May  9 16:56:46 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization. 
> May  9 16:57:09 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:57:11 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/evm_cfgd (pid 10777) terminated with signal
> Killed May  9 16:57:11 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/evm_cfgd: finished initialization. 
> May  9 16:57:12 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open failed
> May  9 16:57:12 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open failed
> May  9 16:57:12 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:57:12 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:57:12 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open success
> May  9 16:57:12 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open success
> May  9 16:57:36 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:57:38 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/evm_cfgd (pid 718) terminated with signal Killed 
> May  9 16:57:38 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/evm_cfgd:
> finished initialization. 
> May  9 16:57:39 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open failed
> May  9 16:57:39 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open failed
> May  9 16:57:39 eng60 : 0:0:evm:NOTICE: evm_closeSess : NCM session
> closed
> May  9 16:57:39 eng60 : 0:0:evm:NOTICE: evm_closeSess : SDM session
> closed
> May  9 16:57:39 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : NCM session
> open success
> May  9 16:57:39 eng60 : 0:0:evm:NOTICE: evm_rcvRmcMsg : SDM session
> open success
> May  9 16:58:03 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:58:04 eng60 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
> before open completed
> May  9 16:58:04 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 25995) terminated with signal
> Killed May  9 16:58:06 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization. 
> May  9 16:58:27 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol1', Id 0x000001b400000065, Event 'Create', State 'Down'
> May  9 16:58:27 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:58:27 eng60 : 0:0:vsd:ERROR: vsd_sendMsg[9123] :
> rmc_send_msg_ex failed. rc=-3
> May  9 16:58:27 eng60 : 0:0:vsd:ERROR: vsd_mountVolProc : Abort mount
> operation for VS[2], req[4] > rsp[1]
> May  9 16:58:30 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:58:31 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sscccc (pid 16877) terminated with signal Killed 
> May  9 16:58:31 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/sscccc:
> finished initialization. 
> May  9 16:58:31 eng60 : 0:0:unused:INFO: sscccc no
> /usr/local/agile/etc/sscccc_hosts_deny file
> May  9 16:58:37 eng60 : 0:0:vsd:ERROR: vsd_mountVolProc : Abort mount
> operation for VS[2], req[4] > rsp[1]
> May  9 16:58:57 eng60 last message repeated 2 times
> May  9 16:58:57 eng60 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
> before open completed
> May  9 16:58:57 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:58:58 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/sendmail (pid 10969) terminated with signal
> Killed May  9 16:59:00 eng60 :
> 0:0:pm:INFO: /usr/local/agile/bin/sendmail: finished initialization. 
> May  9 16:59:02 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol2', Id 0x000001b400000066, Event 'Create', State 'Down'
> May  9 16:59:02 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:59:02 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol3', Id 0x000001b400000067, Event 'Create', State 'Down'
> May  9 16:59:02 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:59:02 eng60 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol4', Id 0x000001b400000068, Event 'Create', State 'Down'
> May  9 16:59:02 eng60 : 0:0:eventd:NOTICE: forwardToLocalApps dstAPP:
> asd_rmc EVENT: Volume RMC Error -3 
> May  9 16:59:07 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol1)
> current state - Mounted-RW State.
> May  9 16:59:07 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol1)
> already mounted in desired state.
> May  9 16:59:07 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol2)
> current state - Mounted-RW State.
> May  9 16:59:07 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol2)
> already mounted in desired state.
> May  9 16:59:07 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol3)
> current state - Mounted-RW State.
> May  9 16:59:07 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol3)
> already mounted in desired state.
> May  9 16:59:07 eng60 : 0:0:ea:NOTICE: ea_movInProc : Volume (vol4)
> current state - Mounted-RW State.
> May  9 16:59:07 eng60 : 0:0:ea:INFO: ea_volRemountProc : Volume (vol4)
> already mounted in desired state.
> May  9 16:59:24 eng60 : 0:0:ssh:INFO: 'root' logged in through remote
> host(10.0.0.94)
> May  9 16:59:25 eng60 : 0:0:pm:ERROR: pm_sig_handler:
> /usr/local/agile/bin/crashsaved (pid 32099) terminated with signal
> Killed 
> May  9 16:59:25 eng60 : 0:0:pm:INFO: /usr/local/agile/bin/crashsaved:
> finished initialization. 
> panic: malloc: out of space in kmem_map
> dumpsys: cannot open crashdump device: 16
> 
> dumping to dev 411, offset 8
> dump not yet implementedSystem restart.
> 
> PMON CPU 00000000 Initializing. Standby...
> EXCEPTIONPC=00000000 CONFIG=005264b0 STATUS=00400000
> Setting up SDRAM controllers
> R9K MEMORY SIZE=40000000
> SDM  REG=00000047
> SDT  REG=283f8482
> SDRAM Memory setup done.
> Clearing cache size memory...
> Cache init done.
> Initializing ECC memory.
> Decompression bss start.
> Init bss.
> 
> 
> 
> PowerOn Self Test........OK
> 
> Initializing System......please wait
> 
> Initializing Shared Memory (1024 MB) for ECC
> Taking PCI out of reset.
> Waiting for FP/TXRX out of reset......
> 
> 
> pid=20929, daemon=chassisd
> pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> pid=22618, daemon=vtmd
> pid=23456, daemon=tape-driver
> pid=24682, daemon=auth-agent
> pid=26301, daemon=asd
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.96 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  2063 q9  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=28664, daemon=timekeeper
> pid=32099, daemon=crashsaved
> 
> num=18, length=24, name=vtmd
> kill -9 22618
> # Killing vtmd pid 22618
> ps ax | grep vtmd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 25766 q9  R+      0:00.00 grep vtmd (sh)
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:50:10 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.01 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=5022, daemon=cluster_relay
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8085 ??  I       0:00.08 /usr/local/agile/bin/sendmail -Q30m -r
> pid=8039, daemon=sdm_cfgd
> pid=8085, daemon=sendmail
>  8202 ??  S       0:00.04 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.29 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> pid=8202, daemon=elog
> pid=8699, daemon=sanmd
> pid=10777, daemon=evm_cfgd
> pid=12658, daemon=snmpd
> pid=13027, daemon=ipmd
> pid=14473, daemon=eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.07 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.47 /usr/local/agile/bin/sshd
> 18322 ??  S       0:00.95 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.32 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.09 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.11 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.18 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 23456 ??  S       0:00.14 /usr/local/agile/bin/tape-driver
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> pid=18322, daemon=ea
> pid=18816, daemon=pm
> pid=19184, daemon=sscccc
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> pid=23456, daemon=tape-driver
> pid=24682, daemon=auth-agent
> pid=26301, daemon=asd
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.96 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 18960 qa  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=28664, daemon=timekeeper
> pid=32099, daemon=crashsaved
> 
> num=11, length=24, name=sshd
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:50:37 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.01 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=5022, daemon=cluster_relay
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8085 ??  I       0:00.08 /usr/local/agile/bin/sendmail -Q30m -r
> pid=8039, daemon=sdm_cfgd
> pid=8085, daemon=sendmail
>  8202 ??  S       0:00.04 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.29 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> pid=8202, daemon=elog
> pid=8699, daemon=sanmd
> pid=10777, daemon=evm_cfgd
> pid=12658, daemon=snmpd
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.07 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.47 /usr/local/agile/bin/sshd
> 18322 ??  S       0:00.95 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.32 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.09 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.11 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.18 /usr/local/agile/bin/chassisd -r
> pid=13027, daemon=ipmd
> pid=14473, daemon=eventd
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> pid=18322, daemon=ea
> pid=18816, daemon=pm
> pid=19184, daemon=sscccc
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> pid=21333, daemon=spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 23456 ??  S       0:00.14 /usr/local/agile/bin/tape-driver
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.96 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 28732 qb  S+      0:00.00 grep /usr/local/agile/bin
> # pid=21489, daemon=ndmp_cfgd
> pid=23456, daemon=tape-driver
> pid=24682, daemon=auth-agent
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> pid=32099, daemon=crashsaved
> 
> num=2, length=24, name=sendmail
> kill -9 8085
> # Killing sendmail pid 8085
> ps ax | grep sendmail
>  8481 ??  S       0:00.08 /usr/local/agile/bin/sendmail -Q30m -r
> 16237 qb  R+      0:00.00 grep sendmail (sh)
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:50:59 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.02 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=5022, daemon=cluster_relay
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8202 ??  S       0:00.04 /usr/local/agile/bin/elog -r
> pid=8039, daemon=sdm_cfgd
> pid=8202, daemon=elog
>  8481 ??  I       0:00.08 /usr/local/agile/bin/sendmail -Q30m -r
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.29 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> pid=8481, daemon=sendmail
> pid=8699, daemon=sanmd
> pid=10777, daemon=evm_cfgd
> pid=12658, daemon=snmpd
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.07 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.47 /usr/local/agile/bin/sshd
> 18322 ??  S       0:00.95 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.34 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.09 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.11 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.18 /usr/local/agile/bin/chassisd -r
> pid=13027, daemon=ipmd
> pid=14473, daemon=eventd
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> pid=18322, daemon=ea
> pid=18816, daemon=pm
> pid=19184, daemon=sscccc
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 23456 ??  S       0:00.14 /usr/local/agile/bin/tape-driver
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> # pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> pid=23456, daemon=tape-driver
> pid=24682, daemon=auth-agent
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> pid=32099, daemon=crashsaved
> 
> num=19, length=24, name=tape-driver
> kill -9 23456
> # Killing tape-driver pid 23456
> ps ax | grep tape-driver
> 27313 ??  S       0:00.04 /usr/local/agile/bin/tape-driver -r
> 11863 qc  S+      0:00.00 grep tape-driver
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:51:26 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.02 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=5022, daemon=cluster_relay
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8202 ??  S       0:00.07 /usr/local/agile/bin/elog -r
>  8481 ??  I       0:00.08 /usr/local/agile/bin/sendmail -Q30m -r
> pid=8039, daemon=sdm_cfgd
> pid=8202, daemon=elog
> pid=8481, daemon=sendmail
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.29 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.08 /usr/local/agile/bin/ncmd -r
> pid=8699, daemon=sanmd
> pid=10777, daemon=evm_cfgd
> pid=12658, daemon=snmpd
> pid=13027, daemon=ipmd
> pid=14473, daemon=eventd
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> 16585 ??  Ss      0:00.47 /usr/local/agile/bin/sshd
> 18322 ??  S       0:00.95 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.35 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.09 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.11 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> pid=16585, daemon=sshd
> pid=18322, daemon=ea
> pid=18816, daemon=pm
> pid=19184, daemon=sscccc
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> 27313 ??  S       0:00.04 /usr/local/agile/bin/tape-driver -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 17481 qd  S+      0:00.00 grep /usr/local/agile/bin
> pid=24682, daemon=auth-agent
> pid=26301, daemon=asd
> pid=27313, daemon=tape-driver
> pid=28664, daemon=timekeeper
> pid=32099, daemon=crashsaved
> #
> num=13, length=24, name=pm
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:51:52 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.02 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=5022, daemon=cluster_relay
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8202 ??  S       0:00.07 /usr/local/agile/bin/elog -r
> pid=8039, daemon=sdm_cfgd
> pid=8202, daemon=elog
>  8481 ??  I       0:00.08 /usr/local/agile/bin/sendmail -Q30m -r
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.29 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> pid=8481, daemon=sendmail
> pid=8699, daemon=sanmd
> pid=10777, daemon=evm_cfgd
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.08 /usr/local/agile/bin/ncmd -r
> pid=12658, daemon=snmpd
> pid=13027, daemon=ipmd
> pid=14473, daemon=eventd
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> 16585 ??  Ss      0:00.48 /usr/local/agile/bin/sshd
> 18322 ??  S       0:00.95 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.36 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.09 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.11 /usr/local/agile/bin/vsd -r
> pid=16585, daemon=sshd
> pid=18322, daemon=ea
> pid=18816, daemon=pm
> pid=19184, daemon=sscccc
> pid=20443, daemon=vsd
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> 27313 ??  S       0:00.04 /usr/local/agile/bin/tape-driver -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 11499 qe  R+      0:00.00 grep /usr/local/agile/bin (sh)
> pid=20929, daemon=chassisd
> pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> pid=24682, daemon=auth-agent
> pid=26301, daemon=asd
> pid=27313, daemon=tape-driver
> pid=28664, daemon=timekeeper
> pid=32099, daemon=crashsaved
> #
> num=3, length=24, name=sendmail
> kill -9 8481
> # Killing sendmail pid 8481
> ps ax | grep sendmail
> 18229 ??  S       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
>  1890 qe  S+      0:00.00 grep sendmail
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:52:14 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.02 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=5022, daemon=cluster_relay
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8202 ??  S       0:00.12 /usr/local/agile/bin/elog -r
> pid=8039, daemon=sdm_cfgd
> pid=8202, daemon=elog
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.29 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> pid=8699, daemon=sanmd
> pid=10777, daemon=evm_cfgd
> pid=12658, daemon=snmpd
> pid=13027, daemon=ipmd
> pid=14473, daemon=eventd
> pid=15806, daemon=vtmd
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18322 ??  S       0:01.03 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.43 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.17 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.12 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> pid=18229, daemon=sendmail
> pid=18322, daemon=ea
> pid=18816, daemon=pm
> pid=19184, daemon=sscccc
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> pid=24682, daemon=auth-agent
> pid=26301, daemon=asd
> 27313 ??  S       0:00.04 /usr/local/agile/bin/tape-driver -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 10362 qf  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=27313, daemon=tape-driver
> pid=28664, daemon=timekeeper
> pid=32099, daemon=crashsaved
> 
> num=21, length=24, name=tape-driver
> kill -9 27313
> # Killing tape-driver pid 27313
> ps ax | grep tape-driver
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  6714 qf  R+      0:00.00 grep tape-driver (sh)
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:52:41 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.02 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=5022, daemon=cluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
>  8202 ??  S       0:00.12 /usr/local/agile/bin/elog -r
> pid=6146, daemon=tape-driver
> pid=8039, daemon=sdm_cfgd
> pid=8202, daemon=elog
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.19 /usr/local/agile/bin/ipmd -r
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> pid=8699, daemon=sanmd
> pid=10777, daemon=evm_cfgd
> pid=12658, daemon=snmpd
> pid=13027, daemon=ipmd
> pid=14473, daemon=eventd
> pid=15806, daemon=vtmd
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18322 ??  S       0:01.03 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.45 /usr/local/agile/bin/pm
> 19184 ??  S       0:00.17 /usr/local/agile/bin/sscccc -r
> 20443 ??  S       0:00.12 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> pid=18229, daemon=sendmail
> pid=18322, daemon=ea
> pid=18816, daemon=pm
> pid=19184, daemon=sscccc
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> pid=24682, daemon=auth-agent
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 29675 r0  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=32099, daemon=crashsaved
> 
> num=15, length=24, name=sscccc
> kill -9 19184
> # Killing sscccc pid 19184
> ps ax | grep sscccc
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:53:08 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.02 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=5022, daemon=cluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
> pid=6146, daemon=tape-driver
> pid=8039, daemon=sdm_cfgd
>  8202 ??  S       0:00.12 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.20 /usr/local/agile/bin/ipmd -r
> pid=8202, daemon=elog
> pid=8699, daemon=sanmd
> pid=10777, daemon=evm_cfgd
> pid=12658, daemon=snmpd
> pid=13027, daemon=ipmd
> 14473 ??  S       0:01.00 /usr/local/agile/bin/eventd
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> pid=14473, daemon=eventd
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> pid=16877, daemon=sscccc
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18322 ??  S       0:01.03 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.49 /usr/local/agile/bin/pm
> 20443 ??  S       0:00.12 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> pid=18229, daemon=sendmail
> pid=18322, daemon=ea
> pid=18816, daemon=pm
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.05 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>   177 r1  R+      0:00.00 grep /usr/local/agile/bin
> pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> pid=24682, daemon=auth-agent
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> pid=32099, daemon=crashsaved
> #
> num=8, length=24, name=eventd
> kill -9 14473
> # Killing eventd pid 14473
> ps ax | grep eventd
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> 27221 r1  S+      0:00.00 grep eventd
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:53:34 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  2604 ??  S<      0:01.03 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=5022, daemon=cluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8039 ??  S       0:00.34 /usr/local/agile/bin/sdm_cfgd -r
> pid=6146, daemon=tape-driver
> pid=8039, daemon=sdm_cfgd
>  8202 ??  S       0:00.13 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.07 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> pid=8202, daemon=elog
> pid=8699, daemon=sanmd
> pid=9368, daemon=eventd
> pid=10777, daemon=evm_cfgd
> pid=12658, daemon=snmpd
> 13027 ??  S       0:00.23 /usr/local/agile/bin/ipmd -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> pid=13027, daemon=ipmd
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> pid=16877, daemon=sscccc
> pid=18229, daemon=sendmail
> 18322 ??  S       0:01.03 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.55 /usr/local/agile/bin/pm
> 20443 ??  S       0:00.12 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24682 ??  S       0:00.06 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.06 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.97 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 30872 r2  R+      0:00.00 grep /usr/local/agile/bin (sh)
> pid=18322, daemon=ea
> pid=18816, daemon=pm
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> pid=24682, daemon=auth-agent
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> pid=32099, daemon=crashsaved
> #
> num=2, length=24, name=sdm_cfgd
> kill -9 8039
> # Killing sdm_cfgd pid 8039
> ps ax | grep sdm_cfgd
>  1343 ??  S       0:00.12 /usr/local/agile/bin/sdm_cfgd -r
> 30194 r2  R+      0:00.00 grep sdm_cfgd (sh)
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:54:01 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.12 /usr/local/agile/bin/sdm_cfgd -r
> pid=1343, daemon=sdm_cfgd
>  2604 ??  S<      0:01.03 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=5022, daemon=cluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.15 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
> pid=6146, daemon=tape-driver
> pid=8202, daemon=elog
> pid=8699, daemon=sanmd
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.26 /usr/local/agile/bin/ipmd -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18322 ??  S       0:01.03 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.58 /usr/local/agile/bin/pm
> 20443 ??  S       0:00.12 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.17 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> pid=9368, daemon=eventd
> pid=10777, daemon=evm_cfgd
> pid=12658, daemon=snmpd
> pid=13027, daemon=ipmd
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> pid=16877, daemon=sscccc
> pid=18229, daemon=sendmail
> pid=18322, daemon=ea
> pid=18816, daemon=pm
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> 24682 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.06 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.98 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  9766 r3  S+      0:00.00 grep /usr/local/agile/bin
> # pid=24682, daemon=auth-agent
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> pid=32099, daemon=crashsaved
> 
> num=20, length=24, name=auth-agent
> kill -9 24682
> # Killing auth-agent pid 24682
> ps ax | grep auth-agent
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
>  9278 r3  R+      0:00.00 grep auth-agent (sh)
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:54:28 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.13 /usr/local/agile/bin/sdm_cfgd -r
> pid=1343, daemon=sdm_cfgd
>  2604 ??  S<      0:01.03 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=5022, daemon=cluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.16 /usr/local/agile/bin/elog -r
> pid=6146, daemon=tape-driver
> pid=8202, daemon=elog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> 13027 ??  S       0:00.28 /usr/local/agile/bin/ipmd -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> pid=8699, daemon=sanmd
> pid=9368, daemon=eventd
> pid=10777, daemon=evm_cfgd
> pid=12658, daemon=snmpd
> pid=13027, daemon=ipmd
> pid=15806, daemon=vtmd
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18322 ??  S       0:01.03 /usr/local/agile/bin/ea
> 18816 ??  Ss      0:03.64 /usr/local/agile/bin/pm
> 20443 ??  S       0:00.12 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.18 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> pid=16877, daemon=sscccc
> pid=18229, daemon=sendmail
> pid=18322, daemon=ea
> pid=18816, daemon=pm
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> pid=24910, daemon=auth-agent
> 26301 ??  S       0:00.06 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> 28773 ??  S<      0:00.99 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  9784 r4  R       0:00.00 grep /usr/local/agile/bin (sh)
> # pid=32099, daemon=crashsaved
> 
> num=14, length=24, name=ea
> kill -9 18322
> # Killing ea pid 18322
> ps ax | grep ea
> 14882 ??  S       0:00.07 /usr/local/agile/bin/ea -r
> 10271 r4  R+      0:00.00 grep ea (sh)
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:54:55 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.13 /usr/local/agile/bin/sdm_cfgd -r
> pid=1343, daemon=sdm_cfgd
>  2604 ??  S<      0:01.04 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=5022, daemon=cluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.17 /usr/local/agile/bin/elog -r
> pid=6146, daemon=tape-driver
> pid=8202, daemon=elog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 12658 ??  S       0:00.11 /usr/local/agile/bin/snmpd -f -r
> pid=8699, daemon=sanmd
> pid=9368, daemon=eventd
> pid=10777, daemon=evm_cfgd
> pid=12658, daemon=snmpd
> 13027 ??  S       0:00.31 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.07 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18816 ??  Ss      0:03.69 /usr/local/agile/bin/pm
> 20443 ??  S       0:00.13 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> pid=13027, daemon=ipmd
> pid=14882, daemon=ea
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> pid=16877, daemon=sscccc
> pid=18229, daemon=sendmail
> pid=18816, daemon=pm
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> 21333 ??  S       0:01.18 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.06 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:00.99 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 16918 r5  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> pid=24910, daemon=auth-agent
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> pid=32099, daemon=crashsaved
> 
> num=7, length=24, name=snmpd
> kill -9 12658
> # Killing snmpd pid 12658
> ps ax | grep snmpd
> 19586 ??  S       0:00.10 /usr/local/agile/bin/snmpd -f -r
>  5402 r5  S+      0:00.00 grep snmpd
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:55:22 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.13 /usr/local/agile/bin/sdm_cfgd -r
> pid=1343, daemon=sdm_cfgd
>  2604 ??  S<      0:01.04 /usr/local/agile/bin/cluster_server
>  5022 ??  S       0:00.05 /usr/local/agile/bin/cluster_relay -r
> pid=5022, daemon=cluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.17 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
> pid=6146, daemon=tape-driver
> pid=8202, daemon=elog
> pid=8699, daemon=sanmd
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> pid=9368, daemon=eventd
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 13027 ??  S       0:00.34 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.07 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> pid=10777, daemon=evm_cfgd
> pid=13027, daemon=ipmd
> pid=14882, daemon=ea
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18816 ??  Ss      0:03.73 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.10 /usr/local/agile/bin/snmpd -f -r
> pid=16585, daemon=sshd
> pid=16877, daemon=sscccc
> pid=18229, daemon=sendmail
> pid=18816, daemon=pm
> pid=19586, daemon=snmpd
> 20443 ??  S       0:00.13 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.18 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.06 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> pid=24910, daemon=auth-agent
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> 28773 ??  S<      0:00.99 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  1967 r6  S+      0:00.00 grep /usr/local/agile/bin
> # pid=32099, daemon=crashsaved
> 
> num=1, length=24, name=cluster_relay
> kill -9 5022
> # Killing cluster_relay pid 5022
> ps ax | grep cluster_relay
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
> 18374 r6  S+      0:00.00 grep cluster_relay
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:55:48 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.13 /usr/local/agile/bin/sdm_cfgd -r
> pid=1343, daemon=sdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
>  2604 ??  S<      0:01.04 /usr/local/agile/bin/cluster_server
> pid=2071, daemon=cluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.17 /usr/local/agile/bin/elog -r
> pid=6146, daemon=tape-driver
> pid=8202, daemon=elog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.07 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.08 /usr/local/agile/bin/evm_cfgd -r
> 13027 ??  S       0:00.36 /usr/local/agile/bin/ipmd -r
> pid=8699, daemon=sanmd
> pid=9368, daemon=eventd
> pid=10777, daemon=evm_cfgd
> pid=13027, daemon=ipmd
> 14882 ??  S       0:00.07 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.22 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> pid=14882, daemon=ea
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> pid=16877, daemon=sscccc
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18816 ??  Ss      0:03.75 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.10 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:00.14 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21333 ??  S       0:01.19 /usr/local/agile/bin/spm
> 21489 ??  S       0:00.04 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.06 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.00 /usr/local/agile/bin/cluster_contrl
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 19561 r7  S+      0:00.00 grep /usr/local/agile/bin
> # pid=18229, daemon=sendmail
> pid=18816, daemon=pm
> pid=19586, daemon=snmpd
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21333, daemon=spm
> pid=21489, daemon=ndmp_cfgd
> pid=24910, daemon=auth-agent
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> pid=32099, daemon=crashsaved
> 
> num=18, length=24, name=spm
> kill -9 21333
> # Killing spm pid 21333
> ps ax | grep spm
> 31173 ??  S       0:00.06 /usr/local/agile/bin/spm -r
>  4901 r7  R+      0:00.00 grep spm (sh)
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:56:15 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.43 /usr/local/agile/bin/sdm_cfgd -r
> pid=1343, daemon=sdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
>  2604 ??  D<      0:01.06 /usr/local/agile/bin/cluster_server
> pid=2071, daemon=cluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.50 /usr/local/agile/bin/elog -r
> pid=6146, daemon=tape-driver
> pid=8202, daemon=elog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.19 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.49 /usr/local/agile/bin/evm_cfgd -r
> pid=8699, daemon=sanmd
> pid=9368, daemon=eventd
> pid=10777, daemon=evm_cfgd
> 13027 ??  S       0:00.39 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.41 /usr/local/agile/bin/ncmd -r
> pid=13027, daemon=ipmd
> pid=14882, daemon=ea
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18229 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 18816 ??  Ss      0:03.79 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.16 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.91 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> pid=16585, daemon=sshd
> pid=16877, daemon=sscccc
> pid=18229, daemon=sendmail
> pid=18816, daemon=pm
> pid=19586, daemon=snmpd
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> 21489 ??  S       0:00.07 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.08 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.12 /usr/local/agile/bin/cluster_contrl
> 31173 ??  S       0:00.25 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  4052 r8  R+      0:00.01 grep /usr/local/agile/bin
> # pid=21489, daemon=ndmp_cfgd
> pid=24910, daemon=auth-agent
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> pid=31173, daemon=spm
> pid=32099, daemon=crashsaved
> 
> num=13, length=24, name=sendmail
> kill -9 18229
> # Killing sendmail pid 18229
> ps ax | grep sendmail
> 25995 ??  S       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
>  3852 r8  R+      0:00.00 grep sendmail (sh)
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:56:42 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.43 /usr/local/agile/bin/sdm_cfgd -r
> pid=1343, daemon=sdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
>  2604 ??  S<      0:01.07 /usr/local/agile/bin/cluster_server
> pid=2071, daemon=cluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.52 /usr/local/agile/bin/elog -r
> pid=6146, daemon=tape-driver
> pid=8202, daemon=elog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.19 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10777 ??  S       0:01.49 /usr/local/agile/bin/evm_cfgd -r
> 13027 ??  S       0:00.40 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> pid=8699, daemon=sanmd
> pid=9368, daemon=eventd
> pid=10777, daemon=evm_cfgd
> pid=13027, daemon=ipmd
> pid=14882, daemon=ea
> 15806 ??  S       0:00.06 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.41 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.49 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18816 ??  Ss      0:03.84 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.16 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.95 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> pid=16877, daemon=sscccc
> pid=18816, daemon=pm
> pid=19586, daemon=snmpd
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> 21489 ??  S       0:00.07 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 25995 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.08 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> pid=21489, daemon=ndmp_cfgd
> pid=24910, daemon=auth-agent
> pid=25995, daemon=sendmail
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> 28773 ??  S<      0:01.18 /usr/local/agile/bin/cluster_contrl
> 31173 ??  S       0:00.25 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  9307 r9  S+      0:00.00 grep /usr/local/agile/bin
> # pid=31173, daemon=spm
> pid=32099, daemon=crashsaved
> 
> num=6, length=24, name=evm_cfgd
> kill -9 10777
> # Killing evm_cfgd pid 10777
> ps ax | grep evm_cfgd
>   718 ??  S       0:00.07 /usr/local/agile/bin/evm_cfgd -r
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:57:09 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>   718 ??  S       0:00.07 /usr/local/agile/bin/evm_cfgd -r
> pid=718, daemon=evm_cfgd
>  1343 ??  S       0:00.43 /usr/local/agile/bin/sdm_cfgd -r
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
> pid=1343, daemon=sdm_cfgd
> pid=2071, daemon=cluster_relay
>  2604 ??  S<      0:01.07 /usr/local/agile/bin/cluster_server
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
> pid=6146, daemon=tape-driver
>  8202 ??  S       0:00.56 /usr/local/agile/bin/elog -r
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.19 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 13027 ??  S       0:00.40 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.07 /usr/local/agile/bin/vtmd -r
> pid=8202, daemon=elog
> pid=8699, daemon=sanmd
> pid=9368, daemon=eventd
> pid=13027, daemon=ipmd
> pid=14882, daemon=ea
> pid=15806, daemon=vtmd
> 15926 ??  S       0:00.49 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.50 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18816 ??  Ss      0:03.87 /usr/local/agile/bin/pm
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> pid=16877, daemon=sscccc
> pid=18816, daemon=pm
> 19586 ??  S       0:00.16 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.95 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21489 ??  S       0:00.07 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 25995 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26301 ??  S       0:00.09 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.18 /usr/local/agile/bin/cluster_contrl
> 31173 ??  S       0:00.28 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  7912 ra  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=19586, daemon=snmpd
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21489, daemon=ndmp_cfgd
> pid=24910, daemon=auth-agent
> pid=25995, daemon=sendmail
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> pid=31173, daemon=spm
> pid=32099, daemon=crashsaved
> 
> num=0, length=24, name=evm_cfgd
> kill -9 718
> # Killing evm_cfgd pid 718
> ps ax | grep evm_cfgd
> 26078 ??  S       0:00.10 /usr/local/agile/bin/evm_cfgd -r
>  1763 ra  R+      0:00.00 grep evm_cfgd (sh)
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:57:36 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.43 /usr/local/agile/bin/sdm_cfgd -r
> pid=1343, daemon=sdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
> pid=2071, daemon=cluster_relay
>  2604 ??  S<      0:01.08 /usr/local/agile/bin/cluster_server
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.57 /usr/local/agile/bin/elog -r
> pid=6146, daemon=tape-driver
> pid=8202, daemon=elog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.19 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 13027 ??  S       0:00.40 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> pid=8699, daemon=sanmd
> pid=9368, daemon=eventd
> pid=13027, daemon=ipmd
> pid=14882, daemon=ea
> 15806 ??  S       0:00.07 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.49 /usr/local/agile/bin/ncmd -r
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> 16585 ??  Ss      0:00.51 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18816 ??  Ss      0:03.93 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.16 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.95 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21489 ??  S       0:00.07 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 25995 ??  I       0:00.10 /usr/local/agile/bin/sendmail -Q30m -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26078 ??  S       0:00.10 /usr/local/agile/bin/evm_cfgd -r
> pid=16585, daemon=sshd
> pid=16877, daemon=sscccc
> pid=18816, daemon=pm
> pid=19586, daemon=snmpd
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21489, daemon=ndmp_cfgd
> pid=24910, daemon=auth-agent
> pid=25995, daemon=sendmail
> pid=26078, daemon=evm_cfgd
> 26301 ??  S       0:00.09 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.18 /usr/local/agile/bin/cluster_contrl
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> 31173 ??  S       0:00.28 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 17773 rb  R+      0:00.01 grep /usr/local/agile/bin
> # pid=31173, daemon=spm
> pid=32099, daemon=crashsaved
> 
> num=18, length=24, name=sendmail
> kill -9 25995
> # Killing sendmail pid 25995
> ps ax | grep sendmail
> 10969 ??  S       0:00.12 /usr/local/agile/bin/sendmail -Q30m -r
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:58:03 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.44 /usr/local/agile/bin/sdm_cfgd -r
> pid=1343, daemon=sdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
>  2604 ??  S<      0:01.08 /usr/local/agile/bin/cluster_server
> pid=2071, daemon=cluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.58 /usr/local/agile/bin/elog -r
> pid=6146, daemon=tape-driver
> pid=8202, daemon=elog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.20 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10969 ??  I       0:00.12 /usr/local/agile/bin/sendmail -Q30m -r
> 13027 ??  S       0:00.40 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> pid=8699, daemon=sanmd
> pid=9368, daemon=eventd
> pid=10969, daemon=sendmail
> pid=13027, daemon=ipmd
> pid=14882, daemon=ea
> 15806 ??  S       0:00.07 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.57 /usr/local/agile/bin/ncmd -r
> 16585 ??  Ss      0:00.51 /usr/local/agile/bin/sshd
> 16877 ??  S       0:00.08 /usr/local/agile/bin/sscccc -r
> 18816 ??  Ss      0:03.98 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.17 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.95 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21489 ??  S       0:00.08 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26078 ??  S       0:00.12 /usr/local/agile/bin/evm_cfgd -r
> 26301 ??  S       0:00.09 /usr/local/agile/bin/asd -r
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> pid=16585, daemon=sshd
> pid=16877, daemon=sscccc
> pid=18816, daemon=pm
> pid=19586, daemon=snmpd
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21489, daemon=ndmp_cfgd
> pid=24910, daemon=auth-agent
> pid=26078, daemon=evm_cfgd
> pid=26301, daemon=asd
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.19 /usr/local/agile/bin/cluster_contrl
> 31173 ??  S       0:00.30 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>  2061 rc  R+      0:00.00 grep /usr/local/agile/bin (sh)
> pid=28664, daemon=timekeeper
> pid=31173, daemon=spm
> pid=32099, daemon=crashsaved
> #
> num=12, length=24, name=sscccc
> kill -9 16877
> # Killing sscccc pid 16877
> ps ax | grep sscccc
> 18080 ??  S       0:00.10 /usr/local/agile/bin/sscccc -r
> 20149 rc  R+      0:00.00 grep sscccc (sh)
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:58:30 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.44 /usr/local/agile/bin/sdm_cfgd -r
> pid=1343, daemon=sdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
>  2604 ??  S<      0:01.08 /usr/local/agile/bin/cluster_server
> pid=2071, daemon=cluster_relay
>  6146 ??  S       0:00.05 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.59 /usr/local/agile/bin/elog -r
> pid=6146, daemon=tape-driver
> pid=8202, daemon=elog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.20 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> 10969 ??  I       0:00.12 /usr/local/agile/bin/sendmail -Q30m -r
> 13027 ??  S       0:00.40 /usr/local/agile/bin/ipmd -r
> pid=8699, daemon=sanmd
> pid=9368, daemon=eventd
> pid=10969, daemon=sendmail
> pid=13027, daemon=ipmd
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.07 /usr/local/agile/bin/vtmd -r
> 15926 ??  S       0:00.57 /usr/local/agile/bin/ncmd -r
> pid=14882, daemon=ea
> pid=15806, daemon=vtmd
> pid=15926, daemon=ncmd
> 16585 ??  Ss      0:00.52 /usr/local/agile/bin/sshd
> 18080 ??  S       0:00.10 /usr/local/agile/bin/sscccc -r
> 18816 ??  Ss      0:04.01 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.17 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.97 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21489 ??  S       0:00.08 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26078 ??  S       0:00.12 /usr/local/agile/bin/evm_cfgd -r
> 26301 ??  S       0:00.09 /usr/local/agile/bin/asd -r
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.19 /usr/local/agile/bin/cluster_contrl
> 31173 ??  S       0:00.30 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
>   205 rd  R+      0:00.00 grep /usr/local/agile/bin (sh)
> pid=16585, daemon=sshd
> pid=18080, daemon=sscccc
> pid=18816, daemon=pm
> pid=19586, daemon=snmpd
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21489, daemon=ndmp_cfgd
> pid=24910, daemon=auth-agent
> pid=26078, daemon=evm_cfgd
> pid=26301, daemon=asd
> pid=28664, daemon=timekeeper
> pid=31173, daemon=spm
> pid=32099, daemon=crashsaved
> #
> num=6, length=24, name=sendmail
> kill -9 10969
> # Killing sendmail pid 10969
> ps ax | grep sendmail
> 11605 ??  S       0:00.09 /usr/local/agile/bin/sendmail -Q30m -r
> ***** Result is true
> 
> spawn ssh root@10.2.1.18
> root@10.2.1.18's password:
> Last login: Wed May  9 16:58:57 2007 from 10.0.0.94
> OpenBSD 2.8 (BOBCAT) #0: Tue May  1 15:40:01 PDT 2007
> 
> Terminal type? [xterm] xterm
> Don't login as root, use su
> eng60# sh
> # ps ax | grep /usr/local/agile/bin
>  1343 ??  S       0:00.76 /usr/local/agile/bin/sdm_cfgd -r
> pid=1343, daemon=sdm_cfgd
>  2071 ??  S       0:00.06 /usr/local/agile/bin/cluster_relay -r
> pid=2071, daemon=cluster_relay
>  2604 ??  S<      0:01.10 /usr/local/agile/bin/cluster_server
>  6146 ??  S       0:00.06 /usr/local/agile/bin/tape-driver -r
>  8202 ??  S       0:00.65 /usr/local/agile/bin/elog -r
> pid=6146, daemon=tape-driver
> pid=8202, daemon=elog
>  8699 ??  S       0:00.08 /usr/local/agile/bin/sanmd -r
>  9368 ??  S       0:00.23 /usr/local/agile/bin/eventd -r
> 10048 ??  S<      0:00.30 /usr/local/agile/bin/cluster_contrl
> pid=8699, daemon=sanmd
> pid=9368, daemon=eventd
> 11605 ??  I       0:00.09 /usr/local/agile/bin/sendmail -Q30m -r
> 13027 ??  S       0:00.40 /usr/local/agile/bin/ipmd -r
> 14882 ??  S       0:00.14 /usr/local/agile/bin/ea -r
> 15806 ??  S       0:00.07 /usr/local/agile/bin/vtmd -r
> pid=11605, daemon=sendmail
> pid=13027, daemon=ipmd
> pid=14882, daemon=ea
> pid=15806, daemon=vtmd
> 15926 ??  S       0:00.79 /usr/local/agile/bin/ncmd -r
> pid=15926, daemon=ncmd
> 16585 ??  Ss      0:00.52 /usr/local/agile/bin/sshd
> 18080 ??  S       0:00.10 /usr/local/agile/bin/sscccc -r
> 18816 ??  Ss      0:04.09 /usr/local/agile/bin/pm
> 19586 ??  S       0:00.17 /usr/local/agile/bin/snmpd -f -r
> 20443 ??  S       0:12.98 /usr/local/agile/bin/vsd -r
> 20929 ??  S       0:00.19 /usr/local/agile/bin/chassisd -r
> 21489 ??  S       0:00.09 /usr/local/agile/bin/ndmp_cfgd -r
> 24910 ??  S       0:00.07 /usr/local/agile/bin/auth-agent -r
> pid=16585, daemon=sshd
> pid=18080, daemon=sscccc
> pid=18816, daemon=pm
> pid=19586, daemon=snmpd
> pid=20443, daemon=vsd
> pid=20929, daemon=chassisd
> pid=21489, daemon=ndmp_cfgd
> pid=24910, daemon=auth-agent
> 26024 ??  I       0:00.02 /usr/local/agile/bin/registryMgr
> 26078 ??  S       0:00.42 /usr/local/agile/bin/evm_cfgd -r
> 26301 ??  S       0:00.09 /usr/local/agile/bin/asd -r
> pid=26078, daemon=evm_cfgd
> pid=26301, daemon=asd
> 28664 ??  S       0:00.07 /usr/local/agile/bin/timekeeper
> 28773 ??  S<      0:01.25 /usr/local/agile/bin/cluster_contrl
> 31173 ??  S       0:00.44 /usr/local/agile/bin/spm -r
> 32099 ??  I       0:00.04 /usr/local/agile/bin/crashsaved
> 22881 re  R+      0:00.00 grep /usr/local/agile/bin (sh)
> # pid=28664, daemon=timekeeper
> pid=31173, daemon=spm
> pid=32099, daemon=crashsaved
> 
> num=23, length=24, name=crashsaved
> kill -9 32099
> # Killing crashsaved pid 32099
> ***** Result is false
> 
> >>> Result is false
> 
> [sandrineb@sandrineb cluster]$
> 
> 
> 
> Release_Project: Delorean
> 
> 
