X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C89675.5D9BE8D0@onstor-exch02.onstor.net>; Fri, 4 Apr 2008 10:00:20 -0700
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
Subject: RE: Submittal 16 Beta Release Candidate availble
Date: Fri, 4 Apr 2008 10:00:20 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E05C74632@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E12BFD7@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Submittal 16 Beta Release Candidate availble
Thread-Index: AciWZ+gnjyM7a7VTRX68hON2Pa1e2gAADuIKAADaaa8AAmoKkA==
From: "Sandrine Boulanger" <sandrine.boulanger@onstor.com>
To: "Narain Ramadass" <narain.ramadass@onstor.com>,
	"Larry Scheer" <larry.scheer@onstor.com>,
	"Sripal Surendiran (HCL)" <sripal.surendiran@onstor.com>
Cc: "Andy Sharp" <andy.sharp@onstor.com>,
	"Manohar Divate" <manohar.divate@onstor.com>,
	"Rendell Fong" <rendell.fong@onstor.com>,
	"dl-Cougar" <dl-Cougar@onstor.com>,
	"dl-QA" <dl-qa@onstor.com>

Yes, May and Durai also ran into it after a system upgrade to secondary
flash.
May also lost the config for her sc2 interface and Durai lost the
default route on ssc.

-----Original Message-----
From: Narain Ramadass=20
Sent: Friday, April 04, 2008 8:50 AM
To: Larry Scheer; Sripal Surendiran (HCL)
Cc: Sandrine Boulanger; Andy Sharp; Manohar Divate; Rendell Fong;
dl-Cougar; dl-QA
Subject: RE: Submittal 16 Beta Release Candidate availble

The hostname getting hosed seems to be a common problem with system
upgrade. Jobi's filer has the same problem. Need to figure that out
soon.


-----Original Message-----
From: Larry Scheer
Sent: Fri 4/4/2008 8:35 AM
To: Sripal Surendiran (HCL); Narain Ramadass
Cc: Sandrine Boulanger; Andy Sharp; Manohar Divate; Rendell Fong;
dl-Cougar; dl-QA
Subject: RE: Submittal 16 Beta Release Candidate availble
=20
Sripal,
  Thanks for this summary. It helps with understanding the behavior we
are seeing.
Sandrine's machine was running submittal 15 already configured. She ran
system upgrade to upgrade the primary flash. When the system was
rebooted the hostname was changed to localhost. Because of that ncmd
would not start.

What scenarios would cause the hostname to change when
/onstor/etc/nasgwayinit.status is not present and one SC interface is
configured?=20

Something mis-fired in this case and we need to find the root cause.

Thanks,

Larry


-----Original Message-----
From: Sripal Surendiran (HCL)
Sent: Fri 4/4/2008 8:23 AM
To: Narain Ramadass
Cc: Sandrine Boulanger; Andy Sharp; Manohar Divate; Rendell Fong; Larry
Scheer; dl-Cougar; dl-QA
Subject: Re: Submittal 16 Beta Release Candidate availble
=20
Narain,

1. /onstor/etc/nasgwayinit.conf file contains information obtained from
the DHCP server during boot up. When nfxsh prompts the user with FTI
question, these values are read by Initial configuration API and shown
to the user as default value.

2. /onstor/etc/nasgwayinit.status contains the current status of the
filer (Initialized || Uninitialized).=20

3. When the user upgrades to the new Initial configuration, running
nfxsh checks the contents of the /onstor/etc/nasgwayinit.status. If the
/onstor/etc/nasgwayinit.status is not present, it checks to see at least
one of the SC interface is configured before deciding that the filer is
in uninitialized state. If at least one SC interface is configured, then
nfxsh will create a dummy /onstor/etc/nasgwayinit.conf with the
information available from configuration file in the filer.

4. It also sets the filer state to *Initialized* in
/onstor/etc/nasgwayinit.status file.

5. Since Sandrine filer is already in initialized
state(nasgwayinit.status file containing Initialized entry), removing
/onstor/etc/nasgwayinit.conf file from filer FTI doesn't care about
recreating it.

-Sripal

Narain Ramadass wrote:=20

	Sripal,
=09
	Please investigate the defect Sandrine reports about the conf
file not being created when the filer is configured.
=09
	Once again: as per design:-
=09
	When the filer boots and the conf/semaphore files are not
present, but the filer is configured for SC1 or SC2 (maybe there is
another condition here that I missed?) the FTI screen will not be thrown
and the conf/semaphore files would be created.
=09
	If this is not the behavior on Cougar - think we have a bug
there.
=09
	Narain.
=09
=09
	-----Original Message-----
	From: Sandrine Boulanger
	Sent: Thu 4/3/2008 6:32 PM
	To: Andy Sharp; Manohar Divate
	Cc: Rendell Fong; Narain Ramadass; Larry Scheer; dl-Cougar;
dl-QA
	Subject: RE: Submittal 16 Beta Release Candidate availble
=09
	localhost:/onstor/conf# vi /etc/hostname =3D> changed local host
to g14r10
=09
	localhost:/onstor/conf# rm -rf /onstor/conf/nasgwayinit.conf =3D>
I removed it since Naraian said it would be created if not there when at
least one of the sc interfaces is configured
=09
	localhost:/onstor/conf# vi /etc/network/interfaces =3D> commented
out the 2 lines
=09
=09
=09
	reboot
=09
=09
=09
	now the system is up, but /onstor/conf/nasgwayinit.conf has not
been created (!)
=09
=09
=09
=09
=09
	Got this error for the core volume BTW:
=09
	Apr  3 18:25:06 g14r10 : 1:2:coredump:INFO: 52: core dump state:
on - volume: coreg14
=09
	Apr  3 18:25:06 g14r10 : 0:0:evm:NOTICE: evm_closeLunReq: SDM
error[-1] for close of lunId[0x0], rc[-1]
=09
	Apr  3 18:25:06 g14r10 : 0:0:evm:INFO: evm_closeAllLunsInVolume:
Cannot close all luns for volume[coreg14], rc[-5593]
=09
=09
=09
	-----Original Message-----
	From: Andy Sharp
	Sent: Thursday, April 03, 2008 6:10 PM
	To: Manohar Divate
	Cc: Sandrine Boulanger; Rendell Fong; Narain Ramadass; Larry
Scheer; dl-Cougar; dl-QA
	Subject: Re: Submittal 16 Beta Release Candidate availble
=09
=09
=09
	Manny is of course, quite correct: always listen to me.
=09
=09
=09
	I've been poking around on Sandrine's system, g14r10, and there
is more
=09
	than a little weirdness.  Here's the scoop: it was an _system
upgrade_
=09
	to primary flash; upgrading from a debug build to an opt build;
with the
=09
	new FTI code merged in; with SC2 not configured.  What could go
wrong
=09
	with that?
=09
=09
=09
	I believe this is a known bug, actually, and a fix for it is
pending.
=09
	It might be combining with the bug Rendell mentioned which I'm
not
=09
	familiar with (is there a bug filed?).  The known bug I'm
referring to
=09
	isn't necessarily all that known when doing a system upgrade,
but it is
=09
	known to be a problem if you do a flash_install, followed by a
"copy
=09
	config from standby" and you don't have SC2 configured, although
it's
=09
	rumoured to only affect bobcat/bsd.
=09
=09
=09
	Also, there is some bogasity in the /etc/network/interfaces
file:
=09
=09
=09
	# The secondary network interface
=09
	auto eth1
=09
	iface eth1 inet static
=09
	  address address
=09
	  netmask netmask
=09
=09
=09
	The 'address', 'netmask' and 'auto eth1' lines should be
commented out.
=09
=09
=09
	Go ahead and fix the interfaces file, and fix the /etc/hostname
=09
	and /onstor/conf/nasgwayinit.conf files and see if the filer can
resume
=09
	operation.
=09
=09
=09
=09
=09
	On Thu, 3 Apr 2008 17:53:56 -0700 "Manohar Divate"
=09
	<manohar.divate@onstor.com> <mailto:manohar.divate@onstor.com>
wrote:
=09
=09
=09
	> Not me , All clear and looks everything OK on my 2 node
cluster after
=09
	> flash install
=09
	>
=09
	> listen to Andy (do a flash install)
=09
	>
=09
	>
=09
	>
=09
	>
=09
	>
=09
	> -Manny
=09
	>
=09
	>
=09
	> You are good to go if you use this procedure
=09
	>
=09
	> Do a NfXSH>system copy init
=09
	> Nfxsh>exit
=09
	>
=09
	> #mount -r
=09
	>
10.0.0.236:/nx-d_buildup/build-trees/R4.0.0.0/R4.0.0.0-040308/nfx-tree/B
=09
	> uild/cg/opt/Release /mnt
=09
	>
=09
	> #cd /mnt/etc
=09
	> #./flash_install.sh /mnt 0 or 1 depending on you secondary
=09
	>
=09
	> #Answer N for FIT
=09
	> #Answer Y for copy for upgrade
=09
	>
=09
	> Nfxsh>System reboot -s -f -y
=09
	>
=09
	>
=09
	>
=09
	> _____________________________________________
=09
	> From: Sandrine Boulanger
=09
	> Sent: Thursday, April 03, 2008 5:44 PM
=09
	> To: Rendell Fong; Narain Ramadass; Larry Scheer; dl-Cougar;
dl-QA
=09
	> Subject: RE: Submittal 16 Beta Release Candidate availble
=09
	>
=09
	> Yes, looks like everyone upgrading to sub#16 who does not have
this
=09
	> file will end up with the same problem.
=09
	> I'll file a defect.
=09
	>
=09
	> _____________________________________________
=09
	> From: Rendell Fong
=09
	> Sent: Thursday, April 03, 2008 5:17 PM
=09
	> To: Sandrine Boulanger; Narain Ramadass; Larry Scheer;
dl-Cougar;
=09
	> dl-QA Subject: RE: Submittal 16 Beta Release Candidate
availble
=09
	>
=09
	> The /onstor/conf/nasgwayinit.conf file is not configured so it
may be
=09
	> contributing to the confusion.
=09
	> I had to manually edit this file on my filer after doing a
system
=09
	> upgrade recently.
=09
	>
=09
	> localhost:/onstor/conf# cat nasgwayinit.conf
=09
	> SC1_IP:
=09
	> SC1_NETMASK:
=09
	> SC2_IP:
=09
	> SC2_NETMASK:
=09
	> GATEWAY:
=09
	> NTP:0.debian.pool.ntp.or
=09
	> HOST_NAME:localhost
=09
	> DOMAIN_NAME:
=09
	> RESOLVER:
=09
	> BOOT_TYPE:STATIC
=09
	>
=09
	> _____________________________________________
=09
	> From: Sandrine Boulanger
=09
	> Sent: Thursday, April 03, 2008 5:12 PM
=09
	> To: Narain Ramadass; Larry Scheer; dl-Cougar; dl-QA
=09
	> Subject: RE: Submittal 16 Beta Release Candidate availble
=09
	>
=09
	> Looks like after the reboot, the name of the filer was changed
to
=09
	> localhost, and that's what ncmd is tripping on, because it's
different
=09
	> from the name in cluster.conf
=09
	>
=09
	> Apr  3 16:28:00 g14r10 pm: pm_terminate: child 1257
(/onstor/bin/asd)
=09
	> terminated
=09
	> Apr  3 16:28:01 g14r10 pm: pm_terminate: child 1256
=09
	> (/onstor/bin/snmpd) terminated
=09
	> Apr  3 16:28:01 g14r10 pm: pm_terminate: child 1100
=09
	> (/onstor/bin/cluster_server) terminated
=09
	> Apr  3 16:28:01 g14r10 pm: pm_terminate: child 1099
=09
	> (/onstor/bin/evm_cfgd) terminated
=09
	> Apr  3 16:28:02 g14r10 pm: pm_terminate: child 1069
=09
	> (/onstor/bin/sdm_cfgd) terminated
=09
	> Apr  3 16:28:02 g14r10 pm: pm_terminate: child 1068
=09
	> (/onstor/bin/chassisd) terminated
=09
	> Apr  3 16:28:02 g14r10 pm: pm_terminate: child 1067
=09
	> (/onstor/bin/timekeeper) terminated
=09
	> Apr  3 16:28:02 g14r10 pm: pm_terminate: child 1066
=09
	> (/onstor/bin/eventd) terminated
=09
	> Apr  3 16:28:02 g14r10 pm: pm_terminate: child 1046
(/onstor/bin/ncmd)
=09
	> terminated
=09
	> Apr  3 16:28:02 g14r10 pm: pm_terminate: child 1036
(/onstor/bin/elog)
=09
	> terminated
=09
	> Apr  3 16:30:27 localhost pm: /onstor/bin/elog: finished
=09
	> initialization. Apr  3 16:30:28 localhost :
=09
	> 0:0:pm:INFO: /onstor/bin/ncmd: finished initialization.
=09
	> Apr  3 16:30:29 localhost : 0:0:pm:ERROR: pm_sig_handler:
=09
	> /onstor/bin/ncmd (pid 987) terminated with signal Aborted
=09
	> Apr  3 16:30:29 localhost : 0:0:pm:INFO: /onstor/bin/eventd:
finished
=09
	> initialization.
=09
	> Apr  3 16:30:30 localhost : 0:0:pm:INFO: /onstor/bin/ncmd:
finished
=09
	> initialization.
=09
	> Apr  3 16:30:31 localhost : 0:0:pm:ERROR: pm_sig_handler:
=09
	> /onstor/bin/ncmd (pid 1000) terminated with signal Aborted
=09
	> Apr  3 16:30:31 localhost : 0:0:pm:INFO:
/onstor/bin/timekeeper:
=09
	> finished initialization.
=09
	> Apr  3 16:30:31 localhost : 0:0:pm:INFO: /onstor/bin/ncmd:
finished
=09
	> initialization.
=09
	> Apr  3 16:30:31 localhost : 0:0:pm:INFO: /onstor/bin/chassisd:
=09
	> finished initialization.
=09
	> Apr  3 16:32:02 localhost : 0:0:cm:NOTICE: CHASSISD: Chassis
Manager:
=09
	> Started
=09
	>
=09
	> _____________________________________________
=09
	> From: Narain Ramadass
=09
	> Sent: Thursday, April 03, 2008 5:00 PM
=09
	> To: Sandrine Boulanger; Larry Scheer; dl-Cougar; dl-QA
=09
	> Subject: RE: Submittal 16 Beta Release Candidate availble
=09
	>
=09
	> Dont know for sure - but this may be a problem with the pmtab
having
=09
	> entries for cluster_contrl?
=09
	>
=09
	> Thought it was automatically started after a recent change
from Chris.
=09
	> Maybe the merge from the FTI branch to dev messed this up. Can
you
=09
	> manually uncomment this in your pmtab and retry Sandrine? The
system
=09
	> does not come up because pm is waiting for cluster_contrl to
ack
=09
	> (which it never will).
=09
	>
=09
	> Thanx,
=09
	> Narain.
=09
	>
=09
	> _____________________________________________
=09
	> From:     Sandrine Boulanger=20
=09
	> Sent:     Thursday, April 03, 2008 4:50 PM
=09
	> To: Sandrine Boulanger; Larry Scheer; dl-Cougar; dl-QA
=09
	> Subject:  RE: Submittal 16 Beta Release Candidate availble
=09
	>
=09
	> Looks like it's crashed too, but does not show in the log
=09
	>
=09
	> localhost diag> system show chassis
=09
	>
=09
	>  module     cpu         state
=09
	> ----------------------------------------------
=09
	>  SSC        SSC         UP
=09
	>  NFPNIM     TXRX0       DOWN
=09
	>             TXRX1       DOWN
=09
	>             FP0         DOWN
=09
	>             FP1         DOWN
=09
	>             FP2         DOWN
=09
	>             FP3         DOWN
=09
	> ----------------------------------------------
=09
	>
=09
	> _____________________________________________
=09
	> From: Sandrine Boulanger
=09
	> Sent: Thursday, April 03, 2008 4:48 PM
=09
	> To: Larry Scheer; dl-Cougar; dl-QA
=09
	> Subject: RE: Submittal 16 Beta Release Candidate availble
=09
	>
=09
	> I upgraded this single node to sub#16 without issue, but then
I ran
=09
	> into a crash, and after reboot this is what I get.
=09
	> How come ncmd keeps initializing and it does not get past
=09
	> cluster_contrl?
=09
	>
=09
	> localhost:~# ps ax | grep onstor
=09
	>   826 ?        Ss     0:00 /onstor/bin/sshd -u0
=09
	>   968 ?        Ss     0:00 /onstor/bin/pm
=09
	>   974 ?        S      0:02 /onstor/bin/elog
=09
	>   992 ?        S      0:00 /onstor/bin/eventd
=09
	>  1001 ?        S      0:00 /onstor/bin/timekeeper
=09
	>  1016 ?        S      0:00 /onstor/bin/chassisd
=09
	>  1024 ?        S      0:00 /onstor/bin/sdm_cfgd
=09
	>  1034 ?        S      0:01 /onstor/bin/evm_cfgd
=09
	>  1037 ?        S<     0:00 /onstor/bin/cluster_server
=09
	>  1038 ?        R<    13:25 /onstor/bin/cluster_contrl
=09
	>  1039 ?        S<     0:00 /onstor/bin/cluster_contrl
=09
	>  1636 ?        Ss     0:00 /bin/sh /onstor/bin/emrscron -g
h_res_stats
=09
	>  1683 pts/0    R+     0:00 grep onstor
=09
	> localhost:~#
=09
	>
=09
	>
=09
	> Apr  3 16:30:27 localhost syslogd 1.4.1#18: restart.
=09
	> Apr  3 16:30:27 localhost pm: /onstor/bin/elog: finished
=09
	> initialization. Apr  3 16:30:28 localhost kernel: fp1: 9:
=09
	> ispfc:ISPFC_CS_NSDB_CHANGE,[8015] on port [10d00]
=09
	> Apr  3 16:30:28 localhost : 0:0:pm:INFO: /onstor/bin/ncmd:
finished
=09
	> initialization.
=09
	> Apr  3 16:30:29 localhost : 0:0:pm:INFO: /onstor/bin/eventd:
finished
=09
	> initialization.
=09
	> Apr  3 16:30:30 localhost : 0:0:pm:INFO: /onstor/bin/ncmd:
finished
=09
	> initialization.
=09
	> Apr  3 16:30:31 localhost : 0:0:pm:INFO:
/onstor/bin/timekeeper:
=09
	> finished initialization.
=09
	> Apr  3 16:30:31 localhost : 0:0:pm:INFO: /onstor/bin/ncmd:
finished
=09
	> initialization.
=09
	> Apr  3 16:30:31 localhost : 0:0:pm:INFO: /onstor/bin/chassisd:
=09
	> finished initialization.
=09
	> Apr  3 16:30:32 localhost kernel: fp2: 10: ispfc: ispfc:sp1.1
=09
	> Fibrechannel link now online
=09
	> Apr  3 16:30:32 localhost kernel: fp3: 11:
=09
	> ispfc:ISPFC_CS_NSDB_CHANGE,[8014] LoopID_LoginState [10004]
=09
	> Apr  3 16:30:50 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:31:18 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:31:49 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:32:02 localhost : 0:0:cm:NOTICE: CHASSISD: Chassis
Manager:
=09
	> Started
=09
	> Apr  3 16:32:02 localhost : 0:0:pm:INFO: /onstor/bin/sdm_cfgd:
=09
	> finished initialization.
=09
	> Apr  3 16:32:02 localhost : 0:0:sdm:INFO: ONStor Storage
Device
=09
	> Manager (c)2003: Started
=09
	> Apr  3 16:32:02 localhost : 0:0:pm:INFO: /onstor/bin/ncmd:
finished
=09
	> initialization.
=09
	> Apr  3 16:32:03 localhost : 0:0:pm:INFO: /onstor/bin/evm_cfgd:
=09
	> finished initialization.
=09
	> Apr  3 16:32:03 localhost : 0:0:pm:INFO: /onstor/bin/ncmd:
finished
=09
	> initialization.
=09
	> Apr  3 16:32:03 localhost : 0:0:cluster2:INFO:
ClusterCtrl_InitUbik:
=09
	> connection opened e0a020a
=09
	> Apr  3 16:32:03 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : SDM
session
=09
	> open success
=09
	> Apr  3 16:32:04 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to cluster2, flags
820a,
=09
	> name evm, rc -20, retrying... Apr  3 16:32:04 localhost :
=09
	> 0:0:cluster2:INFO: cluster_lookup_sess: fail to open sess for
app
=09
	> cluster2, rc -20 Apr  3 16:32:04 localhost : 0:0:evm:INFO:
=09
	> evm_rcvRmcMsg : NCM session open failed
=09
	> Apr  3 16:32:04 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:32:05 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name elog, rc -20, retrying...
=09
	> Apr  3 16:32:05 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:32:05 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:32:05 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:32:05 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:32:05 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:32:06 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:32:06 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:32:06 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:32:06 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:32:06 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:32:06 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:32:06 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:32:06 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:32:06 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name sdm, rc -20, retrying...
=09
	> Apr  3 16:32:06 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name elog, rc -20, retrying...
=09
	> Apr  3 16:32:06 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:32:06 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:32:06 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:32:06 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:32:06 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:32:06 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:32:06 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:32:06 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:32:13 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:32:13 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:32:13 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:32:20 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:32:22 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:32:23 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:32:23 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:32:51 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:33:22 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:33:51 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:33:51 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:33:51 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:33:51 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:33:51 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:33:52 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:33:52 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:33:52 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:33:52 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:33:52 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name sdm, rc -20, retrying...
=09
	> Apr  3 16:33:53 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:33:53 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:33:53 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:33:53 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name elog, rc -20, retrying...
=09
	> Apr  3 16:33:53 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:33:54 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:33:54 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:33:54 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:33:55 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:33:55 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:33:55 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:33:55 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:33:55 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:33:55 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:33:56 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:33:56 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:33:56 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:33:56 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:33:57 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name nfxsh-1396, rc -20, retrying...
=09
	> Apr  3 16:33:57 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name elog, rc -20, retrying...
=09
	> Apr  3 16:33:57 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:33:57 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:33:57 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:33:57 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:33:57 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:33:58 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:33:58 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:33:58 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:34:24 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:34:55 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:35:15 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:35:16 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:35:16 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:35:16 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:35:16 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:35:16 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:35:16 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name sdm, rc -20, retrying...
=09
	> Apr  3 16:35:17 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name nfxsh-1396, rc -20, retrying...
=09
	> Apr  3 16:35:17 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:35:17 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:35:17 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:35:17 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:35:18 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name elog, rc -20, retrying...
=09
	> Apr  3 16:35:18 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:35:18 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:35:18 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:35:18 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:35:19 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:35:19 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:35:19 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:35:19 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:35:19 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:35:19 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:35:20 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:35:20 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:35:20 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:35:20 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:35:20 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name elog, rc -20, retrying...
=09
	> Apr  3 16:35:21 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name nfxsh-1451, rc -20, retrying...
=09
	> Apr  3 16:35:21 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:35:21 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:35:21 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:35:21 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:35:21 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:35:21 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:35:22 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:35:22 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:35:26 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:35:57 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:36:28 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:36:42 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:36:42 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:36:43 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:36:43 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:36:43 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:36:43 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:36:43 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:36:44 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:36:44 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:36:44 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:36:44 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name nfxsh-1451, rc -20, retrying...
=09
	> Apr  3 16:36:44 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:36:44 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:36:45 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:36:45 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name elog, rc -20, retrying...
=09
	> Apr  3 16:36:45 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:36:45 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:36:45 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:36:45 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:36:46 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:36:46 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:36:46 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:36:46 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:36:46 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:36:46 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:36:46 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:36:46 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name elog, rc -20, retrying...
=09
	> Apr  3 16:36:46 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:36:46 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:36:46 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:36:46 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name nfxsh-1470, rc -20, retrying...
=09
	> Apr  3 16:36:46 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:36:46 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name nfxsh-1479, rc -20, retrying...
=09
	> Apr  3 16:36:46 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:36:46 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:36:46 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:36:46 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:36:56 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:36:56 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:36:56 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:36:56 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:36:56 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:36:56 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:36:59 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:37:06 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:37:06 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:37:06 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:37:06 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:37:06 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:37:15 localhost last message repeated 2 times
=09
	> Apr  3 16:37:15 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:37:16 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:37:16 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:37:16 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:37:24 localhost last message repeated 2 times
=09
	> Apr  3 16:37:25 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:37:25 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:37:25 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:37:25 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:37:25 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:37:30 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:38:01 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:38:08 localhost : 0:0:nfxsh:NOTICE: cmd[0]: -> EMRS:
tried
=09
	> (5 times) without success to get the EMRS config from nfxsh :
=09
	> status[2] Apr  3 16:38:08 localhost : 0:0:cluster2:INFO:
=09
	> cluster_lookup_sess: fail to open sess for app cluster2, rc
-20
=09
	> Apr  3 16:38:09 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:38:09 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:38:09 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:38:09 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:38:09 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:38:09 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:38:09 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:38:10 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:38:10 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name elog, rc -20, retrying...
=09
	> Apr  3 16:38:10 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name nfxsh-1506, rc -20, retrying...
=09
	> Apr  3 16:38:10 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:38:10 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:38:10 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:38:10 localhost : 0:0:cluster2:INFO:
=09
	> cluster_clientSendRmcRpc: Error sending rpc to clusterrpc,
flags
=09
	> 820a, name sdm, rc -20, retrying...
=09
	> Apr  3 16:38:10 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:38:11 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:38:11 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:38:11 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:38:11 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:38:19 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:38:20 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:38:20 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:38:20 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:38:20 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:38:20 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:38:29 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:38:29 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:38:29 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:38:30 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:38:32 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	> Apr  3 16:38:39 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:38:39 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:38:39 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:38:39 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:38:39 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:38:48 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:38:48 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app cluster2, rc -20
=09
	> Apr  3 16:38:48 localhost : 0:0:evm:INFO: evm_rcvRmcMsg : NCM
session
=09
	> open failed
=09
	> Apr  3 16:38:48 localhost : 0:0:evm:INFO: evm_closeSess : NCM
session
=09
	> closed
=09
	> Apr  3 16:38:48 localhost : 0:0:cluster2:INFO:
cluster_lookup_sess:
=09
	> fail to open sess for app clusterrpc, rc -20
=09
	> Apr  3 16:39:03 localhost kernel: fp0: warning:
=09
	> rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
=09
	>
=09
	> _____________________________________________
=09
	> From: Larry Scheer
=09
	> Sent: Thursday, April 03, 2008 3:03 PM
=09
	> To: dl-Cougar; dl-QA
=09
	> Subject: Submittal 16 Beta Release Candidate availble
=09
	>
=09
	> Submittal 16 made it through basic acceptance tests and is
available
=09
	> for general use.
=09
	>
=09
	> This submittal is the first Beta Release candidate.
=09
	> This submittal has a fix for the blocker defect: 22970/22947
TXRX
=09
	> crashes during mirror test connect.
=09
	> This submittal adds the Bobcat Migration code to the cougar
project.
=09
	> This submittal adds the First Time Install code to the cougar
project.
=09
	>
=09
	>
=09
	> Source tree is located here:
=09
	>
=09
	> n/Build-Trees/R4.0.0.0/R4.0.0.0-040308/nfx-tree
=09
	> /n/Build-Trees/R4.0.0.0/R4.0.0.0-040308/linux
=09
	>
=09
	> R3.3.0.0 images are here:
=09
	> Cheetah optimized:
=09
	> ftp://upgrade@10.2.0.2/home/upgrade/R3.3.0.0-040308.tar.gz
=09
	> <ftp://upgrade@10.2.0.2/home/upgrade/R3.3.0.0-032808.tar.gz
<ftp://upgrade@10.2.0.2/home/upgrade/R3.3.0.0-032808.tar.gz%20> >
=09
	> Cheetah debug:
=09
	> ftp://upgrade@10.2.0.2/home/upgrade/R3.3.0.0DBG-040308.tar.gz
=09
	> <ftp://upgrade@10.2.0.2/home/upgrade/R3.3.0.0DBG-032808.tar.gz
<ftp://upgrade@10.2.0.2/home/upgrade/R3.3.0.0DBG-032808.tar.gz%20> >
=09
	> Bobcat optimized:
=09
	> ftp://upgrade@10.2.0.2/home/upgrade/R3.3.0.0BC-040308.tar.gz
=09
	> <ftp://upgrade@10.2.0.2/home/upgrade/R3.3.0.0BC-032808.tar.gz
<ftp://upgrade@10.2.0.2/home/upgrade/R3.3.0.0BC-032808.tar.gz%20> >
=09
	> Bobcat debug:
=09
	>
ftp://upgrade@10.2.0.2/home/upgrade/R3.3.0.0BCDBG-040308.tar.gz
=09
	>
<ftp://upgrade@10.2.0.2/home/upgrade/R3.3.0.0BCDBG-032808.tar.gz
<ftp://upgrade@10.2.0.2/home/upgrade/R3.3.0.0BCDBG-032808.tar.gz%20> >
=09
	>
=09
	> R4.0.0.0 images are here:
=09
	> Cougar debug:
=09
	> ftp://upgrade@10.2.0.2/R4.0.0.0CGDBG-040308.tar.gz
=09
	> <ftp://upgrade@10.2.0.2/R4.0.0.0CGDBG-032808.tar.gz
<ftp://upgrade@10.2.0.2/R4.0.0.0CGDBG-032808.tar.gz%20> >
=09
	> Cougar optimized:
=09
	> ftp://upgrade@10.2.0.2/R4.0.0.0CG-040308.tar.gz
=09
	> <ftp://upgrade@10.2.0.2/R4.0.0.0CG-032808.tar.gz
<ftp://upgrade@10.2.0.2/R4.0.0.0CG-032808.tar.gz%20> >
=09
	>
=09
	>
=09
	> If you have not upgraded your cougar to the latest PROM
version
=09
	> (1.0.5) it is recommended that you do. Please follow the
instructions
=09
	> in section 5 and 6 of the attached submittal notes carefully
to
=09
	> upgrade your proms.
=09
	>
=09
	> Submittal 16 notes are attached. Please read for more details
about
=09
	> this build and for instructions specific to this product.
=09
	>
=09
	>  << File: Cougar_Submittal_16_Notes.htm >>
=09
=09
=09



