X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C81B40.18E9614B@onstor-exch02.onstor.net>; Tue, 30 Oct 2007 13:59:08 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C81B40.18E9614B"
Content-class: urn:content-classes:message
Subject: FW: cluster db prob
Date: Tue, 30 Oct 2007 13:59:08 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E030E397C@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: cluster db prob
Thread-Index: AcgafVOVnUr5R5x2TxmEN98NIqLGrQACCJugAAHVx9AABWgWsAApfHNQ
From: "Mike Lee" <mike.lee@onstor.com>
To: "Andy Sharp" <andy.sharp@onstor.com>
Cc: "Vikas Saini" <vikas.saini@onstor.com>,
	"Tim Gardner" <tim.gardner@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C81B40.18E9614B
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Andy:
The message of interest that suggests a panic is highlighted in red.
Thanks.
-Mike

>  -----Original Message-----
> From: 	Vikas Saini =20
> Sent:	Monday, October 29, 2007 2:02 PM
> To:	Mike Lee
> Subject:	RE: cluster db prob
>=20
> Starting NIS services: ypbindstart-stop-daemon: nothing in /proc - not
> mounted? (Success)
> binding to YP server...................................fp0:
> [dp_proxy_init]: pass 1
> tx0: [dp_proxy_init]: pass 1
> tx0: pte_setup, region =3D 4, phys_offset =3D 0x58000000, virt =3D
> 0x4000000000, psize =3D 28000000, vsize =3D 28000000, pgsize =3D =
8000000,
> readwrite_size =3D 0x28000000
> fp0: pte_setup, region =3D 2, phys_offset =3D 0x30000000, virt =3D
> 0x2000000000, psize =3D 50000000, vsize =3D 50000000, pgsize =3D =
8000000,
> readwrite_size =3D 0x50000000
> fp0: pte_setup, region =3D 6, phys_offset =3D 0x30000000, virt =3D
> 0x6000000000, psize =3D 50000000, vsize =3D 50000000, pgsize =3D =
8000000,
> readwrite_size =3D 0x50000000
> fp0: Zeroing the WKA area, 0x73000000
> tx0: pte_setup, region =3D 7, phys_offset =3D 0x58000000, virt =3D
> 0x7000000000, psize =3D 28000000, vsize =3D 28000000, pgsize =3D =
8000000,
> readwrite_size =3D 0x28000000
> tx0: Zeroing the WKA area, 0x73000000
> ....tx0: unusable buffer, 0x200ffffc00, phys_base =3D cffffc00, =
phys_end
> =3D 100000300
> ....failed (backgrounded).
> .
> fp0: [dp_proxy_init]: pass 2
> fp0: sb1250dm_initModule: sb1250dmCfg[0]=3D0x1000507e00
> fp0: sb1250dm_initModule: sb1250dmCfg[1]=3D0x1000507c80
> fp0: sb1250dm_initModule: sb1250dmCfg[2]=3D0x1000507b00
> fp0: sb1250dm_initModule: sb1250dmCfg[3]=3D0x1000507980
> fp0: esm_stackInit(): Enter
> fp0: esm_stackInit(): Leave
> fp0: bmc12500_install_allocators: install RUNTIME allocators
> fp0: bmc12500Eth_initModule w/AUTO TX Retrans enabled
> fp0: bcm12500Eth_initModule SEQ RRX Retrans enabled
> fp0: isModel2280: Model 2220/40/60
> fp0: rmc: rmc_init(): RMC version 2.0.2 - myslot[1] mycpu[2]
> myapp_id[79]
> fp0: fs_initModule
> Starting MTA:open: Read-only file system
> touch: cannot touch `/var/lib/exim4/config.autogenerated.tmp':
> Read-only file system
> chown: cannot access `/var/lib/exim4/config.autogenerated.tmp': No
> such file or directory
> chown: changing ownership of `/var/lib/exim4/config.autogenerated':
> Read-only file system
> chmod: cannot access `/var/lib/exim4/config.autogenerated.tmp': No
> such file or directory
> chmod: changing permissions of `/var/lib/exim4/config.autogenerated':
> Read-only file system
> /usr/sbin/update-exim4.conf: line 286: cannot create temp file for
> here document: Read-only file system
> /usr/sbin/update-exim4.conf: line 435:
> /var/lib/exim4/config.autogenerated.tmp: Read-only file system
> 2013-03-04 08:29:16 Cannot open main log file
> "/var/log/exim4/mainlog": Read-only file system: euid=3D0 egid=3D0
> 2013-03-04 08:29:16 non-existent configuration file(s):
> /var/lib/exim4/config.autogenerated.tmp
> 2013-03-04 08:29:16 Cannot open main log file
> "/var/log/exim4/mainlog": Read-only file system: euid=3D0 egid=3D0
> exim[756]: 2013-03-04 08:29:16 non-existent configuration file(s):
> /var/lib/exim4/config.autogenerated.tmp
> exim[756]: 2013-03-04 08:29:16 Cannot open main log file
> "/var/log/exim4/mainlog": Read-only file system: euid=3D0 egid=3D0
> exim[756]: exim: could not open panic log - aborting: see message(s)
> above
> exim: could not open panic log - aborting: see message(s) above
> Invalid new configfile /var/lib/exim4/config.autogenerated.tmp
> not installing /var/lib/exim4/config.autogenerated.tmp to=20
> /var/lib/exim4/config.autogenerated
> tx0: ! phyid 11 reg 0 wrote 1340 got 1140
> tx0: gt_mii_writeSMI: write verify failed! phyid 12 reg 0 wrote 1340
> got 1140
> * Not starting internet superserver: no services enabled.
> tx0: gt_mii_writeSMI: write verify failed! phyid 13 reg 0 wrote 1340
> got 1140
> tx0: Initializing profiler _start@0xffffffff83000000
> _end@0xffffffff83754b90 textsize 7687056.
> tx0: altcpu_start(1, 0xffffffff834489dc)
> tx1: ECC exception handler Initialized
> tx1: handler already registered for ipl =3D 3 (0xffffffff834459d8)
> tx1: handler already registered for ipl =3D 2 (0xffffffff834459d8)
> tx0: writing 0xffffffff8547bfe8 to 0xffffffff808af3d8
> tx0: writing 0xffffffff839a3c70 to 0xffffffff808af3f8
> tx0: writing 0x1 to 0xffffffff808af418
> tx0: writing 0xffffffff834489dc to 0xffffffff808af3b8
> tx1: rmc: rmc_init(): RMC version 2.0.2 - myslot[1] mycpu[1]
> myapp_id[79]
> tx1: sb1250dm_openChnl: sb1250dmCfg[2]=3D0x1001d6fc00
> tx1: Initializing profiler _start@0xffffffff83000000
> _end@0xffffffff83754b90 textsize 7687056.
> Starting OpenBSD Secure Shell server: sshdfp0: dump ra sm create
> fp0: fs_initModule done on slot 1 cpu 2
> fp0: efs_sscAppId: 130, nfsAppId: 20, mcpuAppId: 67
> fp0: sb1250dm_openChnl: sb1250dmCfg[0]=3D0x1000507e00
> fp0: sb1250dm_openChnl: sb1250dmCfg[1]=3D0x1000507c80
> fp0: tpl_fp_init: init completeInitializing profiler
> _start@0xffffffff83000000 _end@0xffffffff83727800 textsize 7501824.
> tx0: TXRX0:1 > 2: Port : fp1.0 is DOWN
> tx0:=20
> tx0: 3: luc_link_down: 359: lport fp1.0 DOWN.
> tx0:=20
> tx0: 4: luc_link_down:381: Spurious link down notification. Port
> fp1.0.
> tx0:=20
> tx0: 5: Port : fp1.1 is DOWN
> tx0:=20
> tx0: 6: luc_link_down: 359: lport fp1.1 DOWN.
> tx0:=20
> tx0: 7: luc_link_down:381: Spurious link down notification. Port
> fp1.1.
> tx0:=20
> fp0: core_init_buffers: total buffers =3D 504
> fp0: altcpu_start(1, 0xffffffff83401b20)
> fp0: writing 0xffffffff86205fe8 to 0xffffffff808af3d8
> fp0: writing 0xffffffff83758f90 to 0xffffffff808af3f8
> fp0: writing 0x1 to 0xffffffff808af418
> fp0: writing 0xffffffff83401b20 to 0xffffffff808af3b8
> fp1: ECC exception handler Initialized
> fp1: handler already registered for ipl =3D 3 (0xffffffff833fe7a8)
> fp1: handler already registered for ipl =3D 2 (0xffffffff833fe7a8)
> fp1: Initializing profiler _start@0xffffffff83000000
> _end@0xffffffff83727800 textsize 7501824.
> tx0: 8: Port : fp1.2 is DOWN
> tx0:=20
> tx0: 9: luc_link_down: 359: lport fp1.2 DOWN.
> tx0:=20
> tx0: 10: luc_link_down:381: Spurious link down notification. Port
> fp1.2.
> tx0:=20
> tx0: 11: Port : fp1.3 is DOWN
> tx0:=20
> tx0: 12: luc_link_down: 359: lport fp1.3 DOWN.
> tx0:=20
> tx0: 13: luc_link_down:381: Spurious link down notification. Port
> fp1.3.
> tx0:=20
> .
> Starting NFS common utilities: statdstart-stop-daemon: nothing in
> /proc - not mounted? (Success)
> .
> Starting NTP server: ntpd.
> Starting deferred execution scheduler: atd.
> Starting periodic command scheduler: crond/usr/sbin/cron: can't open
> or create /var/run/crond.pid: Read-only file system
>  failed!
> Starting ONStor services: mgmtbusstart-stop-daemon: nothing in /proc -
> not mounted? (Success)
> .
> fp0: FP0:1 > warning: rmc_pm_handle_failure(): sess
> {unknown_app:pm.0.0} down
> fp0: rmc_context_rm_sess: removing listen session[0x100050a000] from
> list, flags[c2020x].
> tx1: TXRX1:1 > warning: rmc_pm_handle_failure(): sess
> {unknown_app:pm.0.0} down
> tx1: rmc_context_rm_sess: removing listen session[0x100512f6c0] from
> list, flags[c2020x].
> fp0: warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
> fp0: rmc_context_rm_sess: removing listen session[0x100050a000] from
> list, flags[c2020x].
> tx1: warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
> tx1: rmc_context_rm_sess: removing listen session[0x100512f6c0] from
> list, flags[c2020x].
> fp0: warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
> fp0: rmc_context_rm_sess: removing listen session[0x100050a000] from
> list, flags[c2020x].
> tx1: warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
> tx1: rmc_context_rm_sess: removing listen session[0x100512f6c0] from
> list, flags[c2020x].
> INIT: Id "T0" respawning too fast: disabled for 5 minutes
> INIT: no more processes left in this runlevel
>=20
> _____________________________________________
> From: Mike Lee=20
> Sent: Monday, October 29, 2007 1:51 PM
> To: Vikas Saini
> Subject: RE: cluster db prob
>=20
> Vikas:=20
> Tim is asking me to investigate if it turns out to be a bug.
> So, please keep me posted.  Thanks.
> -Mike
>=20
>  -----Original Message-----
> From: 	Vikas Saini =20
> Sent:	Monday, October 29, 2007 1:49 PM
> To:	Mike Lee
> Subject:	RE: cluster db prob
>=20
> It is a clustering problem... here is the email which I sent earlier
> to these guys...... I am setting up my env to reproduce the
> problem...What details u r looking for, I can collect it incase I can
> reproduce it.
>=20
>=20
> Clustering is behaving in a very unreliable manner in bobcat Linux.
> Here is what I tried since last Thursday and output I got.
>=20
> Eng63 is the PCC and it has around 5-6 vsvr and 5-6 volumes.
>=20
> I got another system eng60. Both eng63 and eng60 were on same bobcat
> Linux build which I believe is sub5.
>=20
> I added eng60 to eng63 cluster (cluster add followed by cluster
> commit), it didn't work and I kept on getting ubik and cluster related
> error messages on both eng63 and eng60.
>=20
> I rebooted eng60... no change in behavior
>=20
> I rebooted both eng63 and eng60 and still same problem.
>=20
> I removed eng60 from eng63 cluster (cluster delete followed by cluster
> commit), it rebooted eng60 and removed it from cluster but I also lost
> my clusterdb on eng63. All the vsvr information and domain information
> is gone...
>=20
>=20
> Anyway I tried eng60 back to eng63 cluster and it worked fine. I tried
> 2 times and both times it was ok.
>=20
> After 2 successful attempts, I tried to remove eng60 from eng63
> cluster and now eng60 is in weird state where it never came up after
> reboot. I am getting following error messages=20
>=20
> fp0: FP0:1 > warning: rmc_pm_handle_failure(): sess
> {unknown_app:pm.0.0} down
> tx1: TXRX1:1 > warning: rmc_pm_handle_failure(): sess
> {unknown_app:pm.0.0} down
> fp0: warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
> tx1: warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
> fp0: warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
> tx1: warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
> INIT: Id "T0" respawning too fast: disabled for 5 minutes
> INIT: no more processes left in this runlevel
> fp0: warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
> tx1: warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
> fp0: warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} down
>=20
>=20
>=20
> Also on eng63, since its cluster was gone, I tried to restore the
> clusterDB by copying it from a previous SGA, after copying clusterDB
> when I rebooted even eng63 went to a hung state getting same error
> messages described above.
>=20
> I am going to open a generic defect for clustering and copy all the
> info but looks like we might have quite a bit of issues for clustering
> to work in bobcat Linux.
>=20
>=20
>=20
> _____________________________________________
> From: Mike Lee=20
> Sent: Monday, October 29, 2007 1:45 PM
> To: Vikas Saini
> Subject: cluster db prob
>=20
> Vikas:
> Tim mentioned you found a cluster db problem today on bobcat-linux.
> Can u pls tell me where I can find details?
> Thanks.
> -Mike

------_=_NextPart_001_01C81B40.18E9614B
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7652.24">
<TITLE>FW: cluster db prob</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P DIR=3DLTR><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">Andy:</FONT></P>

<P DIR=3DLTR><FONT COLOR=3D"#0000FF" SIZE=3D2 FACE=3D"Arial">The message =
of interest that suggests a panic is highlighted in red.</FONT></P>

<P DIR=3DLTR><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">Thanks.</FONT></P>

<P DIR=3DLTR><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">-Mike</FONT></P>

<P DIR=3DLTR><FONT FACE=3D"Arial"><SPAN =
LANG=3D"en-us"></SPAN></FONT><SPAN LANG=3D"en-us">&nbsp;<FONT SIZE=3D1 =
FACE=3D"Tahoma">-----Original Message-----</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">From: &nbsp;</FONT></B> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Vikas Saini&nbsp; </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Sent:&nbsp;&nbsp;</FONT></B> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Monday, October 29, 2007 2:02 PM</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">To:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></B> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Mike Lee</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Subject:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT>=
</B> <FONT SIZE=3D1 FACE=3D"Tahoma">RE: cluster db =
prob</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Starting NIS services: ypbindstart-stop-daemon: nothing =
in /proc - not mounted? (Success)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">binding to YP =
server...................................fp0: [dp_proxy_init]: pass =
1</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: [dp_proxy_init]: pass 1</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: pte_setup, region =3D 4, phys_offset =3D 0x58000000, =
virt =3D 0x4000000000, psize =3D 28000000, vsize =3D 28000000, pgsize =
=3D 8000000, readwrite_size =3D 0x28000000</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: pte_setup, region =3D 2, phys_offset =3D 0x30000000, =
virt =3D 0x2000000000, psize =3D 50000000, vsize =3D 50000000, pgsize =
=3D 8000000, readwrite_size =3D 0x50000000</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: pte_setup, region =3D 6, phys_offset =3D 0x30000000, =
virt =3D 0x6000000000, psize =3D 50000000, vsize =3D 50000000, pgsize =
=3D 8000000, readwrite_size =3D 0x50000000</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: Zeroing the WKA area, 0x73000000</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: pte_setup, region =3D 7, phys_offset =3D 0x58000000, =
virt =3D 0x7000000000, psize =3D 28000000, vsize =3D 28000000, pgsize =
=3D 8000000, readwrite_size =3D 0x28000000</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: Zeroing the WKA area, 0x73000000</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">....tx0: unusable buffer, 0x200ffffc00, phys_base =3D =
cffffc00, phys_end =3D 100000300</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">....failed (backgrounded).</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: [dp_proxy_init]: pass 2</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: sb1250dm_initModule: =
sb1250dmCfg[0]=3D0x1000507e00</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: sb1250dm_initModule: =
sb1250dmCfg[1]=3D0x1000507c80</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: sb1250dm_initModule: =
sb1250dmCfg[2]=3D0x1000507b00</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: sb1250dm_initModule: =
sb1250dmCfg[3]=3D0x1000507980</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: esm_stackInit(): Enter</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: esm_stackInit(): Leave</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: bmc12500_install_allocators: install RUNTIME =
allocators</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: bmc12500Eth_initModule w/AUTO TX Retrans =
enabled</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: bcm12500Eth_initModule SEQ RRX Retrans =
enabled</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: isModel2280: Model 2220/40/60</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: rmc: rmc_init(): RMC version 2.0.2 - myslot[1] =
mycpu[2] myapp_id[79]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: fs_initModule</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Starting MTA:open: Read-only file =
system</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">touch: cannot touch =
`/var/lib/exim4/config.autogenerated.tmp': Read-only file =
system</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">chown: cannot access =
`/var/lib/exim4/config.autogenerated.tmp': No such file or =
directory</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">chown: changing ownership of =
`/var/lib/exim4/config.autogenerated': Read-only file =
system</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">chmod: cannot access =
`/var/lib/exim4/config.autogenerated.tmp': No such file or =
directory</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">chmod: changing permissions of =
`/var/lib/exim4/config.autogenerated': Read-only file =
system</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">/usr/sbin/update-exim4.conf: line 286: cannot create temp =
file for here document: Read-only file system</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">/usr/sbin/update-exim4.conf: line 435: =
/var/lib/exim4/config.autogenerated.tmp: Read-only file =
system</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">2013-03-04 08:29:16 Cannot open main log file =
&quot;/var/log/exim4/mainlog&quot;: Read-only file system: euid=3D0 =
egid=3D0</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">2013-03-04 08:29:16 non-existent configuration file(s): =
/var/lib/exim4/config.autogenerated.tmp</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">2013-03-04 08:29:16 Cannot open main log file =
&quot;/var/log/exim4/mainlog&quot;: Read-only file system: euid=3D0 =
egid=3D0</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">exim[756]: 2013-03-04 08:29:16 non-existent configuration =
file(s): /var/lib/exim4/config.autogenerated.tmp</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">exim[756]: 2013-03-04 08:29:16 Cannot open main log file =
&quot;/var/log/exim4/mainlog&quot;: Read-only file system: euid=3D0 =
egid=3D0</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#FF0000" SIZE=3D2 =
FACE=3D"Arial">exim[756]: exim: could not open panic log - aborting: see =
message(s) above</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">exim: could not open panic log - aborting: see message(s) =
above</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Invalid new configfile =
/var/lib/exim4/config.autogenerated.tmp</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">not installing /var/lib/exim4/config.autogenerated.tmp to =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">/var/lib/exim4/config.autogenerated</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: ! phyid 11 reg 0 wrote 1340 got =
1140</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: gt_mii_writeSMI: write verify failed! phyid 12 reg 0 =
wrote 1340 got 1140</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">* Not starting internet superserver: no services =
enabled.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: gt_mii_writeSMI: write verify failed! phyid 13 reg 0 =
wrote 1340 got 1140</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: Initializing profiler _start@0xffffffff83000000 =
_end@0xffffffff83754b90 textsize 7687056.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: altcpu_start(1, =
0xffffffff834489dc)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx1: ECC exception handler Initialized</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx1: handler already registered for ipl =3D 3 =
(0xffffffff834459d8)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx1: handler already registered for ipl =3D 2 =
(0xffffffff834459d8)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: writing 0xffffffff8547bfe8 to =
0xffffffff808af3d8</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: writing 0xffffffff839a3c70 to =
0xffffffff808af3f8</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: writing 0x1 to 0xffffffff808af418</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: writing 0xffffffff834489dc to =
0xffffffff808af3b8</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx1: rmc: rmc_init(): RMC version 2.0.2 - myslot[1] =
mycpu[1] myapp_id[79]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx1: sb1250dm_openChnl: =
sb1250dmCfg[2]=3D0x1001d6fc00</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx1: Initializing profiler _start@0xffffffff83000000 =
_end@0xffffffff83754b90 textsize 7687056.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Starting OpenBSD Secure Shell server: sshdfp0: dump ra sm =
create</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: fs_initModule done on slot 1 cpu 2</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: efs_sscAppId: 130, nfsAppId: 20, mcpuAppId: =
67</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: sb1250dm_openChnl: =
sb1250dmCfg[0]=3D0x1000507e00</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: sb1250dm_openChnl: =
sb1250dmCfg[1]=3D0x1000507c80</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: tpl_fp_init: init completeInitializing profiler =
_start@0xffffffff83000000 _end@0xffffffff83727800 textsize =
7501824.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: TXRX0:1 &gt; 2: Port : fp1.0 is =
DOWN</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: 3: luc_link_down: 359: lport fp1.0 =
DOWN.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: 4: luc_link_down:381: Spurious link down =
notification. Port fp1.0.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: 5: Port : fp1.1 is DOWN</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: 6: luc_link_down: 359: lport fp1.1 =
DOWN.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: 7: luc_link_down:381: Spurious link down =
notification. Port fp1.1.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: core_init_buffers: total buffers =3D =
504</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: altcpu_start(1, =
0xffffffff83401b20)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: writing 0xffffffff86205fe8 to =
0xffffffff808af3d8</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: writing 0xffffffff83758f90 to =
0xffffffff808af3f8</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: writing 0x1 to 0xffffffff808af418</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: writing 0xffffffff83401b20 to =
0xffffffff808af3b8</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp1: ECC exception handler Initialized</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp1: handler already registered for ipl =3D 3 =
(0xffffffff833fe7a8)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp1: handler already registered for ipl =3D 2 =
(0xffffffff833fe7a8)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp1: Initializing profiler _start@0xffffffff83000000 =
_end@0xffffffff83727800 textsize 7501824.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: 8: Port : fp1.2 is DOWN</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: 9: luc_link_down: 359: lport fp1.2 =
DOWN.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: 10: luc_link_down:381: Spurious link down =
notification. Port fp1.2.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: 11: Port : fp1.3 is DOWN</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: 12: luc_link_down: 359: lport fp1.3 =
DOWN.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: 13: luc_link_down:381: Spurious link down =
notification. Port fp1.3.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx0: </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Starting NFS common utilities: statdstart-stop-daemon: =
nothing in /proc - not mounted? (Success)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Starting NTP server: ntpd.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Starting deferred execution scheduler: =
atd.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Starting periodic command scheduler: crond/usr/sbin/cron: =
can't open or create /var/run/crond.pid: Read-only file =
system</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;failed!</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Starting ONStor services: mgmtbusstart-stop-daemon: =
nothing in /proc - not mounted? (Success)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: FP0:1 &gt; warning: rmc_pm_handle_failure(): sess =
{unknown_app:pm.0.0} down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: rmc_context_rm_sess: removing listen =
session[0x100050a000] from list, flags[c2020x].</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx1: TXRX1:1 &gt; warning: rmc_pm_handle_failure(): sess =
{unknown_app:pm.0.0} down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx1: rmc_context_rm_sess: removing listen =
session[0x100512f6c0] from list, flags[c2020x].</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: warning: rmc_pm_handle_failure(): sess =
{unknown_app:pm.0.0} down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: rmc_context_rm_sess: removing listen =
session[0x100050a000] from list, flags[c2020x].</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx1: warning: rmc_pm_handle_failure(): sess =
{unknown_app:pm.0.0} down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx1: rmc_context_rm_sess: removing listen =
session[0x100512f6c0] from list, flags[c2020x].</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: warning: rmc_pm_handle_failure(): sess =
{unknown_app:pm.0.0} down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">fp0: rmc_context_rm_sess: removing listen =
session[0x100050a000] from list, flags[c2020x].</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx1: warning: rmc_pm_handle_failure(): sess =
{unknown_app:pm.0.0} down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">tx1: rmc_context_rm_sess: removing listen =
session[0x100512f6c0] from list, flags[c2020x].</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">INIT: Id &quot;T0&quot; respawning too fast: disabled for =
5 minutes</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">INIT: no more processes left in this =
runlevel</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 FACE=3D"Tahoma"> Mike =
Lee<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, October 29, 2007 1:51 PM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Vikas Saini<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> RE: cluster db prob</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">Vikas: </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">Tim is asking me to investigate if it turns out to be a =
bug.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">So, please keep me posted.&nbsp; =
Thanks.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">-Mike</FONT></SPAN></P>
<UL DIR=3DLTR>
<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT =
FACE=3D"Arial"></FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us">&nbsp;<FONT SIZE=3D1 FACE=3D"Tahoma">-----Original =
Message-----</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">From: &nbsp;</FONT></B> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Vikas Saini&nbsp; </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Sent:&nbsp;&nbsp;</FONT></B> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Monday, October 29, 2007 1:49 PM</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">To:&nbsp;&nbsp;&nbsp;&nbsp;</FONT></B> <FONT SIZE=3D1 =
FACE=3D"Tahoma">Mike Lee</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><B><FONT SIZE=3D1 =
FACE=3D"Tahoma">Subject:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</FONT>=
</B> <FONT SIZE=3D1 FACE=3D"Tahoma">RE: cluster db =
prob</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">It is a clustering problem... here is the email which I =
sent earlier to these guys&#8230;&#8230; I am setting up my env to =
reproduce the problem&#8230;What details u r looking for, I can collect =
it incase I can reproduce it.</FONT></SPAN></P>
<BR>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Clustering is behaving in a very unreliable manner in =
bobcat Linux. Here is what I tried since last Thursday and output I =
got.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Eng63 is =
the PCC and it has around 5-6 vsvr and 5-6 volumes.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I got =
another system eng60. Both eng63 and eng60 were on same bobcat Linux =
build which I believe is sub5.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I added =
eng60 to eng63 cluster (cluster add followed by cluster commit), it =
didn&#8217;t work and I kept on getting ubik and cluster related error =
messages on both eng63 and eng60.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I =
rebooted eng60&#8230; no change in behavior</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I =
rebooted both eng63 and eng60 and still same problem.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I =
removed eng60 from eng63 cluster (cluster delete followed by cluster =
commit), it rebooted eng60 and removed it from cluster but I also lost =
my clusterdb on eng63. All the vsvr information and domain information =
is gone&#8230;</FONT></SPAN></P>
<BR>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Anyway I =
tried eng60 back to eng63 cluster and it worked fine. I tried 2 times =
and both times it was ok.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">After 2 =
successful attempts, I tried to remove eng60 from eng63 cluster and now =
eng60 is in weird state where it never came up after reboot. I am =
getting following error messages </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">fp0: =
FP0:1 &gt; warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} =
down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">tx1: =
TXRX1:1 &gt; warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} =
down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">fp0: =
warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} =
down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">tx1: =
warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} =
down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">fp0: =
warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} =
down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">tx1: =
warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} =
down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">INIT: Id =
&quot;T0&quot; respawning too fast: disabled for 5 =
minutes</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">INIT: no =
more processes left in this runlevel</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">fp0: =
warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} =
down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">tx1: =
warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} =
down</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">fp0: =
warning: rmc_pm_handle_failure(): sess {unknown_app:pm.0.0} =
down</FONT></SPAN></P>
<BR>
<BR>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Also on =
eng63, since its cluster was gone, I tried to restore the clusterDB by =
copying it from a previous SGA, after copying clusterDB when I rebooted =
even eng63 went to a hung state getting same error messages described =
above.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I am =
going to open a generic defect for clustering and copy all the info but =
looks like we might have quite a bit of issues for clustering to work in =
bobcat Linux.</FONT></SPAN></P>
<BR>
<BR>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B><FONT SIZE=3D2 FACE=3D"Tahoma"> Mike =
Lee<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Sent:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Monday, October 29, 2007 1:45 PM<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">To:</FONT></B><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Vikas Saini<BR>
</FONT><B><FONT SIZE=3D2 FACE=3D"Tahoma">Subject:</FONT></B><FONT =
SIZE=3D2 FACE=3D"Tahoma"> cluster db prob</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Vikas:</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Tim =
mentioned you found a cluster db problem today on =
bobcat-linux.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Can u =
pls tell me where I can find details?</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Thanks.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">-Mike</FONT></SPAN></P>
</UL>
</BODY>
</HTML>
------_=_NextPart_001_01C81B40.18E9614B--
