X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C793ED.F7E40E8A@onstor-exch02.onstor.net>; Fri, 11 May 2007 10:01:07 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C793ED.F7E40E8A"
Content-class: urn:content-classes:message
Subject: RE: MD upgrade procedure
Date: Fri, 11 May 2007 10:01:07 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E02679C9C@onstor-exch02.onstor.net>
In-Reply-To: <6041AEBD8A3E264785ABDA2F60B52DB4499C0A9E@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: MD upgrade procedure
Thread-Index: AceTXJdVzjWTSchUSru5u9TaQDK4VwAAxUkgAAEf1yAAAGY0kAAAGBuQAAAtxZAABX23oAAAVe3gAAFrfVAAGmnd0A==
From: "Sandrine Boulanger" <sandrine.boulanger@onstor.com>
To: "Sandrine Boulanger" <sandrine.boulanger@onstor.com>,
	"Brian DeForest" <brian.deforest@onstor.com>,
	"Raj Kumar" <raj.kumar@onstor.com>,
	"John Keiffer" <john.keiffer@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>,
	"Chris Vandever" <chris.vandever@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C793ED.F7E40E8A
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

H10r9 rebooted yesterday at 19:55. I did not find a reboot cmd in elog,
so I suspected bsd crash running out of memory, but there is no 0.0
crashdump.
Since then, NCM is fine and commands are OK. Could it have been
something related to the cluster daemons?

syslog
May 10 18:23:59 g10r9 sshd[1206]: Accepted password for root from
10.0.0.107 port 1613 ssh2
May 10 18:31:00 g10r9 sshd[30921]: Accepted password for root from
10.0.0.136 port 60219 ssh2
May 10 19:07:21 g10r9 sshd[14482]: Accepted password for root from
10.0.0.107 port 1899 ssh2
May 10 19:55:35 g10r9 syslogd: restart
May 10 19:55:36 g10r9 /bsd: Copyright (c) 1982, 1986, 1989, 1991, 1993
May 10 19:55:36 g10r9 /bsd:     The Regents of the University of
California.  All rights reserved.
May 10 19:55:36 g10r9 /bsd: Copyright (c) 1995-2000 OpenBSD. All rights
reserved.  http://www.OpenBSD.org
May 10 19:55:36 g10r9 /bsd: Copyright (c) 1995-2000 OpenBSD. All rights
reserved.  http://www.OpenBSD.org

elog
May 10 19:42:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1
May 10 19:42:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30
May 10 19:45:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1
May 10 19:45:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30
May 10 19:48:22 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1
May 10 19:48:22 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30
May 10 19:51:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1
May 10 19:51:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30
May 10 19:56:16 g10r9 : 0:0:cm:NOTICE: CHASSISD: Chassis Manager:
Started
May 10 19:56:17 g10r9 : 0:0:cm:NOTICE: CM: opened kseg1
May 10 19:56:32 g10r9 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
CPU 0, State Up
May 10 19:56:32 g10r9 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
CPU 1, State Up
May 10 19:56:32 g10r9 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
CPU 2, State Up
May 10 19:56:32 g10r9 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 1,
CPU 3, State Up
May 10 19:56:32 g10r9 : 0:0:eventd:NOTICE: Process-EVENT CPU: Slot 2,
CPU 0, State Up

g10r9 diag> cluster show summary
Cluster Name: g10r9       Cluster State:   On
------------------------------------------------------
NAS Gateways        IP              State   PCC
------------------------------------------------------
g10r9               10.2.9.10       UP      NO
g11r9               10.2.9.11       UP      YES
------------------------------------------------------
PCC Name               : g11r9
Gateway Groups         : default
No of Gateway Group(s) : 1
No of Gateway(s)       : 2
No of Virtual Server(s): 9
No of Volume(s)        : 9
No of online Volume(s) : 2
No of offline Volume(s): 7
Online Volume capacity : 15103.9 MiB
Percent used           : 0.42%
g10r9 diag> cluster show group
GROUP default
        NAS GATEWAY g10r9
                VSVR VS_MGMT_512
                VSVR JUMBO_G10R9

        NAS GATEWAY g11r9
                VSVR VS_MGMT_903
                VSVR NIS_G10R9
                VSVR WINDOWS_G10R9
                VSVR LOCAL_NIS_G10R9
                VSVR OPENLDAP_G10R9
                VSVR AUTHDOM_G10R9
                VSVR NONETBIOS_G10R9

        NAS GATEWAY virtual-filer

g10r9 diag> elog show config
ELOG configuration
------------------
ELOG State: Enabled
ELOG Display: Disabled
ELOG Level: Notice
ELOG Facility: local0
ELOG Host Addr: 0.0.0.0


_____________________________________________
From: Sandrine Boulanger=20
Sent: Thursday, May 10, 2007 9:31 PM
To: Brian DeForest; Raj Kumar; John Keiffer; Andy Sharp; Chris Vandever
Subject: RE: MD upgrade procedure

Here is the relevant part of elog on g10r9 that shows the db version
upgrade, but also lots of cluster2 errors after that.

Sequence of events:
On g10r9
Before 15:26: first attempt to copy config files with tar
15:26 system reboot -s
15:30 filer came back up to OCT, found out we had missed to mount some
partitions
Rebooted the filer and changed the boot device to go back to 2.2.2 flash
15:40 chassisd started on 2.2.2
Copied the config files again
15:52 system reboot -s to boot on 3.0 flash
15:55 chassisd started on 3.0
15:56 vsvr show all command is OK

On g11r9
16:02 system reboot -s after copying config files

On g10r9
16:03 g11r9 is down, taking over vsvrs and volumes
16:07 g11r9 is back UP
16:12 cluster2 errors (see below)
16:14 Request to upgrade db failed, running in degraded mode, more
cluster2 errors
16:19 Cluster update complete, more cluster2 errors until 18:00 and more




May 10 16:04:31 g10r9 : 0:0:eventd:NOTICE: Process-EVENT Vsvr: Virtual
server 'WINDOWS_G10R9', Id 4, State 'Up'
May 10 16:04:31 g10r9 : 0:0:sanm:NOTICE: Processing fail-over event for
vsId[4]
May 10 16:04:46 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show cluster
: status[0]
May 10 16:05:42 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cluster show cluster
: status[0]
May 10 16:05:45 g10r9 : 0:0:nfxsh:NOTICE: cmd[3]: cluster show cluster
: status[0]
May 10 16:06:16 g10r9 : 0:0:ncm:WARNING: ncmd : ncm_reopen_filer:
ncm_open_filer(g11r9) succeeded=20
May 10 16:06:19 g10r9 : 0:0:ncm:WARNING: ncmd : ncm_reopen_filer:
ncm_open_filer(g11r9) succeeded=20
May 10 16:06:23 g10r9 : 0:0:ncm:WARNING: ncmd : ncm_reopen_filer:
retries to filer(g11r9) failed - aborting attemtpts
May 10 16:06:23 g10r9 : 0:0:ncm:WARNING: ncmd : ncm_handle_net_msg_done:
ncm_repoen filer failed - -1
May 10 16:06:30 g10r9 : 0:0:ncm:WARNING: ncmd : ncm_handle_ncm_msg_rcv:
ncm_forward_to_filer failed - -9
May 10 16:06:39 g10r9 : 0:0:eventd:NOTICE: Process-EVENT Node: Name
'g11r9', State Up, Msg ''
May 10 16:06:43 g10r9 : 0:0:nfxsh:NOTICE: cmd[4]: cluster show cluster
: status[0]
May 10 16:06:47 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.2.9.11=20
May 10 16:06:47 g10r9 : 0:0:cluster2:NOTICE: urecovery_Interact: send
new file in progress, remote ver 1178668330(3343), sending to 10.2.9.11=20
May 10 16:06:55 g10r9 : 0:0:cluster2:NOTICE: urecovery_Interact: send
new file end, code 0=20
May 10 16:07:30 g10r9 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
before open completed
May 10 16:07:30 g10r9 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
before open completed
May 10 16:08:03 g10r9 : 0:0:nfxsh:NOTICE: cmd[5]: vol show : status[0]
May 10 16:12:03 g10r9 : 0:0:cluster2:NOTICE: ubik init with buff size 64

May 10 16:12:26 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:12:26 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:12:37 g10r9 : 0:0:spm:ERROR: failed to get filer list from
cluster daemon=20
May 10 16:12:37 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:13:07 g10r9 last message repeated 3 times
May 10 16:13:12 g10r9 : 0:0:spm:ERROR: failed to get filer list from
cluster daemon=20
May 10 16:13:18 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:13:38 g10r9 last message repeated 2 times
May 10 16:13:47 g10r9 : 0:0:spm:ERROR: failed to get filer list from
cluster daemon=20
May 10 16:13:48 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:13:58 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:14:20 g10r9 : 0:0:cluster2:ERROR: cluster_request_upgrade_db:
no reply bck -1=20
May 10 16:14:20 g10r9 : 0:0:cluster2:ERROR:
ClusterCtrl_ProcessTimerFunc: Request to upgrade clusDb failed, rc 30=20
May 10 16:14:20 g10r9 : 0:0:cluster2:ERROR:
ClusterCtrl_ProcessTimerFunc: Running in degraded mode on version
0x20200, will retry upgrade to version 0x30000 later...=20
May 10 16:14:22 g10r9 : 0:0:spm:ERROR: failed to get filer list from
cluster daemon=20
May 10 16:14:30 g10r9 : 0:0:cluster2:ERROR: cluster_set_lock_on_rpc: no
reply bck -1=20
May 10 16:15:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:15:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:15:35 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:16:15 g10r9 last message repeated 4 times
May 10 16:16:47 g10r9 last message repeated 4 times
May 10 16:16:47 g10r9 : 0:0:vsd:ERROR: vsd_procGetVsListReq : Cannot get
vsId list for node g10r9
May 10 16:16:56 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:16:57 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:16:57 g10r9 : 0:0:vsd:ERROR: vsd_procGetVsListReq : Cannot get
vsId list for node g10r9
May 10 16:17:41 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.12.9.11=20
May 10 16:18:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:18:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:18:26 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:18:26 g10r9 : 0:0:vsd:ERROR: vsd_procGetVsListReq : Cannot get
vsId list for node g10r9
May 10 16:18:29 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.2.9.11=20
May 10 16:18:34 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.12.9.11=20
May 10 16:18:36 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:18:43 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.2.9.11=20
May 10 16:18:46 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:18:48 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.12.9.11=20
May 10 16:18:56 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:19:00 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.2.9.11=20
May 10 16:19:06 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:19:08 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.12.9.11=20
May 10 16:19:13 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.2.9.11=20
May 10 16:19:16 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:19:25 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.12.9.11=20
May 10 16:19:27 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:19:29 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.2.9.11=20
May 10 16:19:32 g10r9 : 0:0:cluster2:ERROR: cluster_upgrade_db_runtime:
ClusDb update complete; successfully updated version from 0x20200 to
0x30000=20
May 10 16:21:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:21:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:21:35 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:24:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:24:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:27:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:27:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:30:14 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:30:14 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:30:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:30:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:33:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:33:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:33:34 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:36:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:36:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:36:35 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:39:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:39:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:39:34 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:41:45 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.107, port 4873; Assigning fd(12)
May 10 16:41:46 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.107, port 4874; Assigning fd(12)
May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.107, port 4875; Assigning fd(12)
May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.107, port 4876; Assigning fd(13)
May 10 16:42:10 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system show chassis :
status[0]
May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.107, port 4884; Assigning fd(12)
May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.107, port 4885; Assigning fd(13)
May 10 16:42:12 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system show nodename :
status[0]
May 10 16:42:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:42:24 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show summary :
status[11]
May 10 16:42:25 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:42:25 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:42:55 g10r9 : 0:0:nfxsh:NOTICE: cmd[6]: cluster show cluster
: status[0]
May 10 16:43:13 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:43:13 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: cluster show summary :
status[11]
May 10 16:43:31 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system show nodename :
status[0]
May 10 16:43:42 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:43:42 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show summary :
status[11]
May 10 16:44:43 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:44:48 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:45:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:45:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:48:14 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.154, port 3315; Assigning fd(12)
May 10 16:48:19 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.154, port 3317; Assigning fd(13)
May 10 16:48:19 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:48:19 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.154, port 3318; Assigning fd(13)
May 10 16:48:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:48:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:48:31 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system show chassis :
status[0]
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.154, port 3319; Assigning fd(13)
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.154, port 3320; Assigning fd(12)
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.154, port 3321; Assigning fd(13)
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.154, port 3322; Assigning fd(12)
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.154, port 3323; Assigning fd(13)
May 10 16:48:32 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system show nodename :
status[0]
May 10 16:48:44 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:48:44 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show summary :
status[11]
May 10 16:49:37 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: elog show log 20 :
status[0]
May 10 16:49:55 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:49:55 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show summary :
status[11]
May 10 16:50:36 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:50:36 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cl show gr :
status[11]
May 10 16:51:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:51:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:51:34 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:53:25 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: cl show cl : status[0]
May 10 16:53:39 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:53:39 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cl show gr :
status[11]
May 10 16:53:59 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:53:59 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:54:03 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:54:03 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cl show sum :
status[11]
May 10 16:54:21 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:54:21 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:54:21 g10r9 : 0:0:nfxsh:NOTICE: cmd[3]: vsvr show all :
status[11]
May 10 16:56:52 g10r9 : 0:0:nfxsh:NOTICE: cmd[4]: elog show log :
status[0]
May 10 16:57:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:57:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:58:31 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.154, port 3349; Assigning fd(12)
May 10 16:58:44 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:58:44 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: cluster show summary :
status[11]
May 10 16:58:47 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.154, port 3350; Assigning fd(13)
May 10 16:58:53 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:58:53 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 17:00:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:00:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:00:34 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:03:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:03:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:06:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:06:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:09:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:09:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:12:00 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:12:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:12:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:14:40 g10r9 : 0:0:cluster2:NOTICE: ubik init with buff size 64

May 10 17:14:40 g10r9 : 0:0:pm:ERROR: pm_sig_handler:
/usr/local/agile/bin/vsd (pid 27612) terminated with signal Killed=20
May 10 17:14:41 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: volume show -r :
status[0]
May 10 17:16:53 g10r9 : 1:0:ipm:NOTICE: 21: vstack_vlink_create:294 : vl
FO active 1 prefered 1=20
May 10 17:16:53 g10r9 : 1:0:ipm:NOTICE: 22: vstack_vlink_create:329:
vlink lp link op-state DOWN=20
May 10 17:16:54 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D1 already exists.=20
May 10 17:16:54 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D10 already exists.=20
May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume (nis-vol)
current state - Mounted-RW State.
May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume (local)
current state - Mounted-RW State.
May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume (win-vol)
current state - Mounted-RW State.
May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume
(local-nis-vol) current state - Mounted-RW State.
May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume
(openldap-vol) current state - Mounted-RW State.
May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume
(authdom-vol) current state - Mounted-RW State.
May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume (jumbo-vol)
current state - Mounted-RW State.
May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume
(vol_mgmt_512) current state - Mounted-RW State.
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D3 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D4 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D5 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D6 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxldap_createVsContext[257]:
Context for VS=3D6 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D7 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxldap_createVsContext[257]:
Context for VS=3D7 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D8 already exists.=20
May 10 17:18:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:18:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:21:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:21:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:24:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:24:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:25:03 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: volume show -r :
status[0]
May 10 17:27:22 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:27:22 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:30:15 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:30:15 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:30:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:30:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:33:22 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:33:22 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:36:22 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:36:22 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:39:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:39:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:42:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:42:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:45:45 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: elog find cluster :
status[0]
May 10 17:46:59 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: elog find upgrade :
status[0]
May 10 17:47:25 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cluster show summary
: status[0]
May 10 17:48:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:48:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:51:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:51:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:52:11 g10r9 : 0:0:nfxsh:NOTICE: cmd[3]: system version -s :
status[0]
May 10 17:54:22 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:54:22 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:58:05 g10r9 : 0:0:nfxsh:NOTICE: cmd[4]: elog show config  :
status[11]
May 10 17:58:05 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:58:05 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 18:00:02 g10r9 newsyslog[12273]: logfile turned over

_____________________________________________
From: Brian DeForest=20
Sent: Thursday, May 10, 2007 8:44 PM
To: Sandrine Boulanger; Raj Kumar; John Keiffer; Andy Sharp; Chris
Vandever
Subject: RE: MD upgrade procedure

Subsequent testing showed the cluster commands repeatedly working ok
from the CLI on both nodes, though admittedly this was just one test.
John was going to test some more via NCM.   I think John plans to send a
more detailed update as well.

_____________________________________________
From: Sandrine Boulanger=20
Sent: Thursday, May 10, 2007 8:34 PM
To: Brian DeForest; Raj Kumar; John Keiffer; Andy Sharp
Subject: RE: MD upgrade procedure

g10r9 and g11r9.onstor.lab
I'll go through the elog right now to see if I can catch anything. It
does not look like this would be related to the conf file since 'cluster
show summary' uses the cluster db, which has been upgraded to version
3.0 on both filers (I already verified that).

_____________________________________________
From: Brian DeForest=20
Sent: Thursday, May 10, 2007 5:55 PM
To: Raj Kumar; John Keiffer; Andy Sharp; Sandrine Boulanger
Subject: RE: MD upgrade procedure

We should start w/ the elogs.   Can you provide a pointer to those?

_____________________________________________
From: Raj Kumar=20
Sent: Thursday, May 10, 2007 5:51 PM
To: Brian DeForest; John Keiffer; Andy Sharp; Sandrine Boulanger
Subject: RE: MD upgrade procedure

The workaround tried is based on the error reported by "sys copy", which
may or may not be accurate. We may be missing something (which is also
missed by sys config copy but not reported as error)

_____________________________________________
From: Brian DeForest=20
Sent: Thursday, May 10, 2007 5:47 PM
To: Raj Kumar; John Keiffer; Andy Sharp; Sandrine Boulanger
Subject: RE: MD upgrade procedure

As I understand, the steps performed were exactly the same as what
'system config copy' does.   Suspect this could be a different issue.  =20

_____________________________________________
From: Raj Kumar=20
Sent: Thursday, May 10, 2007 5:36 PM
To: John Keiffer; Andy Sharp; Sandrine Boulanger
Cc: Brian DeForest
Subject: RE: MD upgrade procedure

Also NCM having some problem. I don't like to spend anymore time on this
and fixing this workaround. I think its time we call this off and fix
the original issue (system copy config from 2.2 to 3.0)

Thanks.

_____________________________________________
From: John Keiffer=20
Sent: Thursday, May 10, 2007 5:03 PM
To: Raj Kumar; Andy Sharp; Sandrine Boulanger
Subject: RE: MD upgrade procedure
Importance: High

There are cluster problems.

_____________________________________________
From: Raj Kumar=20
Sent: Thursday, May 10, 2007 4:40 PM
To: Andrew LeFebvre; Sandrine Boulanger; John Keiffer
Subject: MD upgrade procedure

Guys,

Have we nailed the procedure and tested it?

Thanks

------_=_NextPart_001_01C793ED.F7E40E8A
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7652.24">
<TITLE>RE: MD upgrade procedure</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">H10r9 rebooted yesterday at 19:55. I did not =
find</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial"> a reboot cmd in elog, so I =
suspected bsd crash running out of memory, but there is no 0.0 =
crashdump.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Since then, NCM is fin</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">e</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial"> and =
commands are OK.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Could it =
have been something related to the cluster daemons?</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">syslog</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 18:23:59 g10r9 sshd[1206]: Accepted password for =
root from 10.0.0.107 port 1613 ssh2</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 18:31:00 g10r9 sshd[30921]: Accepted password for =
root from 10.0.0.136 port 60219 ssh2</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:07:21 g10r9 sshd[14482]: Accepted password for =
root from 10.0.0.107 port 1899 ssh2</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:55:35 g10r9 syslogd: =
restart</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:55:36 g10r9 /bsd: Copyright (c) 1982, 1986, =
1989, 1991, 1993</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:55:36 g10r9 /bsd:&nbsp;&nbsp;&nbsp;&nbsp; The =
Regents of the University of California.&nbsp; All rights =
reserved.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:55:36 g10r9 /bsd: Copyright (c) 1995-2000 =
OpenBSD. All rights reserved.&nbsp; <A =
HREF=3D"http://www.OpenBSD.org">http://www.OpenBSD.org</A></FONT></SPAN><=
/P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:55:36 g10r9 /bsd: Copyright (c) 1995-2000 =
OpenBSD. All rights reserved.&nbsp;</FONT></SPAN><SPAN LANG=3D"en-us"> =
</SPAN><A HREF=3D"http://www.OpenBSD.org"><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><U><FONT COLOR=3D"#0000FF" =
SIZE=3D2 FACE=3D"Arial">http://www.OpenBSD.org</FONT></U></SPAN><SPAN =
LANG=3D"en-us"></SPAN></A><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">elog</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:42:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:42:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode =
30</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:45:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:45:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode =
30</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:48:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:48:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode =
30</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:51:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:51:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode =
30</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:56:16 g10r9 : 0:0:cm:NOTICE: CHASSISD: Chassis =
Manager: Started</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:56:17 g10r9 : 0:0:cm:NOTICE: CM: opened =
kseg1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:56:32 g10r9 : 0:0:eventd:NOTICE: Process-EVENT =
CPU: Slot 1, CPU 0, State Up</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:56:32 g10r9 : 0:0:eventd:NOTICE: Process-EVENT =
CPU: Slot 1, CPU 1, State Up</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:56:32 g10r9 : 0:0:eventd:NOTICE: Process-EVENT =
CPU: Slot 1, CPU 2, State Up</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:56:32 g10r9 : 0:0:eventd:NOTICE: Process-EVENT =
CPU: Slot 1, CPU 3, State Up</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 19:56:32 g10r9 : 0:0:eventd:NOTICE: Process-EVENT =
CPU: Slot 2, CPU 0, State Up</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">g10r9 diag&gt; cluster show summary</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Cluster Name: g10r9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
Cluster State:&nbsp;&nbsp; On</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">------------------------------------------------------</FO=
NT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">NAS Gateways&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
IP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp; State&nbsp;&nbsp; PCC</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">------------------------------------------------------</FO=
NT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">g10r9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
10.2.9.10&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
UP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NO</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">g11r9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
10.2.9.11&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
UP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; YES</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">------------------------------------------------------</FO=
NT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">PCC =
Name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp; : g11r9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Gateway =
Groups&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : =
default</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">No of Gateway Group(s) : 1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">No of Gateway(s)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : =
2</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">No of Virtual Server(s): 9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">No of Volume(s)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
: 9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">No of online Volume(s) : 2</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">No of offline Volume(s): 7</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Online Volume capacity : 15103.9 MiB</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Percent =
used&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : =
0.42%</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">g10r9 diag&gt; cluster show group</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">GROUP default</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NAS GATEWAY =
g10r9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VSVR VS_MGMT_512</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VSVR JUMBO_G10R9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NAS GATEWAY =
g11r9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VSVR VS_MGMT_903</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VSVR NIS_G10R9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VSVR WINDOWS_G10R9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VSVR LOCAL_NIS_G10R9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VSVR OPENLDAP_G10R9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VSVR AUTHDOM_G10R9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VSVR NONETBIOS_G10R9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NAS GATEWAY =
virtual-filer</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">g10r9 diag&gt; elog show config</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">ELOG configuration</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">------------------</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">ELOG State: Enabled</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">ELOG Display: Disabled</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">ELOG Level: Notice</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">ELOG Facility: local0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">ELOG Host Addr: 0.0.0.0</FONT></SPAN></P>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
9:31 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest; Raj =
Kumar; John Keiffer; Andy Sharp; Chris Vandever<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Here is the relevant part of elog on g10r9 that shows the =
db version upgrade, but also lots of cluster2 errors after =
that.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Sequence of events:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">On g10r9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Before 15:26: first attempt to copy config files with =
tar</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">15:26 system reboot &#8211;s</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">15:30 filer came back up to OCT, found out we had missed =
to mount some partitions</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Rebooted the filer and changed the boot device to go back =
to 2.2.2 flash</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">15:40 chassisd started on 2.2.2</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Copied the config files again</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">15:52 system reboot &#8211;s to boot on 3.0 =
flash</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">15:55 chassisd started on 3.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">15:56 vsvr show all command is OK</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">On g11r9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">16:02 system reboot &#8211;s after copying config =
files</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">On g10r9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">16:03 g11r9 is down, taking over vsvrs and =
volumes</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">16:07 g11r9 is back UP</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">16:12 cluster2 errors (see below)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">16:14 Request to upgrade db failed, running in degraded =
mode, more cluster2 errors</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">16:19 Cluster update complete, more cluster2 errors until =
18:00 and more</FONT></SPAN></P>
<BR>
<BR>
<BR>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:04:31 g10r9 : 0:0:eventd:NOTICE: Process-EVENT =
Vsvr: Virtual server 'WINDOWS_G10R9', Id 4, State 'Up'</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:04:31 g10r9 : 0:0:sanm:NOTICE: Processing =
fail-over event for vsId[4]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:04:46 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster =
show cluster&nbsp; : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:05:42 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cluster =
show cluster&nbsp; : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:05:45 g10r9 : 0:0:nfxsh:NOTICE: cmd[3]: cluster =
show cluster&nbsp; : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:16 g10r9 : 0:0:ncm:WARNING: ncmd : =
ncm_reopen_filer: ncm_open_filer(g11r9) succeeded </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:19 g10r9 : 0:0:ncm:WARNING: ncmd : =
ncm_reopen_filer: ncm_open_filer(g11r9) succeeded </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:23 g10r9 : 0:0:ncm:WARNING: ncmd : =
ncm_reopen_filer: retries to filer(g11r9) failed - aborting =
attemtpts</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:23 g10r9 : 0:0:ncm:WARNING: ncmd : =
ncm_handle_net_msg_done: ncm_repoen filer failed - -1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:30 g10r9 : 0:0:ncm:WARNING: ncmd : =
ncm_handle_ncm_msg_rcv: ncm_forward_to_filer failed - =
-9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:39 g10r9 : 0:0:eventd:NOTICE: Process-EVENT =
Node: Name 'g11r9', State Up, Msg ''</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:43 g10r9 : 0:0:nfxsh:NOTICE: cmd[4]: cluster =
show cluster&nbsp; : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:47 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.2.9.11 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:47 g10r9 : 0:0:cluster2:NOTICE: =
urecovery_Interact: send new file in progress, remote ver =
1178668330(3343), sending to 10.2.9.11 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:55 g10r9 : 0:0:cluster2:NOTICE: =
urecovery_Interact: send new file end, code 0 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:07:30 g10r9 : 0:0:spm:WARNING: spm_sendNcmMsg: =
attempt send before open completed</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:07:30 g10r9 : 0:0:spm:WARNING: spm_sendNcmMsg: =
attempt send before open completed</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:08:03 g10r9 : 0:0:nfxsh:NOTICE: cmd[5]: vol =
show : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:12:03 g10r9 : 0:0:cluster2:NOTICE: ubik init =
with buff size 64 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:12:26 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:12:26 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:12:37 g10r9 : 0:0:spm:ERROR: failed to get =
filer list from cluster daemon </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:12:37 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:07 g10r9 last message repeated 3 =
times</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:12 g10r9 : 0:0:spm:ERROR: failed to get =
filer list from cluster daemon </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:18 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:38 g10r9 last message repeated 2 =
times</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:47 g10r9 : 0:0:spm:ERROR: failed to get =
filer list from cluster daemon </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:48 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:58 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:14:20 g10r9 : 0:0:cluster2:ERROR: =
cluster_request_upgrade_db: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:14:20 g10r9 : 0:0:cluster2:ERROR: =
ClusterCtrl_ProcessTimerFunc: Request to upgrade clusDb failed, rc 30 =
</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:14:20 g10r9 : 0:0:cluster2:ERROR: =
ClusterCtrl_ProcessTimerFunc: Running in degraded mode on version =
0x20200, will retry upgrade to version 0x30000 later... =
</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:14:22 g10r9 : 0:0:spm:ERROR: failed to get =
filer list from cluster daemon </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:14:30 g10r9 : 0:0:cluster2:ERROR: =
cluster_set_lock_on_rpc: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:15:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:15:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:15:35 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:16:15 g10r9 last message repeated 4 =
times</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:16:47 g10r9 last message repeated 4 =
times</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:16:47 g10r9 : 0:0:vsd:ERROR: =
vsd_procGetVsListReq : Cannot get vsId list for node =
g10r9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:16:56 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:16:57 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:16:57 g10r9 : 0:0:vsd:ERROR: =
vsd_procGetVsListReq : Cannot get vsId list for node =
g10r9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:17:41 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.12.9.11 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:26 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:26 g10r9 : 0:0:vsd:ERROR: =
vsd_procGetVsListReq : Cannot get vsId list for node =
g10r9</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:29 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.2.9.11 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:34 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.12.9.11 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:36 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:43 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.2.9.11 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:46 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:48 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.12.9.11 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:56 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:00 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.2.9.11 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:06 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:08 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.12.9.11 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:13 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.2.9.11 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:16 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:25 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.12.9.11 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:27 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:29 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.2.9.11 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:32 g10r9 : 0:0:cluster2:ERROR: =
cluster_upgrade_db_runtime: ClusDb update complete; successfully updated =
version from 0x20200 to 0x30000 </FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:21:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:21:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:21:35 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:24:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:24:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:27:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:27:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:30:14 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:30:14 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:30:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:30:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:33:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:33:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:33:34 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:36:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:36:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:36:35 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:39:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:39:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:39:34 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:41:45 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.107, port 4873; Assigning fd(12)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:41:46 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.107, port 4874; Assigning fd(12)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.107, port 4875; Assigning fd(12)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.107, port 4876; Assigning fd(13)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:10 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system =
show chassis : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.107, port 4884; Assigning fd(12)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.107, port 4885; Assigning fd(13)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:12 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system =
show nodename : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:24 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster =
show summary : status[11]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:25 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:25 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:55 g10r9 : 0:0:nfxsh:NOTICE: cmd[6]: cluster =
show cluster&nbsp; : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:43:13 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:43:13 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: cluster =
show summary : status[11]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:43:31 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system =
show nodename : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:43:42 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:43:42 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster =
show summary : status[11]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:44:43 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:44:48 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:45:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:45:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:14 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.154, port 3315; Assigning fd(12)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:19 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.154, port 3317; Assigning fd(13)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:19 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:19 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.154, port 3318; Assigning fd(13)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system =
show chassis : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.154, port 3319; Assigning fd(13)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.154, port 3320; Assigning fd(12)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.154, port 3321; Assigning fd(13)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.154, port 3322; Assigning fd(12)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.154, port 3323; Assigning fd(13)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:32 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system =
show nodename : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:44 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:44 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster =
show summary : status[11]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:49:37 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: elog =
show log 20 : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:49:55 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:49:55 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster =
show summary : status[11]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:50:36 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:50:36 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cl show =
gr : status[11]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:51:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:51:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:51:34 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:53:25 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: cl show =
cl : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:53:39 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:53:39 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cl show =
gr : status[11]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:53:59 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:53:59 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:54:03 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:54:03 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cl show =
sum : status[11]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:54:21 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:54:21 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:54:21 g10r9 : 0:0:nfxsh:NOTICE: cmd[3]: vsvr =
show all : status[11]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:56:52 g10r9 : 0:0:nfxsh:NOTICE: cmd[4]: elog =
show log : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:57:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:57:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:58:31 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.154, port 3349; Assigning fd(12)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:58:44 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:58:44 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: cluster =
show summary : status[11]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:58:47 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.154, port 3350; Assigning fd(13)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:58:53 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:58:53 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:00:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:00:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:00:34 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:03:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:03:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:06:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:06:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:09:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:09:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:12:00 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:12:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:12:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:14:40 g10r9 : 0:0:cluster2:NOTICE: ubik init =
with buff size 64 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:14:40 g10r9 : 0:0:pm:ERROR: pm_sig_handler: =
/usr/local/agile/bin/vsd (pid 27612) terminated with signal Killed =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:14:41 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: volume =
show -r : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:53 g10r9 : 1:0:ipm:NOTICE: 21: =
vstack_vlink_create:294 : vl FO active 1 prefered 1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:53 g10r9 : 1:0:ipm:NOTICE: 22: =
vstack_vlink_create:329: vlink lp link op-state DOWN </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:54 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D1 already exists. =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:54 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D10 already exists. =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (nis-vol) current state - Mounted-RW State.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (local) current state - Mounted-RW State.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (win-vol) current state - Mounted-RW State.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (local-nis-vol) current state - Mounted-RW =
State.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (openldap-vol) current state - Mounted-RW =
State.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (authdom-vol) current state - Mounted-RW State.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (jumbo-vol) current state - Mounted-RW State.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (vol_mgmt_512) current state - Mounted-RW =
State.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D3 already exists. =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D4 already exists. =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D5 already exists. =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D6 already exists. =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxldap_createVsContext[257]: Context for VS=3D6 already exists. =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D7 already exists. =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxldap_createVsContext[257]: Context for VS=3D7 already exists. =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D8 already exists. =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:18:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:18:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:21:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:21:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:24:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:24:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:25:03 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: volume =
show -r : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:27:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:27:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:30:15 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:30:15 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:30:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:30:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:33:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:33:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:36:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:36:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:39:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:39:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:42:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:42:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:45:45 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: elog =
find cluster : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:46:59 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: elog =
find upgrade : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:47:25 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cluster =
show summary&nbsp; : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:48:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:48:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:51:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:51:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:52:11 g10r9 : 0:0:nfxsh:NOTICE: cmd[3]: system =
version -s : status[0]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:54:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:54:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:58:05 g10r9 : 0:0:nfxsh:NOTICE: cmd[4]: elog =
show config&nbsp; : status[11]</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:58:05 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:58:05 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 18:00:02 g10r9 newsyslog[12273]: logfile turned =
over</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
8:44 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Sandrine Boulanger; Raj =
Kumar; John Keiffer; Andy Sharp; Chris Vandever<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Subsequent testing showed the cluster commands repeatedly =
working ok from the CLI on both nodes, though admittedly this was just =
one test.&nbsp;&nbsp; John was going to test some more via =
NCM.&nbsp;&nbsp; I think John plans to send a more detailed update as =
well.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
8:34 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest; Raj =
Kumar; John Keiffer; Andy Sharp<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">g10r9 and g11r9.onstor.lab</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">I&#8217;ll go through the elog right now to see if I can =
catch anything. It does not look like this would be related to the conf =
file since &#8216;cluster show summary&#8217; uses the cluster db, which =
has been upgraded to version 3.0 on both filers (I already verified =
that).</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
5:55 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar; John Keiffer; =
Andy Sharp; Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">We should start w/ the elogs.&nbsp;&nbsp; Can you provide =
a pointer to those?</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
5:51 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest; John =
Keiffer; Andy Sharp; Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">The workaround tried is based on the error reported by =
&#8220;sys copy&#8221;, which may or may not be accurate. We may be =
missing something (which is also missed by sys config copy but not =
reported as error)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
5:47 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar; John Keiffer; =
Andy Sharp; Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">As I understand, the steps performed were exactly the =
same as what &#8216;system config copy&#8217; does.&nbsp;&nbsp; Suspect =
this could be a different issue.&nbsp;&nbsp; </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
5:36 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> John Keiffer; Andy Sharp; =
Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Also NCM having some problem. I don&#8217;t like to spend =
anymore time on this and fixing this workaround. I think its time we =
call this off and fix the original issue (system copy config from 2.2 to =
3.0)</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Thanks.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> John Keiffer<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
5:03 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar; Andy Sharp; =
Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Importance:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> High</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">There are cluster problems.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
4:40 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Andrew LeFebvre; Sandrine =
Boulanger; John Keiffer<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Guys,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Have =
we nailed the procedure and tested it?</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Thanks</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

</BODY>
</HTML>
------_=_NextPart_001_01C793ED.F7E40E8A--
