X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C793F7.DF039416@onstor-exch02.onstor.net>; Fri, 11 May 2007 11:12:01 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C793F7.DF039416"
Content-class: urn:content-classes:message
Subject: RE: MD upgrade procedure
Date: Fri, 11 May 2007 11:11:59 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E03A8E4C7@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E0138C667@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: MD upgrade procedure
Thread-Index: AceTXJdVzjWTSchUSru5u9TaQDK4VwAAxUkgAAEf1yAAAGY0kAAAGBuQAAAtxZAABX23oAAAVe3gAAFrfVAAG+0b4AAAajHAAAAOBpAAAJYfQA==
References: <BB375AF679D4A34E9CA8DFA650E2B04E03A8E493@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E0138C667@onstor-exch02.onstor.net>
From: "Brian DeForest" <brian.deforest@onstor.com>
To: "Chris Vandever" <chris.vandever@onstor.com>,
	"Sandrine Boulanger" <sandrine.boulanger@onstor.com>,
	"Raj Kumar" <raj.kumar@onstor.com>,
	"John Keiffer" <john.keiffer@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C793F7.DF039416
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

The goal is to get MD installed.   If there's an old bug, we can work on
that in parallel.   I suggest we resume the installation.

_____________________________________________
From: Chris Vandever=20
Sent: Friday, May 11, 2007 10:57 AM
To: Brian DeForest; Sandrine Boulanger; Raj Kumar; John Keiffer; Andy
Sharp
Subject: RE: MD upgrade procedure

I think you're jumping the gun.  We need to confirm that restarting ncmd
fixes the problem.

Plus, I believe the subsequent problems are RMC related, not necessarily
ncmd, so I'm concerned that restarting ncmd may not be sufficient for
them.

ChrisV

_____________________________________________
From: Brian DeForest=20
Sent: Friday, May 11, 2007 10:54 AM
To: Chris Vandever; Sandrine Boulanger; Raj Kumar; John Keiffer; Andy
Sharp
Subject: RE: MD upgrade procedure

So since this is not a new problem, can we continue w/ the MD upgrade
today then?

_____________________________________________
From: Chris Vandever=20
Sent: Friday, May 11, 2007 10:48 AM
To: Sandrine Boulanger; Brian DeForest; Raj Kumar; John Keiffer; Andy
Sharp
Subject: RE: MD upgrade procedure

It looks like an ncmd problem to me.  We've seen this problem in the
field.  The filer running the ncmd that is NOT complaining is the one
whose ncmd is broken and needs to be restarted.

May 10 16:06:23 g10r9 : 0:0:ncm:WARNING: ncmd : ncm_handle_net_msg_done:
ncm_repoen filer failed - -1
May 10 16:06:30 g10r9 : 0:0:ncm:WARNING: ncmd : ncm_handle_ncm_msg_rcv:
ncm_forward_to_filer failed - -9

Clearly there's a communication problem.  All of the clustering
complaints are due to an RMC communication failure between the cluster
APIs in libcluster.so and cluster_contrl.

May 10 16:12:26 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20

ChrisV
_____________________________________________
From: Sandrine Boulanger=20
Sent: Thursday, May 10, 2007 9:31 PM
To: Brian DeForest; Raj Kumar; John Keiffer; Andy Sharp; Chris Vandever
Subject: RE: MD upgrade procedure

Here is the relevant part of elog on g10r9 that shows the db version
upgrade, but also lots of cluster2 errors after that.

Sequence of events:
On g10r9
Before 15:26: first attempt to copy config files with tar
15:26 system reboot -s
15:30 filer came back up to OCT, found out we had missed to mount some
partitions
Rebooted the filer and changed the boot device to go back to 2.2.2 flash
15:40 chassisd started on 2.2.2
Copied the config files again
15:52 system reboot -s to boot on 3.0 flash
15:55 chassisd started on 3.0
15:56 vsvr show all command is OK

On g11r9
16:02 system reboot -s after copying config files

On g10r9
16:03 g11r9 is down, taking over vsvrs and volumes
16:07 g11r9 is back UP
16:12 cluster2 errors (see below)
16:14 Request to upgrade db failed, running in degraded mode, more
cluster2 errors
16:19 Cluster update complete, more cluster2 errors until 18:00 and more




May 10 16:04:31 g10r9 : 0:0:eventd:NOTICE: Process-EVENT Vsvr: Virtual
server 'WINDOWS_G10R9', Id 4, State 'Up'
May 10 16:04:31 g10r9 : 0:0:sanm:NOTICE: Processing fail-over event for
vsId[4]
May 10 16:04:46 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show cluster
: status[0]
May 10 16:05:42 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cluster show cluster
: status[0]
May 10 16:05:45 g10r9 : 0:0:nfxsh:NOTICE: cmd[3]: cluster show cluster
: status[0]
May 10 16:06:16 g10r9 : 0:0:ncm:WARNING: ncmd : ncm_reopen_filer:
ncm_open_filer(g11r9) succeeded=20
May 10 16:06:19 g10r9 : 0:0:ncm:WARNING: ncmd : ncm_reopen_filer:
ncm_open_filer(g11r9) succeeded=20
May 10 16:06:23 g10r9 : 0:0:ncm:WARNING: ncmd : ncm_reopen_filer:
retries to filer(g11r9) failed - aborting attemtpts
May 10 16:06:23 g10r9 : 0:0:ncm:WARNING: ncmd : ncm_handle_net_msg_done:
ncm_repoen filer failed - -1
May 10 16:06:30 g10r9 : 0:0:ncm:WARNING: ncmd : ncm_handle_ncm_msg_rcv:
ncm_forward_to_filer failed - -9
May 10 16:06:39 g10r9 : 0:0:eventd:NOTICE: Process-EVENT Node: Name
'g11r9', State Up, Msg ''
May 10 16:06:43 g10r9 : 0:0:nfxsh:NOTICE: cmd[4]: cluster show cluster
: status[0]
May 10 16:06:47 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.2.9.11=20
May 10 16:06:47 g10r9 : 0:0:cluster2:NOTICE: urecovery_Interact: send
new file in progress, remote ver 1178668330(3343), sending to 10.2.9.11=20
May 10 16:06:55 g10r9 : 0:0:cluster2:NOTICE: urecovery_Interact: send
new file end, code 0=20
May 10 16:07:30 g10r9 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
before open completed
May 10 16:07:30 g10r9 : 0:0:spm:WARNING: spm_sendNcmMsg: attempt send
before open completed
May 10 16:08:03 g10r9 : 0:0:nfxsh:NOTICE: cmd[5]: vol show : status[0]
May 10 16:12:03 g10r9 : 0:0:cluster2:NOTICE: ubik init with buff size 64

May 10 16:12:26 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:12:26 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:12:37 g10r9 : 0:0:spm:ERROR: failed to get filer list from
cluster daemon=20
May 10 16:12:37 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:13:07 g10r9 last message repeated 3 times
May 10 16:13:12 g10r9 : 0:0:spm:ERROR: failed to get filer list from
cluster daemon=20
May 10 16:13:18 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:13:38 g10r9 last message repeated 2 times
May 10 16:13:47 g10r9 : 0:0:spm:ERROR: failed to get filer list from
cluster daemon=20
May 10 16:13:48 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:13:58 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:14:20 g10r9 : 0:0:cluster2:ERROR: cluster_request_upgrade_db:
no reply bck -1=20
May 10 16:14:20 g10r9 : 0:0:cluster2:ERROR:
ClusterCtrl_ProcessTimerFunc: Request to upgrade clusDb failed, rc 30=20
May 10 16:14:20 g10r9 : 0:0:cluster2:ERROR:
ClusterCtrl_ProcessTimerFunc: Running in degraded mode on version
0x20200, will retry upgrade to version 0x30000 later...=20
May 10 16:14:22 g10r9 : 0:0:spm:ERROR: failed to get filer list from
cluster daemon=20
May 10 16:14:30 g10r9 : 0:0:cluster2:ERROR: cluster_set_lock_on_rpc: no
reply bck -1=20
May 10 16:15:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:15:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:15:35 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:16:15 g10r9 last message repeated 4 times
May 10 16:16:47 g10r9 last message repeated 4 times
May 10 16:16:47 g10r9 : 0:0:vsd:ERROR: vsd_procGetVsListReq : Cannot get
vsId list for node g10r9
May 10 16:16:56 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:16:57 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:16:57 g10r9 : 0:0:vsd:ERROR: vsd_procGetVsListReq : Cannot get
vsId list for node g10r9
May 10 16:17:41 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.12.9.11=20
May 10 16:18:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:18:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:18:26 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:18:26 g10r9 : 0:0:vsd:ERROR: vsd_procGetVsListReq : Cannot get
vsId list for node g10r9
May 10 16:18:29 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.2.9.11=20
May 10 16:18:34 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.12.9.11=20
May 10 16:18:36 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:18:43 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.2.9.11=20
May 10 16:18:46 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:18:48 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.12.9.11=20
May 10 16:18:56 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:19:00 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.2.9.11=20
May 10 16:19:06 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:19:08 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.12.9.11=20
May 10 16:19:13 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.2.9.11=20
May 10 16:19:16 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:19:25 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.12.9.11=20
May 10 16:19:27 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:19:29 g10r9 : 0:0:cluster2:NOTICE: ubik:server 10.2.9.11 is
back up: will be contacted through 10.2.9.11=20
May 10 16:19:32 g10r9 : 0:0:cluster2:ERROR: cluster_upgrade_db_runtime:
ClusDb update complete; successfully updated version from 0x20200 to
0x30000=20
May 10 16:21:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:21:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:21:35 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:24:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:24:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:27:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:27:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:30:14 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:30:14 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:30:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:30:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:33:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:33:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:33:34 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:36:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:36:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:36:35 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:39:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:39:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:39:34 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:41:45 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.107, port 4873; Assigning fd(12)
May 10 16:41:46 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.107, port 4874; Assigning fd(12)
May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.107, port 4875; Assigning fd(12)
May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.107, port 4876; Assigning fd(13)
May 10 16:42:10 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system show chassis :
status[0]
May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.107, port 4884; Assigning fd(12)
May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.107, port 4885; Assigning fd(13)
May 10 16:42:12 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system show nodename :
status[0]
May 10 16:42:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:42:24 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show summary :
status[11]
May 10 16:42:25 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:42:25 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:42:55 g10r9 : 0:0:nfxsh:NOTICE: cmd[6]: cluster show cluster
: status[0]
May 10 16:43:13 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:43:13 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: cluster show summary :
status[11]
May 10 16:43:31 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system show nodename :
status[0]
May 10 16:43:42 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:43:42 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show summary :
status[11]
May 10 16:44:43 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:44:48 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:45:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:45:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:48:14 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.154, port 3315; Assigning fd(12)
May 10 16:48:19 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.154, port 3317; Assigning fd(13)
May 10 16:48:19 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:48:19 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.154, port 3318; Assigning fd(13)
May 10 16:48:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:48:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:48:31 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system show chassis :
status[0]
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.154, port 3319; Assigning fd(13)
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.154, port 3320; Assigning fd(12)
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.154, port 3321; Assigning fd(13)
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.154, port 3322; Assigning fd(12)
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.154, port 3323; Assigning fd(13)
May 10 16:48:32 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system show nodename :
status[0]
May 10 16:48:44 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:48:44 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show summary :
status[11]
May 10 16:49:37 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: elog show log 20 :
status[0]
May 10 16:49:55 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:49:55 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster show summary :
status[11]
May 10 16:50:36 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:50:36 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cl show gr :
status[11]
May 10 16:51:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:51:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:51:34 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:53:25 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: cl show cl : status[0]
May 10 16:53:39 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:53:39 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cl show gr :
status[11]
May 10 16:53:59 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 16:53:59 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:54:03 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:54:03 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cl show sum :
status[11]
May 10 16:54:21 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:54:21 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:54:21 g10r9 : 0:0:nfxsh:NOTICE: cmd[3]: vsvr show all :
status[11]
May 10 16:56:52 g10r9 : 0:0:nfxsh:NOTICE: cmd[4]: elog show log :
status[0]
May 10 16:57:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:57:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 16:58:31 g10r9 : 0:0:sscccc:NOTICE: New client 0: addr
10.0.0.154, port 3349; Assigning fd(12)
May 10 16:58:44 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 16:58:44 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: cluster show summary :
status[11]
May 10 16:58:47 g10r9 : 0:0:sscccc:NOTICE: New client 1: addr
10.0.0.154, port 3350; Assigning fd(13)
May 10 16:58:53 g10r9 : 0:0:sscccc:NOTICE: closing session for client 1
May 10 16:58:53 g10r9 : 0:0:sscccc:NOTICE: closing session for client 0
May 10 17:00:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:00:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:00:34 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:03:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:03:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:06:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:06:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:09:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:09:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:12:00 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:12:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:12:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:14:40 g10r9 : 0:0:cluster2:NOTICE: ubik init with buff size 64

May 10 17:14:40 g10r9 : 0:0:pm:ERROR: pm_sig_handler:
/usr/local/agile/bin/vsd (pid 27612) terminated with signal Killed=20
May 10 17:14:41 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: volume show -r :
status[0]
May 10 17:16:53 g10r9 : 1:0:ipm:NOTICE: 21: vstack_vlink_create:294 : vl
FO active 1 prefered 1=20
May 10 17:16:53 g10r9 : 1:0:ipm:NOTICE: 22: vstack_vlink_create:329:
vlink lp link op-state DOWN=20
May 10 17:16:54 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D1 already exists.=20
May 10 17:16:54 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D10 already exists.=20
May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume (nis-vol)
current state - Mounted-RW State.
May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume (local)
current state - Mounted-RW State.
May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume (win-vol)
current state - Mounted-RW State.
May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume
(local-nis-vol) current state - Mounted-RW State.
May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume
(openldap-vol) current state - Mounted-RW State.
May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume
(authdom-vol) current state - Mounted-RW State.
May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume (jumbo-vol)
current state - Mounted-RW State.
May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : Volume
(vol_mgmt_512) current state - Mounted-RW State.
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D3 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D4 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D5 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D6 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxldap_createVsContext[257]:
Context for VS=3D6 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D7 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxldap_createVsContext[257]:
Context for VS=3D7 already exists.=20
May 10 17:17:00 g10r9 : 0:0:ea:WARNING: nfxnis_createVsContext: Context
for VS=3D8 already exists.=20
May 10 17:18:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:18:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:21:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:21:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:24:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:24:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:25:03 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: volume show -r :
status[0]
May 10 17:27:22 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:27:22 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:30:15 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:30:15 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:30:24 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:30:24 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:33:22 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:33:22 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:36:22 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:36:22 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:39:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:39:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:42:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:42:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:45:45 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: elog find cluster :
status[0]
May 10 17:46:59 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: elog find upgrade :
status[0]
May 10 17:47:25 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cluster show summary
: status[0]
May 10 17:48:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:48:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:51:23 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:51:23 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:52:11 g10r9 : 0:0:nfxsh:NOTICE: cmd[3]: system version -s :
status[0]
May 10 17:54:22 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:54:22 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 17:58:05 g10r9 : 0:0:nfxsh:NOTICE: cmd[4]: elog show config  :
status[11]
May 10 17:58:05 g10r9 : 0:0:cluster2:ERROR: cluster_getRecordIdByKey: no
reply bck -1=20
May 10 17:58:05 g10r9 : 0:0:cluster2:ERROR: cluster_getFilerNameList:
cannot get cluster rec, rcode 30=20
May 10 18:00:02 g10r9 newsyslog[12273]: logfile turned over

_____________________________________________
From: Brian DeForest=20
Sent: Thursday, May 10, 2007 8:44 PM
To: Sandrine Boulanger; Raj Kumar; John Keiffer; Andy Sharp; Chris
Vandever
Subject: RE: MD upgrade procedure

Subsequent testing showed the cluster commands repeatedly working ok
from the CLI on both nodes, though admittedly this was just one test.
John was going to test some more via NCM.   I think John plans to send a
more detailed update as well.

_____________________________________________
From: Sandrine Boulanger=20
Sent: Thursday, May 10, 2007 8:34 PM
To: Brian DeForest; Raj Kumar; John Keiffer; Andy Sharp
Subject: RE: MD upgrade procedure

g10r9 and g11r9.onstor.lab
I'll go through the elog right now to see if I can catch anything. It
does not look like this would be related to the conf file since 'cluster
show summary' uses the cluster db, which has been upgraded to version
3.0 on both filers (I already verified that).

_____________________________________________
From: Brian DeForest=20
Sent: Thursday, May 10, 2007 5:55 PM
To: Raj Kumar; John Keiffer; Andy Sharp; Sandrine Boulanger
Subject: RE: MD upgrade procedure

We should start w/ the elogs.   Can you provide a pointer to those?

_____________________________________________
From: Raj Kumar=20
Sent: Thursday, May 10, 2007 5:51 PM
To: Brian DeForest; John Keiffer; Andy Sharp; Sandrine Boulanger
Subject: RE: MD upgrade procedure

The workaround tried is based on the error reported by "sys copy", which
may or may not be accurate. We may be missing something (which is also
missed by sys config copy but not reported as error)

_____________________________________________
From: Brian DeForest=20
Sent: Thursday, May 10, 2007 5:47 PM
To: Raj Kumar; John Keiffer; Andy Sharp; Sandrine Boulanger
Subject: RE: MD upgrade procedure

As I understand, the steps performed were exactly the same as what
'system config copy' does.   Suspect this could be a different issue.  =20

_____________________________________________
From: Raj Kumar=20
Sent: Thursday, May 10, 2007 5:36 PM
To: John Keiffer; Andy Sharp; Sandrine Boulanger
Cc: Brian DeForest
Subject: RE: MD upgrade procedure

Also NCM having some problem. I don't like to spend anymore time on this
and fixing this workaround. I think its time we call this off and fix
the original issue (system copy config from 2.2 to 3.0)

Thanks.

_____________________________________________
From: John Keiffer=20
Sent: Thursday, May 10, 2007 5:03 PM
To: Raj Kumar; Andy Sharp; Sandrine Boulanger
Subject: RE: MD upgrade procedure
Importance: High

There are cluster problems.

_____________________________________________
From: Raj Kumar=20
Sent: Thursday, May 10, 2007 4:40 PM
To: Andrew LeFebvre; Sandrine Boulanger; John Keiffer
Subject: MD upgrade procedure

Guys,

Have we nailed the procedure and tested it?

Thanks

------_=_NextPart_001_01C793F7.DF039416
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7652.24">
<TITLE>RE: MD upgrade procedure</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">The goal is to get MD =
installed.&nbsp;&nbsp; If there</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">&#8217;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial">s an old bug, we can work on that in =
parallel.&nbsp;&nbsp; I suggest we resume the =
installation.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Chris Vandever<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Friday, May 11, 2007 =
10:57 AM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest; Sandrine =
Boulanger; Raj Kumar; John Keiffer; Andy Sharp<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">I think you&#8217;re jumping =
the gun.&nbsp; We need to confirm that restarting ncmd fixes the =
problem.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Plus, I believe the subsequent problems are RMC related, =
not necessarily ncmd, so I&#8217;m concerned that restarting ncmd may =
not be sufficient for them.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">ChrisV</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Friday, May 11, 2007 =
10:54 AM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Chris Vandever; Sandrine =
Boulanger; Raj Kumar; John Keiffer; Andy Sharp<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">So since this is not a new =
problem, can we continue w/ the MD upgrade today then?</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Chris Vandever<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Friday, May 11, 2007 =
10:48 AM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Sandrine Boulanger; Brian =
DeForest; Raj Kumar; John Keiffer; Andy Sharp<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">It looks like an ncmd problem =
to me.&nbsp; We&#8217;ve seen this problem in the field.&nbsp; The filer =
running the ncmd that is NOT complaining is the one whose ncmd is broken =
and needs to be restarted.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:23 g10r9 : 0:0:ncm:WARNING: ncmd : =
ncm_handle_net_msg_done: ncm_repoen filer failed - -1</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:30 g10r9 : 0:0:ncm:WARNING: ncmd : =
ncm_handle_ncm_msg_rcv: ncm_forward_to_filer failed - =
-9</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Clearly there&#8217;s a =
communication problem.&nbsp; All of the clustering complaints are due to =
an RMC communication failure between the cluster APIs in libcluster.so =
and cluster_contrl.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">May 10 16:12:26 g10r9 : =
0:0:cluster2:ERROR: cluster_getRecordIdByKey: no reply bck -1 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">ChrisV</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
9:31 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest; Raj =
Kumar; John Keiffer; Andy Sharp; Chris Vandever<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Here is the relevant part of =
elog on g10r9 that shows the db version upgrade, but also lots of =
cluster2 errors after that.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Sequence of events:</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">On g10r9</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Before 15:26: first attempt to copy config files with =
tar</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">15:26 system reboot &#8211;s</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">15:30 filer came back up to OCT, found out we had missed =
to mount some partitions</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Rebooted the filer and changed the boot device to go back =
to 2.2.2 flash</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">15:40 chassisd started on 2.2.2</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Copied the config files again</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">15:52 system reboot &#8211;s to boot on 3.0 =
flash</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">15:55 chassisd started on 3.0</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">15:56 vsvr show all command is OK</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">On g11r9</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">16:02 system reboot &#8211;s after copying config =
files</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">On g10r9</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">16:03 g11r9 is down, taking over vsvrs and =
volumes</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">16:07 g11r9 is back UP</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">16:12 cluster2 errors (see below)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">16:14 Request to upgrade db failed, running in degraded =
mode, more cluster2 errors</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">16:19 Cluster update complete, more cluster2 errors until =
18:00 and more</FONT></SPAN></P>
<BR>
<BR>
<BR>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:04:31 g10r9 : 0:0:eventd:NOTICE: Process-EVENT =
Vsvr: Virtual server 'WINDOWS_G10R9', Id 4, State 'Up'</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:04:31 g10r9 : 0:0:sanm:NOTICE: Processing =
fail-over event for vsId[4]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:04:46 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster =
show cluster&nbsp; : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:05:42 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cluster =
show cluster&nbsp; : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:05:45 g10r9 : 0:0:nfxsh:NOTICE: cmd[3]: cluster =
show cluster&nbsp; : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:16 g10r9 : 0:0:ncm:WARNING: ncmd : =
ncm_reopen_filer: ncm_open_filer(g11r9) succeeded </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:19 g10r9 : 0:0:ncm:WARNING: ncmd : =
ncm_reopen_filer: ncm_open_filer(g11r9) succeeded </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:23 g10r9 : 0:0:ncm:WARNING: ncmd : =
ncm_reopen_filer: retries to filer(g11r9) failed - aborting =
attemtpts</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:23 g10r9 : 0:0:ncm:WARNING: ncmd : =
ncm_handle_net_msg_done: ncm_repoen filer failed - -1</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:30 g10r9 : 0:0:ncm:WARNING: ncmd : =
ncm_handle_ncm_msg_rcv: ncm_forward_to_filer failed - =
-9</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:39 g10r9 : 0:0:eventd:NOTICE: Process-EVENT =
Node: Name 'g11r9', State Up, Msg ''</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:43 g10r9 : 0:0:nfxsh:NOTICE: cmd[4]: cluster =
show cluster&nbsp; : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:47 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.2.9.11 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:47 g10r9 : 0:0:cluster2:NOTICE: =
urecovery_Interact: send new file in progress, remote ver =
1178668330(3343), sending to 10.2.9.11 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:06:55 g10r9 : 0:0:cluster2:NOTICE: =
urecovery_Interact: send new file end, code 0 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:07:30 g10r9 : 0:0:spm:WARNING: spm_sendNcmMsg: =
attempt send before open completed</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:07:30 g10r9 : 0:0:spm:WARNING: spm_sendNcmMsg: =
attempt send before open completed</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:08:03 g10r9 : 0:0:nfxsh:NOTICE: cmd[5]: vol =
show : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:12:03 g10r9 : 0:0:cluster2:NOTICE: ubik init =
with buff size 64 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:12:26 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:12:26 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:12:37 g10r9 : 0:0:spm:ERROR: failed to get =
filer list from cluster daemon </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:12:37 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:07 g10r9 last message repeated 3 =
times</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:12 g10r9 : 0:0:spm:ERROR: failed to get =
filer list from cluster daemon </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:18 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:38 g10r9 last message repeated 2 =
times</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:47 g10r9 : 0:0:spm:ERROR: failed to get =
filer list from cluster daemon </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:48 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:13:58 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:14:20 g10r9 : 0:0:cluster2:ERROR: =
cluster_request_upgrade_db: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">May 10 =
16:14:20 g10r9 : 0:0:cluster2:ERROR: ClusterCtrl_ProcessTimerFunc: =
Request to upgrade clusDb failed, rc 30 </FONT></B></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><B><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:14:20 g10r9 : 0:0:cluster2:ERROR: =
ClusterCtrl_ProcessTimerFunc: Running in degraded mode on version =
0x20200, will retry upgrade to version 0x30000 later... =
</FONT></B></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">May 10 16:14:22 g10r9 : =
0:0:spm:ERROR: failed to get filer list from cluster daemon =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:14:30 g10r9 : 0:0:cluster2:ERROR: =
cluster_set_lock_on_rpc: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:15:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:15:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:15:35 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:16:15 g10r9 last message repeated 4 =
times</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:16:47 g10r9 last message repeated 4 =
times</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:16:47 g10r9 : 0:0:vsd:ERROR: =
vsd_procGetVsListReq : Cannot get vsId list for node =
g10r9</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:16:56 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:16:57 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:16:57 g10r9 : 0:0:vsd:ERROR: =
vsd_procGetVsListReq : Cannot get vsId list for node =
g10r9</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:17:41 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.12.9.11 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:26 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:26 g10r9 : 0:0:vsd:ERROR: =
vsd_procGetVsListReq : Cannot get vsId list for node =
g10r9</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:29 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.2.9.11 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:34 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.12.9.11 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:36 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:43 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.2.9.11 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:46 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:48 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.12.9.11 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:18:56 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:00 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.2.9.11 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:06 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:08 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.12.9.11 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:13 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.2.9.11 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:16 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:25 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.12.9.11 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:27 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:19:29 g10r9 : 0:0:cluster2:NOTICE: ubik:server =
10.2.9.11 is back up: will be contacted through 10.2.9.11 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">May 10 =
16:19:32 g10r9 : 0:0:cluster2:ERROR: cluster_upgrade_db_runtime: ClusDb =
update complete; successfully updated version from 0x20200 to 0x30000 =
</FONT></B></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">May 10 16:21:24 g10r9 : =
0:0:cluster2:ERROR: cluster_getRecordIdByKey: no reply bck -1 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:21:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:21:35 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:24:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:24:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:27:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:27:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:30:14 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:30:14 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:30:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:30:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:33:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:33:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:33:34 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:36:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:36:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:36:35 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:39:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:39:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:39:34 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:41:45 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.107, port 4873; Assigning fd(12)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:41:46 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.107, port 4874; Assigning fd(12)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.107, port 4875; Assigning fd(12)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:41:48 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.107, port 4876; Assigning fd(13)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:10 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system =
show chassis : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.107, port 4884; Assigning fd(12)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:11 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.107, port 4885; Assigning fd(13)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:12 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system =
show nodename : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:24 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster =
show summary : status[11]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:25 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:25 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:42:55 g10r9 : 0:0:nfxsh:NOTICE: cmd[6]: cluster =
show cluster&nbsp; : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:43:13 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:43:13 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: cluster =
show summary : status[11]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:43:31 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system =
show nodename : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:43:42 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:43:42 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster =
show summary : status[11]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:44:43 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:44:48 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:45:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:45:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:14 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.154, port 3315; Assigning fd(12)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:19 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.154, port 3317; Assigning fd(13)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:19 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:19 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.154, port 3318; Assigning fd(13)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system =
show chassis : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.154, port 3319; Assigning fd(13)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.154, port 3320; Assigning fd(12)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.154, port 3321; Assigning fd(13)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.154, port 3322; Assigning fd(12)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:31 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.154, port 3323; Assigning fd(13)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:32 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: system =
show nodename : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:44 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:48:44 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster =
show summary : status[11]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:49:37 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: elog =
show log 20 : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:49:55 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:49:55 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cluster =
show summary : status[11]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:50:36 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:50:36 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cl show =
gr : status[11]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:51:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:51:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:51:34 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:53:25 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: cl show =
cl : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:53:39 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:53:39 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: cl show =
gr : status[11]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:53:59 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:53:59 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:54:03 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:54:03 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cl show =
sum : status[11]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:54:21 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:54:21 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:54:21 g10r9 : 0:0:nfxsh:NOTICE: cmd[3]: vsvr =
show all : status[11]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:56:52 g10r9 : 0:0:nfxsh:NOTICE: cmd[4]: elog =
show log : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:57:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:57:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:58:31 g10r9 : 0:0:sscccc:NOTICE: New client 0: =
addr 10.0.0.154, port 3349; Assigning fd(12)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:58:44 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:58:44 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: cluster =
show summary : status[11]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:58:47 g10r9 : 0:0:sscccc:NOTICE: New client 1: =
addr 10.0.0.154, port 3350; Assigning fd(13)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:58:53 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 1</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 16:58:53 g10r9 : 0:0:sscccc:NOTICE: closing =
session for client 0</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:00:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:00:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:00:34 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:03:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:03:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:06:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:06:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:09:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:09:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:12:00 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:12:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:12:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:14:40 g10r9 : 0:0:cluster2:NOTICE: ubik init =
with buff size 64 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:14:40 g10r9 : 0:0:pm:ERROR: pm_sig_handler: =
/usr/local/agile/bin/vsd (pid 27612) terminated with signal Killed =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:14:41 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: volume =
show -r : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:53 g10r9 : 1:0:ipm:NOTICE: 21: =
vstack_vlink_create:294 : vl FO active 1 prefered 1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:53 g10r9 : 1:0:ipm:NOTICE: 22: =
vstack_vlink_create:329: vlink lp link op-state DOWN </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:54 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D1 already exists. =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:54 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D10 already exists. =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (nis-vol) current state - Mounted-RW State.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (local) current state - Mounted-RW State.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (win-vol) current state - Mounted-RW State.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:54 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (local-nis-vol) current state - Mounted-RW =
State.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (openldap-vol) current state - Mounted-RW =
State.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (authdom-vol) current state - Mounted-RW State.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (jumbo-vol) current state - Mounted-RW State.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:16:56 g10r9 : 0:0:ea:NOTICE: ea_movInProc : =
Volume (vol_mgmt_512) current state - Mounted-RW =
State.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D3 already exists. =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D4 already exists. =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D5 already exists. =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D6 already exists. =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxldap_createVsContext[257]: Context for VS=3D6 already exists. =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D7 already exists. =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxldap_createVsContext[257]: Context for VS=3D7 already exists. =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:17:00 g10r9 : 0:0:ea:WARNING: =
nfxnis_createVsContext: Context for VS=3D8 already exists. =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:18:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:18:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:21:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:21:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:24:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:24:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:25:03 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: volume =
show -r : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:27:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:27:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:30:15 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:30:15 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:30:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:30:24 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:33:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:33:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:36:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:36:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:39:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:39:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:42:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:42:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:45:45 g10r9 : 0:0:nfxsh:NOTICE: cmd[0]: elog =
find cluster : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:46:59 g10r9 : 0:0:nfxsh:NOTICE: cmd[1]: elog =
find upgrade : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:47:25 g10r9 : 0:0:nfxsh:NOTICE: cmd[2]: cluster =
show summary&nbsp; : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:48:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:48:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:51:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:51:23 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:52:11 g10r9 : 0:0:nfxsh:NOTICE: cmd[3]: system =
version -s : status[0]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:54:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:54:22 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:58:05 g10r9 : 0:0:nfxsh:NOTICE: cmd[4]: elog =
show config&nbsp; : status[11]</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:58:05 g10r9 : 0:0:cluster2:ERROR: =
cluster_getRecordIdByKey: no reply bck -1 </FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 17:58:05 g10r9 : 0:0:cluster2:ERROR: =
cluster_getFilerNameList: cannot get cluster rec, rcode 30 =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">May 10 18:00:02 g10r9 newsyslog[12273]: logfile turned =
over</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
8:44 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Sandrine Boulanger; Raj =
Kumar; John Keiffer; Andy Sharp; Chris Vandever<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Subsequent testing showed the =
cluster commands repeatedly working ok from the CLI on both nodes, =
though admittedly this was just one test.&nbsp;&nbsp; John was going to =
test some more via NCM.&nbsp;&nbsp; I think John plans to send a more =
detailed update as well.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
8:34 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest; Raj =
Kumar; John Keiffer; Andy Sharp<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">g10r9 and =
g11r9.onstor.lab</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">I&#8217;ll go through the elog right now to see if I can =
catch anything. It does not look like this would be related to the conf =
file since &#8216;cluster show summary&#8217; uses the cluster db, which =
has been upgraded to version 3.0 on both filers (I already verified =
that).</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
5:55 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar; John Keiffer; =
Andy Sharp; Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">We should start w/ the =
elogs.&nbsp;&nbsp; Can you provide a pointer to those?</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
5:51 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest; John =
Keiffer; Andy Sharp; Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">The workaround tried is based =
on the error reported by &#8220;sys copy&#8221;, which may or may not be =
accurate. We may be missing something (which is also missed by sys =
config copy but not reported as error)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
5:47 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar; John Keiffer; =
Andy Sharp; Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">As I understand, the steps =
performed were exactly the same as what &#8216;system config copy&#8217; =
does.&nbsp;&nbsp; Suspect this could be a different issue.&nbsp;&nbsp; =
</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
5:36 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> John Keiffer; Andy Sharp; =
Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Cc:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Brian DeForest<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">Also NCM having some problem. =
I don&#8217;t like to spend anymore time on this and fixing this =
workaround. I think its time we call this off and fix the original issue =
(system copy config from 2.2 to 3.0)</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Thanks.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> John Keiffer<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
5:03 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar; Andy Sharp; =
Sandrine Boulanger<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: MD upgrade procedure<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Importance:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> High</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">There are cluster =
problems.</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Raj Kumar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Thursday, May 10, 2007 =
4:40 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Andrew LeFebvre; Sandrine =
Boulanger; John Keiffer<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> MD upgrade procedure</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 FACE=3D"Arial">Guys,</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Have we =
nailed the procedure and tested it?</FONT></SPAN></P>

<P DIR=3DLTR><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Thanks</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

</BODY>
</HTML>
------_=_NextPart_001_01C793F7.DF039416--
