AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20080407134040.1cd4c5cc@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<sandrine.boulanger@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@onstor-exch02.onstor.net/INBOX	0	BB375AF679D4A34E9CA8DFA650E2B04E05C7466E@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Mon, 7 Apr 2008 13:44:59 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: "Sandrine Boulanger" <sandrine.boulanger@onstor.com>
Subject: Re: why eventd sends 'Node going down for reboot' 3 times in a row?
Message-ID: <20080407134459.088c0ded@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E05C7466E@onstor-exch02.onstor.net>
References: <20080407132755.1832bcb2@ripper.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05C7466E@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

It's useless in what regard?  So you received the email 30 minutes
after the system started going down, and didn't receive it immediately
after it rebooted, but I don't see what pressing use could be made of
that.  BTW, this works fine for the "usual" case, where DNS can still
be queried:


Received: from mail.onstor.com ([66.201.51.107]) by onstor-exch02.onstor.net with Microsoft SMTPSVC(6.0.3790.1830);
  Mon, 7 Apr 2008 12:02:15 -0700
Received: from coolcat ([66.201.51.106]) by mail.onstor.com with Microsoft SMTPSVC(6.0.3790.1830);
  Mon, 7 Apr 2008 12:02:15 -0700
Received: from root by coolcat with local (Exim 4.63)
 (envelope-from <coolcat@onstor.com>)
 id 1Jiwbe-0000Xs-Kr
 for andy.sharp@onstor.com; Mon, 07 Apr 2008 12:02:14 -0700
To: andy.sharp@onstor.com
From: coolcat@onstor.com
Subject: NFX Node event - 
Message-Id: <E1Jiwbe-0000Xs-Kr@coolcat>
Date: Mon, 07 Apr 2008 12:02:14 -0700
Return-Path: coolcat@onstor.com
X-OriginalArrivalTime: 07 Apr 2008 19:02:15.0756 (UTC) FILETIME=[E506F0C0:01C898E1]

Auto support mail:
	NFX Node   : 
	Event      : Node
	Application: nfxsh
	Message    : Node going down for reboot! ('system reboot -s' issued from nfxsh).
	Time       : Apr 07 12:02:14 2008
-----
-----
THIS MAIL HAS BEEN AUTOMATICALLY GENERATED.
PLEASE DO NOT REPLY.
-----




On Mon, 7 Apr 2008 13:30:15 -0700 "Sandrine Boulanger"
<sandrine.boulanger@onstor.com> wrote:

> Maybe it's wad as to the way it works now, but it's useless to receive
> the message so late, we should fix it
> 
> -----Original Message-----
> From: Andy Sharp 
> Sent: Monday, April 07, 2008 1:28 PM
> To: Rendell Fong
> Cc: Sandrine Boulanger
> Subject: Re: why eventd sends 'Node going down for reboot' 3 times in
> a row?
> 
> 
> On the question of frozen messages:
> 
> Probably have an old version of some files in /etc/logrotate.d
> or /etc/default or something.  They aren't hurting anything.  If the
> amount of space used by /var/spool/exim4 gets over a certain limit,
> the frozen emails will get tossed.
> 
> On the delayed email:
> 
> The system was going down when the email was queued, and apparently
> DNS or something necessary to get DNS from the management vsvr, was no
> longer working, so the DNS failed.  That meant the email had a retry
> time set.  The system booted back up, and when the retry time expired,
> then the email was sent.  Hence the apparent 33 minute delay.  This
> would seem to be WAD.
> 
> Cheers,
> 
> a
> 
> On Mon, 7 Apr 2008 11:12:04 -0700 "Rendell Fong"
> <rendell.fong@onstor.com> wrote:
> 
> > Many internal email messages are getting generated and queued
> > instead of being thrown away such that the mail queue is getting
> > clogged.  It is delaying transmission of external email messages.
> > You'll need Andy's help on this.  Maybe there's a mail
> > configuration issue.
> > 
> > _____________________________________________
> > From: Sandrine Boulanger 
> > Sent: Monday, April 07, 2008 11:07 AM
> > To: Rendell Fong; Andy Sharp; Larry Scheer
> > Subject: RE: why eventd sends 'Node going down for reboot' 3 times
> > in a row?
> > 
> > OK. I also received the email at 10:43 when the event happened at
> > 10:09, here's the end of the exim mailog file, too many frozen
> > messages
> > 
> > 2008-04-07 09:03:27 End queue run: pid=31673
> > 2008-04-07 09:17:03 1Jiu1n-0008GR-1R <= root@g14r10 U=root P=local
> > S=629 2008-04-07 09:17:03 1Jiu1n-0008GR-1R ** deadend@g14r10
> > <root@g14r10>: Unrouteable address
> > 2008-04-07 09:17:03 1Jiu1n-0008GT-9N <= <> R=1Jiu1n-0008GR-1R
> > U=Debian-exim P=local S=1413
> > 2008-04-07 09:17:03 1Jiu1n-0008GT-9N ** deadend@g14r10
> > <root@g14r10>: Unrouteable address
> > 2008-04-07 09:17:03 1Jiu1n-0008GT-9N Frozen (delivery error message)
> > 2008-04-07 09:17:03 1Jiu1n-0008GR-1R Completed
> > 2008-04-07 09:33:27 Start queue run: pid=31863
> > 2008-04-07 09:33:27 1Jihqx-00071x-5v Message is frozen
> > 2008-04-07 09:33:27 1JiRvr-0005Pi-HI Message is frozen
> > 2008-04-07 09:33:27 1JioPP-0007gM-Lw Message is frozen
> > 2008-04-07 09:33:27 1Jikf9-0007J4-KQ Message is frozen
> > 2008-04-07 09:33:27 1Jif2m-0006l1-0v Message is frozen
> > 2008-04-07 09:33:27 1JiNFX-0004wv-J2 Message is frozen
> > 2008-04-07 09:33:27 1JiXYF-000627-LD Message is frozen
> > 2008-04-07 09:33:27 1JiOQ9-00053v-4I Message is frozen
> > 2008-04-07 09:33:27 1JiJVG-0004ab-Tk Message is frozen
> > 2008-04-07 09:33:27 1JiIZD-0004V2-98 Message is frozen
> > 2008-04-07 09:33:27 1JilbD-0007P4-96 Message is frozen
> > 2008-04-07 09:33:27 1JiHd9-0004PU-0J Message is frozen
> > 2008-04-07 09:33:27 1JiFl1-0004ER-9T Message is frozen
> > 2008-04-07 09:33:27 1Jit5j-0008Ap-0G Message is frozen
> > 2008-04-07 09:33:27 1JiTnz-0005as-8v Message is frozen
> > 2008-04-07 09:33:27 1JinTL-0007ai-Bz Message is frozen
> > 2008-04-07 09:33:27 1JiSru-0005VI-SW Message is frozen
> > 2008-04-07 09:33:27 1JiaMR-0006Ir-Rg Message is frozen
> > 2008-04-07 09:33:27 1Jiin1-00077b-Qa Message is frozen
> > 2008-04-07 09:33:27 1JiEox-00048v-T7 Message is frozen
> > 2008-04-07 09:33:27 1JiVg7-0005qs-Ky Message is frozen
> > 2008-04-07 09:33:27 1JiKRL-0004gC-Cs Message is frozen
> > 2008-04-07 09:33:27 1JiQ3j-0005EY-7x Message is frozen
> > 2008-04-07 09:33:27 1JidAd-0006Zj-1K Message is frozen
> > 2008-04-07 09:33:27 1JirDb-0007xJ-8A Message is frozen
> > 2008-04-07 09:33:27 1JiCwq-0003xt-05 Message is frozen
> > 2008-04-07 09:33:27 1JipLT-0007m0-1e Message is frozen
> > 2008-04-07 09:33:27 1JiLNP-0004ll-QE Message is frozen
> > 2008-04-07 09:33:27 1JiZQN-0006DH-FX Message is frozen
> > 2008-04-07 09:33:27 1JiP7f-00058c-Gm Message is frozen
> > 2008-04-07 09:33:27 1Jifyp-0006qf-9n Message is frozen
> > 2008-04-07 09:33:27 1JiMJT-0004rL-7v Message is frozen
> > 2008-04-07 09:33:27 1Jiu1n-0008GT-9N Message is frozen
> > 2008-04-07 09:33:27 1JiGh5-0004Jx-MT Message is frozen
> > 2008-04-07 09:33:27 1JiQzn-0005K8-1z Message is frozen
> > 2008-04-07 09:33:27 1JiUk3-0005gS-O3 Message is frozen
> > 2008-04-07 09:33:27 1JimXH-0007V4-1v Message is frozen
> > 2008-04-07 09:33:27 1JiDst-00043P-Gj Message is frozen
> > 2008-04-07 09:33:27 1JiqHX-0007re-CR Message is frozen
> > 2008-04-07 09:33:27 1JibIV-0006OS-9h Message is frozen
> > 2008-04-07 09:33:27 1Jijj5-0007DF-64 Message is frozen
> > 2008-04-07 09:33:27 1Jie6h-0006fN-E7 Message is frozen
> > 2008-04-07 09:33:27 1JiYUJ-00067h-2d Message is frozen
> > 2008-04-07 09:33:27 1Jigut-0006wJ-PM Message is frozen
> > 2008-04-07 09:33:27 1JicEZ-0006U5-Mi Message is frozen
> > 2008-04-07 09:33:27 1JiWcB-0005wX-BK Message is frozen
> > 2008-04-07 09:33:27 1Jis9f-000856-6E Message is frozen
> > 2008-04-07 09:33:27 1JiB4h-0003mr-Rt Unfrozen by errmsg timer
> > 2008-04-07 09:33:27 1JiB4h-0003mr-Rt ** deadend@g14r10
> > <root@g14r10>: Unrouteable address
> > 2008-04-07 09:33:27 1JiB4h-0003mr-Rt deadend@g14r10 <root@g14r10>:
> > error ignored
> > 2008-04-07 09:33:27 1JiB4h-0003mr-Rt Completed
> > 2008-04-07 09:33:27 1JiC0l-0003sN-Ck Message is frozen
> > 2008-04-07 09:33:27 End queue run: pid=31863
> > 2008-04-07 10:03:27 Start queue run: pid=32068
> > 2008-04-07 10:03:27 1Jihqx-00071x-5v Message is frozen
> > 2008-04-07 10:03:27 1JicEZ-0006U5-Mi Message is frozen
> > 2008-04-07 10:03:27 1Jigut-0006wJ-PM Message is frozen
> > 2008-04-07 10:03:27 1Jif2m-0006l1-0v Message is frozen
> > 2008-04-07 10:03:27 1JiXYF-000627-LD Message is frozen
> > 2008-04-07 10:03:27 1JiUk3-0005gS-O3 Message is frozen
> > 2008-04-07 10:03:27 1JiQzn-0005K8-1z Message is frozen
> > 2008-04-07 10:03:27 1JilbD-0007P4-96 Message is frozen
> > 2008-04-07 10:03:27 1JiFl1-0004ER-9T Message is frozen
> > 2008-04-07 10:03:27 1JipLT-0007m0-1e Message is frozen
> > 2008-04-07 10:03:27 1JiCwq-0003xt-05 Message is frozen
> > 2008-04-07 10:03:27 1JiSru-0005VI-SW Message is frozen
> > 2008-04-07 10:03:27 1Jiin1-00077b-Qa Message is frozen
> > 2008-04-07 10:03:27 1JiEox-00048v-T7 Message is frozen
> > 2008-04-07 10:03:27 1JiVg7-0005qs-Ky Message is frozen
> > 2008-04-07 10:03:27 1JiKRL-0004gC-Cs Message is frozen
> > 2008-04-07 10:03:27 1JiaMR-0006Ir-Rg Message is frozen
> > 2008-04-07 10:03:27 1JiQ3j-0005EY-7x Message is frozen
> > 2008-04-07 10:03:27 1JidAd-0006Zj-1K Message is frozen
> > 2008-04-07 10:03:27 1JirDb-0007xJ-8A Message is frozen
> > 2008-04-07 10:03:27 1JinTL-0007ai-Bz Message is frozen
> > 2008-04-07 10:03:27 1JiTnz-0005as-8v Message is frozen
> > 2008-04-07 10:03:27 1Jit5j-0008Ap-0G Message is frozen
> > 2008-04-07 10:03:27 1JiLNP-0004ll-QE Message is frozen
> > 2008-04-07 10:03:27 1JiZQN-0006DH-FX Message is frozen
> > 2008-04-07 10:03:27 1JiP7f-00058c-Gm Message is frozen
> > 2008-04-07 10:03:27 1Jifyp-0006qf-9n Message is frozen
> > 2008-04-07 10:03:27 1JiHd9-0004PU-0J Message is frozen
> > 2008-04-07 10:03:27 1JiMJT-0004rL-7v Message is frozen
> > 2008-04-07 10:03:27 1Jiu1n-0008GT-9N Message is frozen
> > 2008-04-07 10:03:27 1JiGh5-0004Jx-MT Message is frozen
> > 2008-04-07 10:03:27 1JiIZD-0004V2-98 Message is frozen
> > 2008-04-07 10:03:27 1JiJVG-0004ab-Tk Message is frozen
> > 2008-04-07 10:03:27 1JiOQ9-00053v-4I Message is frozen
> > 2008-04-07 10:03:27 1JimXH-0007V4-1v Message is frozen
> > 2008-04-07 10:03:27 1JiDst-00043P-Gj Message is frozen
> > 2008-04-07 10:03:27 1JiqHX-0007re-CR Message is frozen
> > 2008-04-07 10:03:27 1JibIV-0006OS-9h Message is frozen
> > 2008-04-07 10:03:27 1JiNFX-0004wv-J2 Message is frozen
> > 2008-04-07 10:03:27 1Jijj5-0007DF-64 Message is frozen
> > 2008-04-07 10:03:27 1Jie6h-0006fN-E7 Message is frozen
> > 2008-04-07 10:03:27 1JiYUJ-00067h-2d Message is frozen
> > 2008-04-07 10:03:27 1Jikf9-0007J4-KQ Message is frozen
> > 2008-04-07 10:03:27 1JioPP-0007gM-Lw Message is frozen
> > 2008-04-07 10:03:27 1JiRvr-0005Pi-HI Message is frozen
> > 2008-04-07 10:03:27 1JiWcB-0005wX-BK Message is frozen
> > 2008-04-07 10:03:27 1Jis9f-000856-6E Message is frozen
> > 2008-04-07 10:03:27 1JiC0l-0003sN-Ck Message is frozen
> > 2008-04-07 10:03:27 End queue run: pid=32068
> > 2008-04-07 10:09:25 1JiuqS-0008MN-Nc <= g14r10@onstor.com U=root
> > P=local S=616
> > 2008-04-07 10:09:28 1JiuqV-0008MR-Tk <= g14r10@onstor.com U=root
> > P=local S=616
> > 2008-04-07 10:09:32 1JiuqS-0008MN-Nc ==
> > sandrine.boulanger@onstor.com R=dnslookup defer (-1): host lookup
> > did not complete 2008-04-07 10:09:33 1Jiuqa-0008OO-Ta <=
> > g14r10@onstor.com U=root P=local S=616
> > 2008-04-07 10:11:48 exim 4.63 daemon started: pid=804, -q30m,
> > listening for SMTP on [127.0.0.1]:25
> > 2008-04-07 10:11:48 Start queue run: pid=809
> > 2008-04-07 10:11:48 1Jis9f-000856-6E Message is frozen
> > 2008-04-07 10:11:48 1Jihqx-00071x-5v Message is frozen
> > 2008-04-07 10:11:48 1JiRvr-0005Pi-HI Message is frozen
> > 2008-04-07 10:11:48 1Jigut-0006wJ-PM Message is frozen
> > 2008-04-07 10:11:48 1Jikf9-0007J4-KQ Message is frozen
> > 2008-04-07 10:11:49 1Jie6h-0006fN-E7 Message is frozen
> > 2008-04-07 10:11:49 1JiNFX-0004wv-J2 Message is frozen
> > 2008-04-07 10:11:50 1JiqHX-0007re-CR Message is frozen
> > 2008-04-07 10:11:50 1JimXH-0007V4-1v Message is frozen
> > 2008-04-07 10:11:51 1Jiuqa-0008OO-Ta ==
> > sandrine.boulanger@onstor.com routing defer (-51): retry time not
> > reached 2008-04-07 10:11:51 1JiOQ9-00053v-4I Message is frozen
> > 2008-04-07 10:11:51 1JiQzn-0005K8-1z Message is frozen
> > 2008-04-07 10:11:51 1JiIZD-0004V2-98 Message is frozen
> > 2008-04-07 10:11:51 1JiMJT-0004rL-7v Message is frozen
> > 2008-04-07 10:11:51 1Jifyp-0006qf-9n Message is frozen
> > 2008-04-07 10:11:51 1JiuqV-0008MR-Tk ==
> > sandrine.boulanger@onstor.com routing defer (-51): retry time not
> > reached 2008-04-07 10:11:52 1JiZQN-0006DH-FX Message is frozen
> > 2008-04-07 10:11:52 1JiFl1-0004ER-9T Message is frozen
> > 2008-04-07 10:11:52 1JipLT-0007m0-1e Message is frozen
> > 2008-04-07 10:11:52 1JiTnz-0005as-8v Message is frozen
> > 2008-04-07 10:11:52 1JiSru-0005VI-SW Message is frozen
> > 2008-04-07 10:11:52 1JiaMR-0006Ir-Rg Message is frozen
> > 2008-04-07 10:11:52 1JiVg7-0005qs-Ky Message is frozen
> > 2008-04-07 10:11:52 1JiEox-00048v-T7 Message is frozen
> > 2008-04-07 10:11:52 1Jiin1-00077b-Qa Message is frozen
> > 2008-04-07 10:11:52 1JiKRL-0004gC-Cs Message is frozen
> > 2008-04-07 10:11:52 1JiQ3j-0005EY-7x Message is frozen
> > 2008-04-07 10:11:52 1JidAd-0006Zj-1K Message is frozen
> > 2008-04-07 10:11:52 1JirDb-0007xJ-8A Message is frozen
> > 2008-04-07 10:11:52 1JinTL-0007ai-Bz Message is frozen
> > 2008-04-07 10:11:52 1JiCwq-0003xt-05 Message is frozen
> > 2008-04-07 10:11:52 1Jit5j-0008Ap-0G Message is frozen
> > 2008-04-07 10:11:52 1JiLNP-0004ll-QE Message is frozen
> > 2008-04-07 10:11:52 1JiP7f-00058c-Gm Message is frozen
> > 2008-04-07 10:11:52 1JiHd9-0004PU-0J Message is frozen
> > 2008-04-07 10:11:52 1JilbD-0007P4-96 Message is frozen
> > 2008-04-07 10:11:52 1Jiu1n-0008GT-9N Message is frozen
> > 2008-04-07 10:11:52 1JiGh5-0004Jx-MT Message is frozen
> > 2008-04-07 10:11:52 1JiJVG-0004ab-Tk Message is frozen
> > 2008-04-07 10:11:52 1JiUk3-0005gS-O3 Message is frozen
> > 2008-04-07 10:11:52 1JiXYF-000627-LD Message is frozen
> > 2008-04-07 10:11:52 1JiDst-00043P-Gj Message is frozen
> > 2008-04-07 10:11:52 1JibIV-0006OS-9h Message is frozen
> > 2008-04-07 10:11:52 1JiuqS-0008MN-Nc ==
> > sandrine.boulanger@onstor.com routing defer (-51): retry time not
> > reached 2008-04-07 10:11:52 1Jijj5-0007DF-64 Message is frozen
> > 2008-04-07 10:11:52 1Jif2m-0006l1-0v Message is frozen
> > 2008-04-07 10:11:52 1JiYUJ-00067h-2d Message is frozen
> > 2008-04-07 10:11:52 1JioPP-0007gM-Lw Message is frozen
> > 2008-04-07 10:11:52 1JicEZ-0006U5-Mi Message is frozen
> > 2008-04-07 10:11:52 1JiWcB-0005wX-BK Message is frozen
> > 2008-04-07 10:11:52 1JiC0l-0003sN-Ck Message is frozen
> > 2008-04-07 10:11:52 End queue run: pid=809
> > 2008-04-07 10:12:47 exim 4.63 daemon started: pid=1489, -q30m,
> > listening for SMTP on [127.0.0.1]:25
> > 2008-04-07 10:12:47 Start queue run: pid=1490
> > 2008-04-07 10:12:47 1JiC0l-0003sN-Ck Message is frozen
> > 2008-04-07 10:12:47 1Jigut-0006wJ-PM Message is frozen
> > 2008-04-07 10:12:47 1Jikf9-0007J4-KQ Message is frozen
> > 2008-04-07 10:12:47 1Jif2m-0006l1-0v Message is frozen
> > 2008-04-07 10:12:47 1Jijj5-0007DF-64 Message is frozen
> > 2008-04-07 10:12:47 1JiuqS-0008MN-Nc ==
> > sandrine.boulanger@onstor.com routing defer (-51): retry time not
> > reached 2008-04-07 10:12:47 1JiNFX-0004wv-J2 Message is frozen
> > 2008-04-07 10:12:47 1JibIV-0006OS-9h Message is frozen
> > 2008-04-07 10:12:47 1JiOQ9-00053v-4I Message is frozen
> > 2008-04-07 10:12:47 1JiQzn-0005K8-1z Message is frozen
> > 2008-04-07 10:12:47 1JiGh5-0004Jx-MT Message is frozen
> > 2008-04-07 10:12:47 1Jiu1n-0008GT-9N Message is frozen
> > 2008-04-07 10:12:47 1JilbD-0007P4-96 Message is frozen
> > 2008-04-07 10:12:47 1JiMJT-0004rL-7v Message is frozen
> > 2008-04-07 10:12:47 1JiHd9-0004PU-0J Message is frozen
> > 2008-04-07 10:12:47 1JiFl1-0004ER-9T Message is frozen
> > 2008-04-07 10:12:47 1JipLT-0007m0-1e Message is frozen
> > 2008-04-07 10:12:48 1JinTL-0007ai-Bz Message is frozen
> > 2008-04-07 10:12:48 1JirDb-0007xJ-8A Message is frozen
> > 2008-04-07 10:12:48 1JidAd-0006Zj-1K Message is frozen
> > 2008-04-07 10:12:48 1JiSru-0005VI-SW Message is frozen
> > 2008-04-07 10:12:48 1JiQ3j-0005EY-7x Message is frozen
> > 2008-04-07 10:12:48 1Jiin1-00077b-Qa Message is frozen
> > 2008-04-07 10:12:48 1JiEox-00048v-T7 Message is frozen
> > 2008-04-07 10:12:48 1JiVg7-0005qs-Ky Message is frozen
> > 2008-04-07 10:12:48 1JiKRL-0004gC-Cs Message is frozen
> > 2008-04-07 10:12:48 1JiaMR-0006Ir-Rg Message is frozen
> > 2008-04-07 10:12:48 1JiTnz-0005as-8v Message is frozen
> > 2008-04-07 10:12:48 1JiCwq-0003xt-05 Message is frozen
> > 2008-04-07 10:12:48 1Jit5j-0008Ap-0G Message is frozen
> > 2008-04-07 10:12:48 1JiLNP-0004ll-QE Message is frozen
> > 2008-04-07 10:12:48 1JiZQN-0006DH-FX Message is frozen
> > 2008-04-07 10:12:48 1JiuqV-0008MR-Tk ==
> > sandrine.boulanger@onstor.com routing defer (-51): retry time not
> > reached 2008-04-07 10:12:48 1JiP7f-00058c-Gm Message is frozen
> > 2008-04-07 10:12:48 1Jifyp-0006qf-9n Message is frozen
> > 2008-04-07 10:12:48 1JiIZD-0004V2-98 Message is frozen
> > 2008-04-07 10:12:48 1JiJVG-0004ab-Tk Message is frozen
> > 2008-04-07 10:12:48 1JiUk3-0005gS-O3 Message is frozen
> > 2008-04-07 10:12:48 1JiXYF-000627-LD Message is frozen
> > 2008-04-07 10:12:48 1Jiuqa-0008OO-Ta ==
> > sandrine.boulanger@onstor.com routing defer (-51): retry time not
> > reached 2008-04-07 10:12:48 1JimXH-0007V4-1v Message is frozen
> > 2008-04-07 10:12:48 1JiDst-00043P-Gj Message is frozen
> > 2008-04-07 10:12:48 1JiqHX-0007re-CR Message is frozen
> > 2008-04-07 10:12:48 1Jie6h-0006fN-E7 Message is frozen
> > 2008-04-07 10:12:48 1JiYUJ-00067h-2d Message is frozen
> > 2008-04-07 10:12:48 1JioPP-0007gM-Lw Message is frozen
> > 2008-04-07 10:12:48 1JicEZ-0006U5-Mi Message is frozen
> > 2008-04-07 10:12:48 1JiRvr-0005Pi-HI Message is frozen
> > 2008-04-07 10:12:48 1Jihqx-00071x-5v Message is frozen
> > 2008-04-07 10:12:48 1JiWcB-0005wX-BK Message is frozen
> > 2008-04-07 10:12:48 1Jis9f-000856-6E Message is frozen
> > 2008-04-07 10:12:48 End queue run: pid=1490
> > 2008-04-07 10:13:16 1JiuuB-0000PP-NT <= g14r10@onstor.com U=root
> > P=local S=576
> > 2008-04-07 10:13:16 1JiuuB-0000PP-NT =>
> > sandrine.boulanger@onstor.com R=dnslookup T=remote_smtp
> > H=mail.onstor.com [66.201.51.107] 2008-04-07 10:13:16
> > 1JiuuB-0000PP-NT Completed 2008-04-07 10:17:03 1Jiuxr-0000lQ-Jr <=
> > root@g14r10 U=root P=local S=629 2008-04-07 10:17:03
> > 1Jiuxr-0000lQ-Jr ** deadend@g14r10 <root@g14r10>: Unrouteable
> > address 2008-04-07 10:17:04 1Jiuxr-0000lS-Ul <= <>
> > R=1Jiuxr-0000lQ-Jr U=Debian-exim P=local S=1413
> > 2008-04-07 10:17:04 1Jiuxr-0000lS-Ul ** deadend@g14r10
> > <root@g14r10>: Unrouteable address
> > 2008-04-07 10:17:04 1Jiuxr-0000lS-Ul Frozen (delivery error message)
> > 2008-04-07 10:17:04 1Jiuxr-0000lQ-Jr Completed
> > 2008-04-07 10:34:49 1JivF2-0000mv-Rr <= g14r10@onstor.com U=root
> > P=local S=611
> > 2008-04-07 10:34:49 1JivF2-0000mv-Rr =>
> > sandrine.boulanger@onstor.com R=dnslookup T=remote_smtp
> > H=mail.onstor.com [66.201.51.107] 2008-04-07 10:34:49
> > 1JivF2-0000mv-Rr Completed 2008-04-07 10:42:47 Start queue run:
> > pid=3341 2008-04-07 10:42:47 1JiC0l-0003sN-Ck Unfrozen by errmsg
> > timer 2008-04-07 10:42:47 1JiC0l-0003sN-Ck ** deadend@g14r10
> > <root@g14r10>: Unrouteable address
> > 2008-04-07 10:42:47 1JiC0l-0003sN-Ck deadend@g14r10 <root@g14r10>:
> > error ignored
> > 2008-04-07 10:42:47 1JiC0l-0003sN-Ck Completed
> > 2008-04-07 10:42:47 1JiWcB-0005wX-BK Message is frozen
> > 2008-04-07 10:42:47 1JioPP-0007gM-Lw Message is frozen
> > 2008-04-07 10:42:47 1Jif2m-0006l1-0v Message is frozen
> > 2008-04-07 10:42:48 1JiuqS-0008MN-Nc =>
> > sandrine.boulanger@onstor.com R=dnslookup T=remote_smtp
> > H=mail.onstor.com [66.201.51.107] 2008-04-07 10:42:48
> > 1JiuqS-0008MN-Nc Completed 2008-04-07 10:42:48 1JiNFX-0004wv-J2
> > Message is frozen 2008-04-07 10:42:48 1JibIV-0006OS-9h Message is
> > frozen 2008-04-07 10:42:48 1JiDst-00043P-Gj Message is frozen
> > 2008-04-07 10:42:48 1JiUk3-0005gS-O3 Message is frozen
> > 2008-04-07 10:42:48 1JiGh5-0004Jx-MT Message is frozen
> > 2008-04-07 10:42:48 1JilbD-0007P4-96 Message is frozen
> > 2008-04-07 10:42:48 1JiMJT-0004rL-7v Message is frozen
> > 2008-04-07 10:42:48 1JiHd9-0004PU-0J Message is frozen
> > 2008-04-07 10:42:48 1JiP7f-00058c-Gm Message is frozen
> > 2008-04-07 10:42:48 1Jiuxr-0000lS-Ul Message is frozen
> > 2008-04-07 10:42:48 1JiTnz-0005as-8v Message is frozen
> > 2008-04-07 10:42:48 1JirDb-0007xJ-8A Message is frozen
> > 2008-04-07 10:42:48 1JidAd-0006Zj-1K Message is frozen
> > 2008-04-07 10:42:48 1JiSru-0005VI-SW Message is frozen
> > 2008-04-07 10:42:48 1JiaMR-0006Ir-Rg Message is frozen
> > 2008-04-07 10:42:48 1Jiin1-00077b-Qa Message is frozen
> > 2008-04-07 10:42:48 1JiEox-00048v-T7 Message is frozen
> > 2008-04-07 10:42:48 1JiVg7-0005qs-Ky Message is frozen
> > 2008-04-07 10:42:48 1JiKRL-0004gC-Cs Message is frozen
> > 2008-04-07 10:42:48 1JiQ3j-0005EY-7x Message is frozen
> > 2008-04-07 10:42:48 1JinTL-0007ai-Bz Message is frozen
> > 2008-04-07 10:42:48 1JiCwq-0003xt-05 Message is frozen
> > 2008-04-07 10:42:48 1JipLT-0007m0-1e Message is frozen
> > 2008-04-07 10:42:48 1Jit5j-0008Ap-0G Message is frozen
> > 2008-04-07 10:42:48 1JiFl1-0004ER-9T Message is frozen
> > 2008-04-07 10:42:48 1JiLNP-0004ll-QE Message is frozen
> > 2008-04-07 10:42:48 1JiZQN-0006DH-FX Message is frozen
> > 2008-04-07 10:42:48 1JiuqV-0008MR-Tk =>
> > sandrine.boulanger@onstor.com R=dnslookup T=remote_smtp
> > H=mail.onstor.com [66.201.51.107] 2008-04-07 10:42:48
> > 1JiuqV-0008MR-Tk Completed 2008-04-07 10:42:48 1Jifyp-0006qf-9n
> > Message is frozen 2008-04-07 10:42:48 1Jiu1n-0008GT-9N Message is
> > frozen 2008-04-07 10:42:48 1JiIZD-0004V2-98 Message is frozen
> > 2008-04-07 10:42:48 1JiJVG-0004ab-Tk Message is frozen
> > 2008-04-07 10:42:48 1JiQzn-0005K8-1z Message is frozen
> > 2008-04-07 10:42:48 1JiOQ9-00053v-4I Message is frozen
> > 2008-04-07 10:42:48 1JiXYF-000627-LD Message is frozen
> > 2008-04-07 10:42:48 1Jiuqa-0008OO-Ta =>
> > sandrine.boulanger@onstor.com R=dnslookup T=remote_smtp
> > H=mail.onstor.com [66.201.51.107] 2008-04-07 10:42:48
> > 1Jiuqa-0008OO-Ta Completed 2008-04-07 10:42:48 1JimXH-0007V4-1v
> > Message is frozen 2008-04-07 10:42:48 1JiqHX-0007re-CR Message is
> > frozen 2008-04-07 10:42:48 1Jijj5-0007DF-64 Message is frozen
> > 2008-04-07 10:42:48 1Jie6h-0006fN-E7 Message is frozen
> > 2008-04-07 10:42:49 1JiYUJ-00067h-2d Message is frozen
> > 2008-04-07 10:42:49 1Jikf9-0007J4-KQ Message is frozen
> > 2008-04-07 10:42:49 1Jigut-0006wJ-PM Message is frozen
> > 2008-04-07 10:42:49 1JicEZ-0006U5-Mi Message is frozen
> > 2008-04-07 10:42:49 1JiRvr-0005Pi-HI Message is frozen
> > 2008-04-07 10:42:49 1Jihqx-00071x-5v Message is frozen
> > 2008-04-07 10:42:49 1Jis9f-000856-6E Message is frozen
> > 2008-04-07 10:42:49 End queue run: pid=3341
> > 2008-04-07 10:57:10 exim 4.63 daemon started: pid=795, -q30m,
> > listening for SMTP on [127.0.0.1]:25
> > 2008-04-07 10:57:10 Start queue run: pid=800
> > 2008-04-07 10:57:10 1JiWcB-0005wX-BK Message is frozen
> > 2008-04-07 10:57:10 1JicEZ-0006U5-Mi Message is frozen
> > 2008-04-07 10:57:10 1Jigut-0006wJ-PM Message is frozen
> > 2008-04-07 10:57:10 1JioPP-0007gM-Lw Message is frozen
> > 2008-04-07 10:57:10 1Jikf9-0007J4-KQ Message is frozen
> > 2008-04-07 10:57:10 1JiYUJ-00067h-2d Message is frozen
> > 2008-04-07 10:57:10 1Jie6h-0006fN-E7 Message is frozen
> > 2008-04-07 10:57:12 1JibIV-0006OS-9h Message is frozen
> > 2008-04-07 10:57:12 1JimXH-0007V4-1v Message is frozen
> > 2008-04-07 10:57:12 1JiUk3-0005gS-O3 Message is frozen
> > 2008-04-07 10:57:12 1JiQzn-0005K8-1z Message is frozen
> > 2008-04-07 10:57:12 1JiJVG-0004ab-Tk Message is frozen
> > 2008-04-07 10:57:12 1JiIZD-0004V2-98 Message is frozen
> > 2008-04-07 10:57:12 1JiGh5-0004Jx-MT Message is frozen
> > 2008-04-07 10:57:12 1Jiu1n-0008GT-9N Message is frozen
> > 2008-04-07 10:57:13 1Jifyp-0006qf-9n Message is frozen
> > 2008-04-07 10:57:13 1JiLNP-0004ll-QE Message is frozen
> > 2008-04-07 10:57:13 1Jit5j-0008Ap-0G Message is frozen
> > 2008-04-07 10:57:13 1JipLT-0007m0-1e Message is frozen
> > 2008-04-07 10:57:13 1JiCwq-0003xt-05 Message is frozen
> > 2008-04-07 10:57:13 1JiTnz-0005as-8v Message is frozen
> > 2008-04-07 10:57:13 1JinTL-0007ai-Bz Message is frozen
> > 2008-04-07 10:57:13 1JirDb-0007xJ-8A Message is frozen
> > 2008-04-07 10:57:13 1JiaMR-0006Ir-Rg Message is frozen
> > 2008-04-07 10:57:13 1JiEox-00048v-T7 Message is frozen
> > 2008-04-07 10:57:13 1Jiin1-00077b-Qa Message is frozen
> > 2008-04-07 10:57:13 1JiVg7-0005qs-Ky Message is frozen
> > 2008-04-07 10:57:13 1JiKRL-0004gC-Cs Message is frozen
> > 2008-04-07 10:57:13 1JiQ3j-0005EY-7x Message is frozen
> > 2008-04-07 10:57:13 1JiSru-0005VI-SW Message is frozen
> > 2008-04-07 10:57:13 1JidAd-0006Zj-1K Message is frozen
> > 2008-04-07 10:57:13 1Jiuxr-0000lS-Ul Message is frozen
> > 2008-04-07 10:57:13 1JiFl1-0004ER-9T Message is frozen
> > 2008-04-07 10:57:13 1JiZQN-0006DH-FX Message is frozen
> > 2008-04-07 10:57:13 1JiP7f-00058c-Gm Message is frozen
> > 2008-04-07 10:57:13 1JiHd9-0004PU-0J Message is frozen
> > 2008-04-07 10:57:13 1JiMJT-0004rL-7v Message is frozen
> > 2008-04-07 10:57:13 1JilbD-0007P4-96 Message is frozen
> > 2008-04-07 10:57:13 1JiOQ9-00053v-4I Message is frozen
> > 2008-04-07 10:57:13 1JiXYF-000627-LD Message is frozen
> > 2008-04-07 10:57:13 1JiDst-00043P-Gj Message is frozen
> > 2008-04-07 10:57:13 1JiqHX-0007re-CR Message is frozen
> > 2008-04-07 10:57:13 1JiNFX-0004wv-J2 Message is frozen
> > 2008-04-07 10:57:13 1Jijj5-0007DF-64 Message is frozen
> > 2008-04-07 10:57:13 1Jif2m-0006l1-0v Message is frozen
> > 2008-04-07 10:57:13 1JiRvr-0005Pi-HI Message is frozen
> > 2008-04-07 10:57:13 1Jihqx-00071x-5v Message is frozen
> > 2008-04-07 10:57:13 1Jis9f-000856-6E Message is frozen
> > 2008-04-07 10:57:13 End queue run: pid=800
> > 2008-04-07 10:57:38 exim 4.63 daemon started: pid=1233, -q30m,
> > listening for SMTP on [127.0.0.1]:25
> > 2008-04-07 10:57:38 Start queue run: pid=1234
> > 2008-04-07 10:57:38 1JiWcB-0005wX-BK Message is frozen
> > 2008-04-07 10:57:38 1JicEZ-0006U5-Mi Message is frozen
> > 2008-04-07 10:57:38 1Jigut-0006wJ-PM Message is frozen
> > 2008-04-07 10:57:38 1JioPP-0007gM-Lw Message is frozen
> > 2008-04-07 10:57:38 1Jikf9-0007J4-KQ Message is frozen
> > 2008-04-07 10:57:38 1JiYUJ-00067h-2d Message is frozen
> > 2008-04-07 10:57:38 1Jif2m-0006l1-0v Message is frozen
> > 2008-04-07 10:57:38 1Jijj5-0007DF-64 Message is frozen
> > 2008-04-07 10:57:38 1JibIV-0006OS-9h Message is frozen
> > 2008-04-07 10:57:38 1JiqHX-0007re-CR Message is frozen
> > 2008-04-07 10:57:38 1JimXH-0007V4-1v Message is frozen
> > 2008-04-07 10:57:38 1JiUk3-0005gS-O3 Message is frozen
> > 2008-04-07 10:57:38 1JiQzn-0005K8-1z Message is frozen
> > 2008-04-07 10:57:38 1JiJVG-0004ab-Tk Message is frozen
> > 2008-04-07 10:57:38 1JiIZD-0004V2-98 Message is frozen
> > 2008-04-07 10:57:38 1JiGh5-0004Jx-MT Message is frozen
> > 2008-04-07 10:57:38 1JilbD-0007P4-96 Message is frozen
> > 2008-04-07 10:57:38 1JiMJT-0004rL-7v Message is frozen
> > 2008-04-07 10:57:38 1Jifyp-0006qf-9n Message is frozen
> > 2008-04-07 10:57:38 1JiP7f-00058c-Gm Message is frozen
> > 2008-04-07 10:57:38 1JiLNP-0004ll-QE Message is frozen
> > 2008-04-07 10:57:38 1Jit5j-0008Ap-0G Message is frozen
> > 2008-04-07 10:57:38 1JipLT-0007m0-1e Message is frozen
> > 2008-04-07 10:57:38 1JiCwq-0003xt-05 Message is frozen
> > 2008-04-07 10:57:38 1JiTnz-0005as-8v Message is frozen
> > 2008-04-07 10:57:38 1JinTL-0007ai-Bz Message is frozen
> > 2008-04-07 10:57:38 1JidAd-0006Zj-1K Message is frozen
> > 2008-04-07 10:57:38 1JiSru-0005VI-SW Message is frozen
> > 2008-04-07 10:57:38 1JiaMR-0006Ir-Rg Message is frozen
> > 2008-04-07 10:57:38 1JiKRL-0004gC-Cs Message is frozen
> > 2008-04-07 10:57:38 1JiEox-00048v-T7 Message is frozen
> > 2008-04-07 10:57:38 1Jiin1-00077b-Qa Message is frozen
> > 2008-04-07 10:57:38 1JiVg7-0005qs-Ky Message is frozen
> > 2008-04-07 10:57:38 1JiQ3j-0005EY-7x Message is frozen
> > 2008-04-07 10:57:38 1JirDb-0007xJ-8A Message is frozen
> > 2008-04-07 10:57:38 1Jiuxr-0000lS-Ul Message is frozen
> > 2008-04-07 10:57:38 1JiFl1-0004ER-9T Message is frozen
> > 2008-04-07 10:57:38 1JiZQN-0006DH-FX Message is frozen
> > 2008-04-07 10:57:38 1JiHd9-0004PU-0J Message is frozen
> > 2008-04-07 10:57:38 1Jiu1n-0008GT-9N Message is frozen
> > 2008-04-07 10:57:38 1JiOQ9-00053v-4I Message is frozen
> > 2008-04-07 10:57:38 1JiXYF-000627-LD Message is frozen
> > 2008-04-07 10:57:38 1JiDst-00043P-Gj Message is frozen
> > 2008-04-07 10:57:38 1JiNFX-0004wv-J2 Message is frozen
> > 2008-04-07 10:57:38 1Jie6h-0006fN-E7 Message is frozen
> > 2008-04-07 10:57:38 1JiRvr-0005Pi-HI Message is frozen
> > 2008-04-07 10:57:38 1Jihqx-00071x-5v Message is frozen
> > 2008-04-07 10:57:38 1Jis9f-000856-6E Message is frozen
> > 2008-04-07 10:57:38 End queue run: pid=1234
> > g14r10:~#
> > 
> > _____________________________________________
> > From: Rendell Fong 
> > Sent: Monday, April 07, 2008 10:58 AM
> > To: Sandrine Boulanger; Andy Sharp; Larry Scheer
> > Subject: RE: why eventd sends 'Node going down for reboot' 3 times
> > in a row?
> > 
> > The chassis daemon is checking every TXRX/FP cpu and sending the
> > node event multiple times since multiple cpus are down.  I suppose
> > once is enough. 
> > 
> > _____________________________________________
> > From: Sandrine Boulanger 
> > Sent: Monday, April 07, 2008 10:51 AM
> > To: Andy Sharp; Larry Scheer; Rendell Fong
> > Subject: why eventd sends 'Node going down for reboot' 3 times in a
> > row?
> > 
> > I don't know yet if vol create triggered the crash, but I don't
> > understand why eventd sends 3 times that the node is going down for
> > reboot (I got 3 autosupport emails too)
> > 
> > Apr  7 09:45:26 g14r10 : 0:0:eventd:NOTICE: Process-EVENT Volume:
> > name 'vol_5', Id 0x000107540000012f, Event 'Create', State 'Down'
> > Apr  7 09:45:30 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> > 1, CPU 0, State Down
> > Apr  7 09:45:30 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> > 1, CPU 1, State Down
> > Apr  7 09:45:30 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> > NFX_EVENT_CPU, state 2, slot 1, cpu 0
> > Apr  7 09:45:30 g14r10 : 0:0:vtm:INFO: vtm_sendCardStateMsg: Sending
> > card DOWN to g14r10
> > Apr  7 09:45:31 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> > NFX_EVENT_CPU, state 2, slot 1, cpu 1
> > Apr  7 09:45:31 g14r10 : 0:0:vtm:INFO: Vtm_ProcCardStateMsg: card
> > state DOWN message from g14r10
> > Apr  7 09:45:31 g14r10 : 0:0:auth_agent:INFO: uninitCifsd:
> > Uninitializing CIFS daemon
> > Apr  7 09:45:31 g14r10 : 0:0:auth_agent:INFO: killCifsd[213]:
> > Killing CIFS daemon (pid=5204) for VS=2[G14R10-1] intentionally.
> > alreadyRebooting=0, force=0
> > Apr  7 09:45:31 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> > 1, CPU 2, State Down
> > Apr  7 09:45:32 g14r10 : 0:0:evm:ERROR: evm_sendCreateLvReq :
> > Timeout creating run time instance of volume vol_5
> > Apr  7 09:45:32 g14r10 : 0:0:evm:ERROR: evm_onlineEvmSrvr : Cannot
> > create LV instance on FP for volume[vol_5].
> > Apr  7 09:45:32 g14r10 : 0:0:evm:ERROR: evm_putVolOnline: Cannot
> > create run time instance of volume[vol_5]
> > Apr  7 09:45:32 g14r10 : 0:0:ea:ERROR: ea_evmCreateVol[145]:
> > Internal EVM error. Operation failed on volume[vol_5].
> > Apr  7 09:45:32 g14r10 : 0:0:sanm:ERROR: SANM: FP NIM down. Aborting
> > all mirror sessions.
> > Apr  7 09:45:32 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> > NFX_EVENT_CPU, state 2, slot 1, cpu 2
> > Apr  7 09:45:32 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> > 1, CPU 3, State Down
> > Apr  7 09:45:32 g14r10 : 0:0:sanm:ERROR: SANM: FP NIM down. Aborting
> > all mirror sessions.
> > Apr  7 09:45:32 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> > NFX_EVENT_CPU, state 2, slot 1, cpu 3
> > Apr  7 09:45:32 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> > 1, CPU 4, State Down
> > Apr  7 09:45:32 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> > NFX_EVENT_CPU, state 2, slot 1, cpu 4
> > Apr  7 09:45:32 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> > 1, CPU 5, State Down
> > Apr  7 09:45:32 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> > NFX_EVENT_CPU, state 2, slot 1, cpu 5
> > Apr  7 09:45:32 g14r10 : 0:0:eventd:NOTICE: Process-EVENT IP i/f: IP
> > 192.167.1.1, Port bp0, State Remove
> > Apr  7 09:45:32 g14r10 : 0:0:eventd:NOTICE: Process-EVENT Volume:
> > name 'vol_mgmt_67412', Id 0x000107540000006d, Event 'Offline',
> > State 'Down' Apr  7 09:45:32 g14r10 : 0:0:eventd:NOTICE:
> > Process-EVENT Volume: name 'sand', Id 0x0000075400000068, Event
> > 'Offline', State 'Down' Apr  7 09:45:32 g14r10 :
> > 0:0:eventd:WARNING: Process-EVENT Vsvr: Virtual server
> > 'VS_MGMT_67412', Id 1, State 'Down' Apr  7 09:45:33 g14r10 :
> > 0:0:eventd:NOTICE: Process-EVENT Volume: name 'sand2-m', Id
> > 0x0001075400000099, Event 'Offline', State 'Down' Apr  7 09:45:33
> > g14r10 : 0:0:eventd:NOTICE: Process-EVENT Volume: name 'sand1', Id
> > 0x000107540000006e, Event 'Offline', State 'Down' Apr  7 09:45:33
> > g14r10 : 0:0:eventd:NOTICE: Process-EVENT IP i/f: IP 192.167.2.1,
> > Port bp0, State Remove Apr  7 09:45:33 g14r10 : 0:0:ea:INFO: Error
> > string (Internal EVM error. Operation failed on volume[vol_5].) len
> > (54) Apr  7 09:45:33 g14r10 : 0:0:eventd:WARNING: Process-EVENT
> > Vsvr: Virtual server 'G14R10-1', Id 2, State 'Down'
> > Apr  7 09:45:33 g14r10 : 0:0:nfxsh:NOTICE: cmd[16]: vol create vol_5
> > 3PARdata_0565_000900565 : status[11]
> > Apr  7 09:45:34 g14r10 : 0:0:vtm:ERROR: vtm_gatherInfo_proc: gather
> > info failed for request from g14r10
> > Apr  7 09:45:35 g14r10 : 0:0:vtm:ERROR: vtm_failover_vsvr_proc: no
> > alternative filer found for vs 2 failover
> > Apr  7 09:45:35 g14r10 : 0:0:vtm:ERROR: vtm_failover_vsvr_proc: no
> > alternative filer found for vs 3 failover
> > Apr  7 09:46:02 g14r10 : 0:0:eventd:NOTICE: handleMessageFailure:
> > Msg failure (-19) for session 0x4add20 app  unused 0. Closing
> > connection. Apr  7 09:46:02 g14r10 : 0:0:eventd:INFO:
> > close_session: Lost connection. Close rmc session 0x4add20 for
> > event session (nil) app unused 0.
> > Apr  7 09:46:02 g14r10 : 0:0:eventd:NOTICE: handleMessageFailure:
> > Msg failure (-19) for session 0x4ae278 app  unused 0. Closing
> > connection. Apr  7 09:46:02 g14r10 : 0:0:eventd:INFO:
> > close_session: Lost connection. Close rmc session 0x4ae278 for
> > event session (nil) app unused 0.
> > Apr  7 09:46:03 g14r10 : 0:0:sdm:INFO: sdm_rcvRmcMsg:RMC Msg type
> > [4] {sdm:Agile1301}->{:scsi2.2.0} msg flags[1004] msg_id[221]
> > msg_seq[11028] status[-19] rc[-20]
> > Apr  7 09:48:36 g14r10 : 0:0:evm:ERROR: evm_readVolumeLabelReq:
> > Unable to read primary label for LUN[3PARdata_0565_000900565]
> > Apr  7 09:48:36 g14r10 : 0:0:evm:ERROR: evm_deleteVol: Cannot read
> > label for LUN[3PARdata_0565_000900565]
> > Apr  7 09:48:36 g14r10 : 0:0:evm:ERROR: evm_volCreateReqProc: Delete
> > failed[4294961669] for volume[vol_5]
> > Apr  7 10:09:24 g14r10 : 0:0:cm:WARNING: TXRX/FP module is down,
> > rebooting system.
> > Apr  7 10:09:24 g14r10 : 0:0:eventd:CRITICAL: Process-EVENT Node:
> > Name 'local', State Down, Msg 'Node going down for reboot!
> > (chassisd: because FP/TXRX module is down).'
> > Apr  7 10:09:25 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent: Lost
> > connect for
> > Apr  7 10:09:25 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent:
> > disconnected
> > 
> > Broadcast message from root@g14r10 (Mon Apr  7 10:09:26 2008):
> > 
> > The system is going down for reboot NOW!
> > Apr  7 10:09:27 g14r10 : 0:0:eventd:CRITICAL: Process-EVENT Node:
> > Name 'local', State Down, Msg 'Node going down for reboot!
> > (chassisd: because FP/TXRX module is down).'
> > Apr  7 10:09:28 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent: Lost
> > connect for
> > Apr  7 10:09:28 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent:
> > disconnected Apr  7 10:09:28 g14r10 : 0:0:cm:WARNING: TXRX/FP module
> > is down, rebooting system.
> > Apr  7 10:09:28 g14r10 : 0:0:eventd:INFO: close_session: Lost
> > connection. Close rmc session 0x4aa4d0 for event session (nil) app
> > unused 0.
> > 
> > Broadcast message from root@g14r10 (Mon Apr  7 10:09:29 2008):
> > 
> > The system is going down for reboot NOW!
> > Apr  7 10:09:32 g14r10 : 0:0:eventd:CRITICAL: Process-EVENT Node:
> > Name 'local', State Down, Msg 'Node going down for reboot!
> > (chassisd: because FP/TXRX module is down).'
> > Apr  7 10:09:33 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent: Lost
> > connect for
> > Connection to 10.2.10.14 closed by remote host.
> > Connection to 10.2.10.14 closed.
> > [sandrineb@sandrineb tst-evm]$
