AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20080407131755.1cd0f84e@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<rendell.fong@onstor.com>,<sandrine.boulanger@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@onstor-exch02.onstor.net/INBOX	0	BB375AF679D4A34E9CA8DFA650E2B04E03B1C176@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Mon, 7 Apr 2008 13:27:55 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: "Rendell Fong" <rendell.fong@onstor.com>
Cc: "Sandrine Boulanger" <sandrine.boulanger@onstor.com>
Subject: Re: why eventd sends 'Node going down for reboot' 3 times in a row?
Message-ID: <20080407132755.1832bcb2@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E03B1C176@onstor-exch02.onstor.net>
References: <BB375AF679D4A34E9CA8DFA650E2B04E05C74665@onstor-exch02.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E03B1C176@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit


On the question of frozen messages:

Probably have an old version of some files in /etc/logrotate.d
or /etc/default or something.  They aren't hurting anything.  If the
amount of space used by /var/spool/exim4 gets over a certain limit, the
frozen emails will get tossed.

On the delayed email:

The system was going down when the email was queued, and apparently DNS
or something necessary to get DNS from the management vsvr, was no
longer working, so the DNS failed.  That meant the email had a retry
time set.  The system booted back up, and when the retry time expired,
then the email was sent.  Hence the apparent 33 minute delay.  This
would seem to be WAD.

Cheers,

a

On Mon, 7 Apr 2008 11:12:04 -0700 "Rendell Fong"
<rendell.fong@onstor.com> wrote:

> Many internal email messages are getting generated and queued instead
> of being thrown away such that the mail queue is getting clogged.  It
> is delaying transmission of external email messages.  You'll need
> Andy's help on this.  Maybe there's a mail configuration issue.
> 
> _____________________________________________
> From: Sandrine Boulanger 
> Sent: Monday, April 07, 2008 11:07 AM
> To: Rendell Fong; Andy Sharp; Larry Scheer
> Subject: RE: why eventd sends 'Node going down for reboot' 3 times in
> a row?
> 
> OK. I also received the email at 10:43 when the event happened at
> 10:09, here's the end of the exim mailog file, too many frozen
> messages
> 
> 2008-04-07 09:03:27 End queue run: pid=31673
> 2008-04-07 09:17:03 1Jiu1n-0008GR-1R <= root@g14r10 U=root P=local
> S=629 2008-04-07 09:17:03 1Jiu1n-0008GR-1R ** deadend@g14r10
> <root@g14r10>: Unrouteable address
> 2008-04-07 09:17:03 1Jiu1n-0008GT-9N <= <> R=1Jiu1n-0008GR-1R
> U=Debian-exim P=local S=1413
> 2008-04-07 09:17:03 1Jiu1n-0008GT-9N ** deadend@g14r10 <root@g14r10>:
> Unrouteable address
> 2008-04-07 09:17:03 1Jiu1n-0008GT-9N Frozen (delivery error message)
> 2008-04-07 09:17:03 1Jiu1n-0008GR-1R Completed
> 2008-04-07 09:33:27 Start queue run: pid=31863
> 2008-04-07 09:33:27 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 09:33:27 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 09:33:27 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 09:33:27 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 09:33:27 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 09:33:27 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 09:33:27 1JiXYF-000627-LD Message is frozen
> 2008-04-07 09:33:27 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 09:33:27 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 09:33:27 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 09:33:27 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 09:33:27 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 09:33:27 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 09:33:27 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 09:33:27 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 09:33:27 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 09:33:27 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 09:33:27 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 09:33:27 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 09:33:27 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 09:33:27 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 09:33:27 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 09:33:27 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 09:33:27 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 09:33:27 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 09:33:27 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 09:33:27 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 09:33:27 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 09:33:27 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 09:33:27 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 09:33:27 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 09:33:27 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 09:33:27 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 09:33:27 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 09:33:27 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 09:33:27 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 09:33:27 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 09:33:27 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 09:33:27 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 09:33:27 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 09:33:27 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 09:33:27 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 09:33:27 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 09:33:27 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 09:33:27 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 09:33:27 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 09:33:27 1Jis9f-000856-6E Message is frozen
> 2008-04-07 09:33:27 1JiB4h-0003mr-Rt Unfrozen by errmsg timer
> 2008-04-07 09:33:27 1JiB4h-0003mr-Rt ** deadend@g14r10 <root@g14r10>:
> Unrouteable address
> 2008-04-07 09:33:27 1JiB4h-0003mr-Rt deadend@g14r10 <root@g14r10>:
> error ignored
> 2008-04-07 09:33:27 1JiB4h-0003mr-Rt Completed
> 2008-04-07 09:33:27 1JiC0l-0003sN-Ck Message is frozen
> 2008-04-07 09:33:27 End queue run: pid=31863
> 2008-04-07 10:03:27 Start queue run: pid=32068
> 2008-04-07 10:03:27 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 10:03:27 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 10:03:27 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 10:03:27 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 10:03:27 1JiXYF-000627-LD Message is frozen
> 2008-04-07 10:03:27 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 10:03:27 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 10:03:27 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 10:03:27 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 10:03:27 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 10:03:27 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 10:03:27 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 10:03:27 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 10:03:27 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 10:03:27 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 10:03:27 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 10:03:27 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 10:03:27 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 10:03:27 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 10:03:27 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 10:03:27 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 10:03:27 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 10:03:27 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 10:03:27 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 10:03:27 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 10:03:27 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 10:03:27 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 10:03:27 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 10:03:27 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 10:03:27 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 10:03:27 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 10:03:27 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 10:03:27 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 10:03:27 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 10:03:27 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 10:03:27 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 10:03:27 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 10:03:27 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 10:03:27 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 10:03:27 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 10:03:27 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 10:03:27 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 10:03:27 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 10:03:27 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 10:03:27 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 10:03:27 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 10:03:27 1Jis9f-000856-6E Message is frozen
> 2008-04-07 10:03:27 1JiC0l-0003sN-Ck Message is frozen
> 2008-04-07 10:03:27 End queue run: pid=32068
> 2008-04-07 10:09:25 1JiuqS-0008MN-Nc <= g14r10@onstor.com U=root
> P=local S=616
> 2008-04-07 10:09:28 1JiuqV-0008MR-Tk <= g14r10@onstor.com U=root
> P=local S=616
> 2008-04-07 10:09:32 1JiuqS-0008MN-Nc == sandrine.boulanger@onstor.com
> R=dnslookup defer (-1): host lookup did not complete
> 2008-04-07 10:09:33 1Jiuqa-0008OO-Ta <= g14r10@onstor.com U=root
> P=local S=616
> 2008-04-07 10:11:48 exim 4.63 daemon started: pid=804, -q30m,
> listening for SMTP on [127.0.0.1]:25
> 2008-04-07 10:11:48 Start queue run: pid=809
> 2008-04-07 10:11:48 1Jis9f-000856-6E Message is frozen
> 2008-04-07 10:11:48 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 10:11:48 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 10:11:48 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 10:11:48 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 10:11:49 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 10:11:49 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 10:11:50 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 10:11:50 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 10:11:51 1Jiuqa-0008OO-Ta == sandrine.boulanger@onstor.com
> routing defer (-51): retry time not reached
> 2008-04-07 10:11:51 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 10:11:51 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 10:11:51 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 10:11:51 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 10:11:51 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 10:11:51 1JiuqV-0008MR-Tk == sandrine.boulanger@onstor.com
> routing defer (-51): retry time not reached
> 2008-04-07 10:11:52 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 10:11:52 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 10:11:52 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 10:11:52 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 10:11:52 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 10:11:52 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 10:11:52 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 10:11:52 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 10:11:52 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 10:11:52 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 10:11:52 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 10:11:52 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 10:11:52 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 10:11:52 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 10:11:52 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 10:11:52 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 10:11:52 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 10:11:52 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 10:11:52 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 10:11:52 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 10:11:52 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 10:11:52 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 10:11:52 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 10:11:52 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 10:11:52 1JiXYF-000627-LD Message is frozen
> 2008-04-07 10:11:52 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 10:11:52 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 10:11:52 1JiuqS-0008MN-Nc == sandrine.boulanger@onstor.com
> routing defer (-51): retry time not reached
> 2008-04-07 10:11:52 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 10:11:52 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 10:11:52 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 10:11:52 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 10:11:52 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 10:11:52 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 10:11:52 1JiC0l-0003sN-Ck Message is frozen
> 2008-04-07 10:11:52 End queue run: pid=809
> 2008-04-07 10:12:47 exim 4.63 daemon started: pid=1489, -q30m,
> listening for SMTP on [127.0.0.1]:25
> 2008-04-07 10:12:47 Start queue run: pid=1490
> 2008-04-07 10:12:47 1JiC0l-0003sN-Ck Message is frozen
> 2008-04-07 10:12:47 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 10:12:47 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 10:12:47 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 10:12:47 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 10:12:47 1JiuqS-0008MN-Nc == sandrine.boulanger@onstor.com
> routing defer (-51): retry time not reached
> 2008-04-07 10:12:47 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 10:12:47 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 10:12:47 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 10:12:47 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 10:12:47 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 10:12:47 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 10:12:47 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 10:12:47 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 10:12:47 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 10:12:47 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 10:12:47 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 10:12:48 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 10:12:48 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 10:12:48 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 10:12:48 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 10:12:48 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 10:12:48 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 10:12:48 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 10:12:48 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 10:12:48 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 10:12:48 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 10:12:48 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 10:12:48 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 10:12:48 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 10:12:48 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 10:12:48 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 10:12:48 1JiuqV-0008MR-Tk == sandrine.boulanger@onstor.com
> routing defer (-51): retry time not reached
> 2008-04-07 10:12:48 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 10:12:48 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 10:12:48 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 10:12:48 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 10:12:48 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 10:12:48 1JiXYF-000627-LD Message is frozen
> 2008-04-07 10:12:48 1Jiuqa-0008OO-Ta == sandrine.boulanger@onstor.com
> routing defer (-51): retry time not reached
> 2008-04-07 10:12:48 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 10:12:48 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 10:12:48 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 10:12:48 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 10:12:48 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 10:12:48 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 10:12:48 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 10:12:48 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 10:12:48 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 10:12:48 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 10:12:48 1Jis9f-000856-6E Message is frozen
> 2008-04-07 10:12:48 End queue run: pid=1490
> 2008-04-07 10:13:16 1JiuuB-0000PP-NT <= g14r10@onstor.com U=root
> P=local S=576
> 2008-04-07 10:13:16 1JiuuB-0000PP-NT => sandrine.boulanger@onstor.com
> R=dnslookup T=remote_smtp H=mail.onstor.com [66.201.51.107]
> 2008-04-07 10:13:16 1JiuuB-0000PP-NT Completed
> 2008-04-07 10:17:03 1Jiuxr-0000lQ-Jr <= root@g14r10 U=root P=local
> S=629 2008-04-07 10:17:03 1Jiuxr-0000lQ-Jr ** deadend@g14r10
> <root@g14r10>: Unrouteable address
> 2008-04-07 10:17:04 1Jiuxr-0000lS-Ul <= <> R=1Jiuxr-0000lQ-Jr
> U=Debian-exim P=local S=1413
> 2008-04-07 10:17:04 1Jiuxr-0000lS-Ul ** deadend@g14r10 <root@g14r10>:
> Unrouteable address
> 2008-04-07 10:17:04 1Jiuxr-0000lS-Ul Frozen (delivery error message)
> 2008-04-07 10:17:04 1Jiuxr-0000lQ-Jr Completed
> 2008-04-07 10:34:49 1JivF2-0000mv-Rr <= g14r10@onstor.com U=root
> P=local S=611
> 2008-04-07 10:34:49 1JivF2-0000mv-Rr => sandrine.boulanger@onstor.com
> R=dnslookup T=remote_smtp H=mail.onstor.com [66.201.51.107]
> 2008-04-07 10:34:49 1JivF2-0000mv-Rr Completed
> 2008-04-07 10:42:47 Start queue run: pid=3341
> 2008-04-07 10:42:47 1JiC0l-0003sN-Ck Unfrozen by errmsg timer
> 2008-04-07 10:42:47 1JiC0l-0003sN-Ck ** deadend@g14r10 <root@g14r10>:
> Unrouteable address
> 2008-04-07 10:42:47 1JiC0l-0003sN-Ck deadend@g14r10 <root@g14r10>:
> error ignored
> 2008-04-07 10:42:47 1JiC0l-0003sN-Ck Completed
> 2008-04-07 10:42:47 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 10:42:47 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 10:42:47 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 10:42:48 1JiuqS-0008MN-Nc => sandrine.boulanger@onstor.com
> R=dnslookup T=remote_smtp H=mail.onstor.com [66.201.51.107]
> 2008-04-07 10:42:48 1JiuqS-0008MN-Nc Completed
> 2008-04-07 10:42:48 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 10:42:48 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 10:42:48 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 10:42:48 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 10:42:48 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 10:42:48 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 10:42:48 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 10:42:48 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 10:42:48 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 10:42:48 1Jiuxr-0000lS-Ul Message is frozen
> 2008-04-07 10:42:48 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 10:42:48 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 10:42:48 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 10:42:48 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 10:42:48 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 10:42:48 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 10:42:48 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 10:42:48 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 10:42:48 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 10:42:48 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 10:42:48 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 10:42:48 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 10:42:48 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 10:42:48 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 10:42:48 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 10:42:48 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 10:42:48 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 10:42:48 1JiuqV-0008MR-Tk => sandrine.boulanger@onstor.com
> R=dnslookup T=remote_smtp H=mail.onstor.com [66.201.51.107]
> 2008-04-07 10:42:48 1JiuqV-0008MR-Tk Completed
> 2008-04-07 10:42:48 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 10:42:48 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 10:42:48 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 10:42:48 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 10:42:48 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 10:42:48 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 10:42:48 1JiXYF-000627-LD Message is frozen
> 2008-04-07 10:42:48 1Jiuqa-0008OO-Ta => sandrine.boulanger@onstor.com
> R=dnslookup T=remote_smtp H=mail.onstor.com [66.201.51.107]
> 2008-04-07 10:42:48 1Jiuqa-0008OO-Ta Completed
> 2008-04-07 10:42:48 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 10:42:48 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 10:42:48 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 10:42:48 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 10:42:49 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 10:42:49 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 10:42:49 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 10:42:49 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 10:42:49 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 10:42:49 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 10:42:49 1Jis9f-000856-6E Message is frozen
> 2008-04-07 10:42:49 End queue run: pid=3341
> 2008-04-07 10:57:10 exim 4.63 daemon started: pid=795, -q30m,
> listening for SMTP on [127.0.0.1]:25
> 2008-04-07 10:57:10 Start queue run: pid=800
> 2008-04-07 10:57:10 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 10:57:10 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 10:57:10 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 10:57:10 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 10:57:10 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 10:57:10 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 10:57:10 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 10:57:12 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 10:57:12 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 10:57:12 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 10:57:12 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 10:57:12 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 10:57:12 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 10:57:12 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 10:57:12 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 10:57:13 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 10:57:13 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 10:57:13 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 10:57:13 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 10:57:13 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 10:57:13 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 10:57:13 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 10:57:13 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 10:57:13 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 10:57:13 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 10:57:13 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 10:57:13 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 10:57:13 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 10:57:13 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 10:57:13 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 10:57:13 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 10:57:13 1Jiuxr-0000lS-Ul Message is frozen
> 2008-04-07 10:57:13 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 10:57:13 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 10:57:13 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 10:57:13 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 10:57:13 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 10:57:13 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 10:57:13 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 10:57:13 1JiXYF-000627-LD Message is frozen
> 2008-04-07 10:57:13 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 10:57:13 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 10:57:13 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 10:57:13 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 10:57:13 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 10:57:13 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 10:57:13 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 10:57:13 1Jis9f-000856-6E Message is frozen
> 2008-04-07 10:57:13 End queue run: pid=800
> 2008-04-07 10:57:38 exim 4.63 daemon started: pid=1233, -q30m,
> listening for SMTP on [127.0.0.1]:25
> 2008-04-07 10:57:38 Start queue run: pid=1234
> 2008-04-07 10:57:38 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 10:57:38 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 10:57:38 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 10:57:38 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 10:57:38 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 10:57:38 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 10:57:38 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 10:57:38 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 10:57:38 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 10:57:38 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 10:57:38 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 10:57:38 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 10:57:38 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 10:57:38 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 10:57:38 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 10:57:38 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 10:57:38 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 10:57:38 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 10:57:38 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 10:57:38 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 10:57:38 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 10:57:38 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 10:57:38 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 10:57:38 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 10:57:38 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 10:57:38 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 10:57:38 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 10:57:38 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 10:57:38 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 10:57:38 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 10:57:38 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 10:57:38 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 10:57:38 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 10:57:38 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 10:57:38 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 10:57:38 1Jiuxr-0000lS-Ul Message is frozen
> 2008-04-07 10:57:38 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 10:57:38 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 10:57:38 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 10:57:38 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 10:57:38 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 10:57:38 1JiXYF-000627-LD Message is frozen
> 2008-04-07 10:57:38 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 10:57:38 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 10:57:38 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 10:57:38 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 10:57:38 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 10:57:38 1Jis9f-000856-6E Message is frozen
> 2008-04-07 10:57:38 End queue run: pid=1234
> g14r10:~#
> 
> _____________________________________________
> From: Rendell Fong 
> Sent: Monday, April 07, 2008 10:58 AM
> To: Sandrine Boulanger; Andy Sharp; Larry Scheer
> Subject: RE: why eventd sends 'Node going down for reboot' 3 times in
> a row?
> 
> The chassis daemon is checking every TXRX/FP cpu and sending the node
> event multiple times since multiple cpus are down.  I suppose once is
> enough. 
> 
> _____________________________________________
> From: Sandrine Boulanger 
> Sent: Monday, April 07, 2008 10:51 AM
> To: Andy Sharp; Larry Scheer; Rendell Fong
> Subject: why eventd sends 'Node going down for reboot' 3 times in a
> row?
> 
> I don't know yet if vol create triggered the crash, but I don't
> understand why eventd sends 3 times that the node is going down for
> reboot (I got 3 autosupport emails too)
> 
> Apr  7 09:45:26 g14r10 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol_5', Id 0x000107540000012f, Event 'Create', State 'Down'
> Apr  7 09:45:30 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> 1, CPU 0, State Down
> Apr  7 09:45:30 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> 1, CPU 1, State Down
> Apr  7 09:45:30 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 2, slot 1, cpu 0
> Apr  7 09:45:30 g14r10 : 0:0:vtm:INFO: vtm_sendCardStateMsg: Sending
> card DOWN to g14r10
> Apr  7 09:45:31 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 2, slot 1, cpu 1
> Apr  7 09:45:31 g14r10 : 0:0:vtm:INFO: Vtm_ProcCardStateMsg: card
> state DOWN message from g14r10
> Apr  7 09:45:31 g14r10 : 0:0:auth_agent:INFO: uninitCifsd:
> Uninitializing CIFS daemon
> Apr  7 09:45:31 g14r10 : 0:0:auth_agent:INFO: killCifsd[213]: Killing
> CIFS daemon (pid=5204) for VS=2[G14R10-1] intentionally.
> alreadyRebooting=0, force=0
> Apr  7 09:45:31 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> 1, CPU 2, State Down
> Apr  7 09:45:32 g14r10 : 0:0:evm:ERROR: evm_sendCreateLvReq : Timeout
> creating run time instance of volume vol_5
> Apr  7 09:45:32 g14r10 : 0:0:evm:ERROR: evm_onlineEvmSrvr : Cannot
> create LV instance on FP for volume[vol_5].
> Apr  7 09:45:32 g14r10 : 0:0:evm:ERROR: evm_putVolOnline: Cannot
> create run time instance of volume[vol_5]
> Apr  7 09:45:32 g14r10 : 0:0:ea:ERROR: ea_evmCreateVol[145]: Internal
> EVM error. Operation failed on volume[vol_5].
> Apr  7 09:45:32 g14r10 : 0:0:sanm:ERROR: SANM: FP NIM down. Aborting
> all mirror sessions.
> Apr  7 09:45:32 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 2, slot 1, cpu 2
> Apr  7 09:45:32 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> 1, CPU 3, State Down
> Apr  7 09:45:32 g14r10 : 0:0:sanm:ERROR: SANM: FP NIM down. Aborting
> all mirror sessions.
> Apr  7 09:45:32 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 2, slot 1, cpu 3
> Apr  7 09:45:32 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> 1, CPU 4, State Down
> Apr  7 09:45:32 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 2, slot 1, cpu 4
> Apr  7 09:45:32 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> 1, CPU 5, State Down
> Apr  7 09:45:32 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 2, slot 1, cpu 5
> Apr  7 09:45:32 g14r10 : 0:0:eventd:NOTICE: Process-EVENT IP i/f: IP
> 192.167.1.1, Port bp0, State Remove
> Apr  7 09:45:32 g14r10 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol_mgmt_67412', Id 0x000107540000006d, Event 'Offline', State 'Down'
> Apr  7 09:45:32 g14r10 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'sand', Id 0x0000075400000068, Event 'Offline', State 'Down'
> Apr  7 09:45:32 g14r10 : 0:0:eventd:WARNING: Process-EVENT Vsvr:
> Virtual server 'VS_MGMT_67412', Id 1, State 'Down'
> Apr  7 09:45:33 g14r10 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'sand2-m', Id 0x0001075400000099, Event 'Offline', State 'Down'
> Apr  7 09:45:33 g14r10 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'sand1', Id 0x000107540000006e, Event 'Offline', State 'Down'
> Apr  7 09:45:33 g14r10 : 0:0:eventd:NOTICE: Process-EVENT IP i/f: IP
> 192.167.2.1, Port bp0, State Remove
> Apr  7 09:45:33 g14r10 : 0:0:ea:INFO: Error string (Internal EVM
> error. Operation failed on volume[vol_5].) len (54)
> Apr  7 09:45:33 g14r10 : 0:0:eventd:WARNING: Process-EVENT Vsvr:
> Virtual server 'G14R10-1', Id 2, State 'Down'
> Apr  7 09:45:33 g14r10 : 0:0:nfxsh:NOTICE: cmd[16]: vol create vol_5
> 3PARdata_0565_000900565 : status[11]
> Apr  7 09:45:34 g14r10 : 0:0:vtm:ERROR: vtm_gatherInfo_proc: gather
> info failed for request from g14r10
> Apr  7 09:45:35 g14r10 : 0:0:vtm:ERROR: vtm_failover_vsvr_proc: no
> alternative filer found for vs 2 failover
> Apr  7 09:45:35 g14r10 : 0:0:vtm:ERROR: vtm_failover_vsvr_proc: no
> alternative filer found for vs 3 failover
> Apr  7 09:46:02 g14r10 : 0:0:eventd:NOTICE: handleMessageFailure: Msg
> failure (-19) for session 0x4add20 app  unused 0. Closing connection.
> Apr  7 09:46:02 g14r10 : 0:0:eventd:INFO: close_session: Lost
> connection. Close rmc session 0x4add20 for event session (nil) app
> unused 0.
> Apr  7 09:46:02 g14r10 : 0:0:eventd:NOTICE: handleMessageFailure: Msg
> failure (-19) for session 0x4ae278 app  unused 0. Closing connection.
> Apr  7 09:46:02 g14r10 : 0:0:eventd:INFO: close_session: Lost
> connection. Close rmc session 0x4ae278 for event session (nil) app
> unused 0.
> Apr  7 09:46:03 g14r10 : 0:0:sdm:INFO: sdm_rcvRmcMsg:RMC Msg type [4]
> {sdm:Agile1301}->{:scsi2.2.0} msg flags[1004] msg_id[221]
> msg_seq[11028] status[-19] rc[-20]
> Apr  7 09:48:36 g14r10 : 0:0:evm:ERROR: evm_readVolumeLabelReq: Unable
> to read primary label for LUN[3PARdata_0565_000900565]
> Apr  7 09:48:36 g14r10 : 0:0:evm:ERROR: evm_deleteVol: Cannot read
> label for LUN[3PARdata_0565_000900565]
> Apr  7 09:48:36 g14r10 : 0:0:evm:ERROR: evm_volCreateReqProc: Delete
> failed[4294961669] for volume[vol_5]
> Apr  7 10:09:24 g14r10 : 0:0:cm:WARNING: TXRX/FP module is down,
> rebooting system.
> Apr  7 10:09:24 g14r10 : 0:0:eventd:CRITICAL: Process-EVENT Node: Name
> 'local', State Down, Msg 'Node going down for reboot! (chassisd:
> because FP/TXRX module is down).'
> Apr  7 10:09:25 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent: Lost
> connect for
> Apr  7 10:09:25 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent:
> disconnected
> 
> Broadcast message from root@g14r10 (Mon Apr  7 10:09:26 2008):
> 
> The system is going down for reboot NOW!
> Apr  7 10:09:27 g14r10 : 0:0:eventd:CRITICAL: Process-EVENT Node: Name
> 'local', State Down, Msg 'Node going down for reboot! (chassisd:
> because FP/TXRX module is down).'
> Apr  7 10:09:28 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent: Lost
> connect for
> Apr  7 10:09:28 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent:
> disconnected Apr  7 10:09:28 g14r10 : 0:0:cm:WARNING: TXRX/FP module
> is down, rebooting system.
> Apr  7 10:09:28 g14r10 : 0:0:eventd:INFO: close_session: Lost
> connection. Close rmc session 0x4aa4d0 for event session (nil) app
> unused 0.
> 
> Broadcast message from root@g14r10 (Mon Apr  7 10:09:29 2008):
> 
> The system is going down for reboot NOW!
> Apr  7 10:09:32 g14r10 : 0:0:eventd:CRITICAL: Process-EVENT Node: Name
> 'local', State Down, Msg 'Node going down for reboot! (chassisd:
> because FP/TXRX module is down).'
> Apr  7 10:09:33 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent: Lost
> connect for
> Connection to 10.2.10.14 closed by remote host.
> Connection to 10.2.10.14 closed.
> [sandrineb@sandrineb tst-evm]$
