X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C898EE.2FF4D6E8@onstor-exch02.onstor.net>; Mon, 7 Apr 2008 13:30:15 -0700
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
Subject: RE: why eventd sends 'Node going down for reboot' 3 times in a row?
Date: Mon, 7 Apr 2008 13:30:15 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E05C7466E@onstor-exch02.onstor.net>
In-Reply-To: <20080407132755.1832bcb2@ripper.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: why eventd sends 'Node going down for reboot' 3 times in a row?
Thread-Index: AciY7d06Jb+s9YVxQvimpadGZetqtQAACrag
From: "Sandrine Boulanger" <sandrine.boulanger@onstor.com>
To: "Andy Sharp" <andy.sharp@onstor.com>,
	"Rendell Fong" <rendell.fong@onstor.com>

Maybe it's wad as to the way it works now, but it's useless to receive
the message so late, we should fix it

-----Original Message-----
From: Andy Sharp=20
Sent: Monday, April 07, 2008 1:28 PM
To: Rendell Fong
Cc: Sandrine Boulanger
Subject: Re: why eventd sends 'Node going down for reboot' 3 times in a
row?


On the question of frozen messages:

Probably have an old version of some files in /etc/logrotate.d
or /etc/default or something.  They aren't hurting anything.  If the
amount of space used by /var/spool/exim4 gets over a certain limit, the
frozen emails will get tossed.

On the delayed email:

The system was going down when the email was queued, and apparently DNS
or something necessary to get DNS from the management vsvr, was no
longer working, so the DNS failed.  That meant the email had a retry
time set.  The system booted back up, and when the retry time expired,
then the email was sent.  Hence the apparent 33 minute delay.  This
would seem to be WAD.

Cheers,

a

On Mon, 7 Apr 2008 11:12:04 -0700 "Rendell Fong"
<rendell.fong@onstor.com> wrote:

> Many internal email messages are getting generated and queued instead
> of being thrown away such that the mail queue is getting clogged.  It
> is delaying transmission of external email messages.  You'll need
> Andy's help on this.  Maybe there's a mail configuration issue.
>=20
> _____________________________________________
> From: Sandrine Boulanger=20
> Sent: Monday, April 07, 2008 11:07 AM
> To: Rendell Fong; Andy Sharp; Larry Scheer
> Subject: RE: why eventd sends 'Node going down for reboot' 3 times in
> a row?
>=20
> OK. I also received the email at 10:43 when the event happened at
> 10:09, here's the end of the exim mailog file, too many frozen
> messages
>=20
> 2008-04-07 09:03:27 End queue run: pid=3D31673
> 2008-04-07 09:17:03 1Jiu1n-0008GR-1R <=3D root@g14r10 U=3Droot =
P=3Dlocal
> S=3D629 2008-04-07 09:17:03 1Jiu1n-0008GR-1R ** deadend@g14r10
> <root@g14r10>: Unrouteable address
> 2008-04-07 09:17:03 1Jiu1n-0008GT-9N <=3D <> R=3D1Jiu1n-0008GR-1R
> U=3DDebian-exim P=3Dlocal S=3D1413
> 2008-04-07 09:17:03 1Jiu1n-0008GT-9N ** deadend@g14r10 <root@g14r10>:
> Unrouteable address
> 2008-04-07 09:17:03 1Jiu1n-0008GT-9N Frozen (delivery error message)
> 2008-04-07 09:17:03 1Jiu1n-0008GR-1R Completed
> 2008-04-07 09:33:27 Start queue run: pid=3D31863
> 2008-04-07 09:33:27 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 09:33:27 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 09:33:27 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 09:33:27 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 09:33:27 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 09:33:27 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 09:33:27 1JiXYF-000627-LD Message is frozen
> 2008-04-07 09:33:27 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 09:33:27 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 09:33:27 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 09:33:27 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 09:33:27 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 09:33:27 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 09:33:27 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 09:33:27 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 09:33:27 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 09:33:27 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 09:33:27 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 09:33:27 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 09:33:27 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 09:33:27 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 09:33:27 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 09:33:27 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 09:33:27 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 09:33:27 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 09:33:27 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 09:33:27 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 09:33:27 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 09:33:27 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 09:33:27 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 09:33:27 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 09:33:27 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 09:33:27 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 09:33:27 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 09:33:27 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 09:33:27 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 09:33:27 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 09:33:27 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 09:33:27 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 09:33:27 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 09:33:27 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 09:33:27 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 09:33:27 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 09:33:27 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 09:33:27 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 09:33:27 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 09:33:27 1Jis9f-000856-6E Message is frozen
> 2008-04-07 09:33:27 1JiB4h-0003mr-Rt Unfrozen by errmsg timer
> 2008-04-07 09:33:27 1JiB4h-0003mr-Rt ** deadend@g14r10 <root@g14r10>:
> Unrouteable address
> 2008-04-07 09:33:27 1JiB4h-0003mr-Rt deadend@g14r10 <root@g14r10>:
> error ignored
> 2008-04-07 09:33:27 1JiB4h-0003mr-Rt Completed
> 2008-04-07 09:33:27 1JiC0l-0003sN-Ck Message is frozen
> 2008-04-07 09:33:27 End queue run: pid=3D31863
> 2008-04-07 10:03:27 Start queue run: pid=3D32068
> 2008-04-07 10:03:27 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 10:03:27 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 10:03:27 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 10:03:27 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 10:03:27 1JiXYF-000627-LD Message is frozen
> 2008-04-07 10:03:27 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 10:03:27 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 10:03:27 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 10:03:27 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 10:03:27 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 10:03:27 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 10:03:27 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 10:03:27 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 10:03:27 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 10:03:27 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 10:03:27 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 10:03:27 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 10:03:27 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 10:03:27 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 10:03:27 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 10:03:27 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 10:03:27 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 10:03:27 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 10:03:27 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 10:03:27 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 10:03:27 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 10:03:27 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 10:03:27 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 10:03:27 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 10:03:27 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 10:03:27 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 10:03:27 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 10:03:27 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 10:03:27 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 10:03:27 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 10:03:27 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 10:03:27 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 10:03:27 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 10:03:27 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 10:03:27 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 10:03:27 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 10:03:27 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 10:03:27 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 10:03:27 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 10:03:27 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 10:03:27 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 10:03:27 1Jis9f-000856-6E Message is frozen
> 2008-04-07 10:03:27 1JiC0l-0003sN-Ck Message is frozen
> 2008-04-07 10:03:27 End queue run: pid=3D32068
> 2008-04-07 10:09:25 1JiuqS-0008MN-Nc <=3D g14r10@onstor.com U=3Droot
> P=3Dlocal S=3D616
> 2008-04-07 10:09:28 1JiuqV-0008MR-Tk <=3D g14r10@onstor.com U=3Droot
> P=3Dlocal S=3D616
> 2008-04-07 10:09:32 1JiuqS-0008MN-Nc =3D=3D =
sandrine.boulanger@onstor.com
> R=3Ddnslookup defer (-1): host lookup did not complete
> 2008-04-07 10:09:33 1Jiuqa-0008OO-Ta <=3D g14r10@onstor.com U=3Droot
> P=3Dlocal S=3D616
> 2008-04-07 10:11:48 exim 4.63 daemon started: pid=3D804, -q30m,
> listening for SMTP on [127.0.0.1]:25
> 2008-04-07 10:11:48 Start queue run: pid=3D809
> 2008-04-07 10:11:48 1Jis9f-000856-6E Message is frozen
> 2008-04-07 10:11:48 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 10:11:48 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 10:11:48 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 10:11:48 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 10:11:49 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 10:11:49 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 10:11:50 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 10:11:50 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 10:11:51 1Jiuqa-0008OO-Ta =3D=3D =
sandrine.boulanger@onstor.com
> routing defer (-51): retry time not reached
> 2008-04-07 10:11:51 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 10:11:51 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 10:11:51 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 10:11:51 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 10:11:51 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 10:11:51 1JiuqV-0008MR-Tk =3D=3D =
sandrine.boulanger@onstor.com
> routing defer (-51): retry time not reached
> 2008-04-07 10:11:52 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 10:11:52 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 10:11:52 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 10:11:52 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 10:11:52 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 10:11:52 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 10:11:52 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 10:11:52 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 10:11:52 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 10:11:52 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 10:11:52 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 10:11:52 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 10:11:52 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 10:11:52 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 10:11:52 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 10:11:52 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 10:11:52 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 10:11:52 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 10:11:52 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 10:11:52 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 10:11:52 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 10:11:52 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 10:11:52 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 10:11:52 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 10:11:52 1JiXYF-000627-LD Message is frozen
> 2008-04-07 10:11:52 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 10:11:52 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 10:11:52 1JiuqS-0008MN-Nc =3D=3D =
sandrine.boulanger@onstor.com
> routing defer (-51): retry time not reached
> 2008-04-07 10:11:52 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 10:11:52 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 10:11:52 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 10:11:52 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 10:11:52 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 10:11:52 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 10:11:52 1JiC0l-0003sN-Ck Message is frozen
> 2008-04-07 10:11:52 End queue run: pid=3D809
> 2008-04-07 10:12:47 exim 4.63 daemon started: pid=3D1489, -q30m,
> listening for SMTP on [127.0.0.1]:25
> 2008-04-07 10:12:47 Start queue run: pid=3D1490
> 2008-04-07 10:12:47 1JiC0l-0003sN-Ck Message is frozen
> 2008-04-07 10:12:47 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 10:12:47 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 10:12:47 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 10:12:47 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 10:12:47 1JiuqS-0008MN-Nc =3D=3D =
sandrine.boulanger@onstor.com
> routing defer (-51): retry time not reached
> 2008-04-07 10:12:47 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 10:12:47 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 10:12:47 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 10:12:47 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 10:12:47 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 10:12:47 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 10:12:47 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 10:12:47 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 10:12:47 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 10:12:47 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 10:12:47 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 10:12:48 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 10:12:48 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 10:12:48 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 10:12:48 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 10:12:48 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 10:12:48 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 10:12:48 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 10:12:48 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 10:12:48 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 10:12:48 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 10:12:48 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 10:12:48 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 10:12:48 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 10:12:48 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 10:12:48 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 10:12:48 1JiuqV-0008MR-Tk =3D=3D =
sandrine.boulanger@onstor.com
> routing defer (-51): retry time not reached
> 2008-04-07 10:12:48 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 10:12:48 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 10:12:48 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 10:12:48 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 10:12:48 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 10:12:48 1JiXYF-000627-LD Message is frozen
> 2008-04-07 10:12:48 1Jiuqa-0008OO-Ta =3D=3D =
sandrine.boulanger@onstor.com
> routing defer (-51): retry time not reached
> 2008-04-07 10:12:48 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 10:12:48 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 10:12:48 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 10:12:48 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 10:12:48 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 10:12:48 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 10:12:48 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 10:12:48 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 10:12:48 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 10:12:48 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 10:12:48 1Jis9f-000856-6E Message is frozen
> 2008-04-07 10:12:48 End queue run: pid=3D1490
> 2008-04-07 10:13:16 1JiuuB-0000PP-NT <=3D g14r10@onstor.com U=3Droot
> P=3Dlocal S=3D576
> 2008-04-07 10:13:16 1JiuuB-0000PP-NT =3D> =
sandrine.boulanger@onstor.com
> R=3Ddnslookup T=3Dremote_smtp H=3Dmail.onstor.com [66.201.51.107]
> 2008-04-07 10:13:16 1JiuuB-0000PP-NT Completed
> 2008-04-07 10:17:03 1Jiuxr-0000lQ-Jr <=3D root@g14r10 U=3Droot =
P=3Dlocal
> S=3D629 2008-04-07 10:17:03 1Jiuxr-0000lQ-Jr ** deadend@g14r10
> <root@g14r10>: Unrouteable address
> 2008-04-07 10:17:04 1Jiuxr-0000lS-Ul <=3D <> R=3D1Jiuxr-0000lQ-Jr
> U=3DDebian-exim P=3Dlocal S=3D1413
> 2008-04-07 10:17:04 1Jiuxr-0000lS-Ul ** deadend@g14r10 <root@g14r10>:
> Unrouteable address
> 2008-04-07 10:17:04 1Jiuxr-0000lS-Ul Frozen (delivery error message)
> 2008-04-07 10:17:04 1Jiuxr-0000lQ-Jr Completed
> 2008-04-07 10:34:49 1JivF2-0000mv-Rr <=3D g14r10@onstor.com U=3Droot
> P=3Dlocal S=3D611
> 2008-04-07 10:34:49 1JivF2-0000mv-Rr =3D> =
sandrine.boulanger@onstor.com
> R=3Ddnslookup T=3Dremote_smtp H=3Dmail.onstor.com [66.201.51.107]
> 2008-04-07 10:34:49 1JivF2-0000mv-Rr Completed
> 2008-04-07 10:42:47 Start queue run: pid=3D3341
> 2008-04-07 10:42:47 1JiC0l-0003sN-Ck Unfrozen by errmsg timer
> 2008-04-07 10:42:47 1JiC0l-0003sN-Ck ** deadend@g14r10 <root@g14r10>:
> Unrouteable address
> 2008-04-07 10:42:47 1JiC0l-0003sN-Ck deadend@g14r10 <root@g14r10>:
> error ignored
> 2008-04-07 10:42:47 1JiC0l-0003sN-Ck Completed
> 2008-04-07 10:42:47 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 10:42:47 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 10:42:47 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 10:42:48 1JiuqS-0008MN-Nc =3D> =
sandrine.boulanger@onstor.com
> R=3Ddnslookup T=3Dremote_smtp H=3Dmail.onstor.com [66.201.51.107]
> 2008-04-07 10:42:48 1JiuqS-0008MN-Nc Completed
> 2008-04-07 10:42:48 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 10:42:48 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 10:42:48 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 10:42:48 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 10:42:48 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 10:42:48 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 10:42:48 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 10:42:48 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 10:42:48 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 10:42:48 1Jiuxr-0000lS-Ul Message is frozen
> 2008-04-07 10:42:48 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 10:42:48 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 10:42:48 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 10:42:48 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 10:42:48 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 10:42:48 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 10:42:48 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 10:42:48 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 10:42:48 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 10:42:48 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 10:42:48 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 10:42:48 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 10:42:48 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 10:42:48 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 10:42:48 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 10:42:48 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 10:42:48 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 10:42:48 1JiuqV-0008MR-Tk =3D> =
sandrine.boulanger@onstor.com
> R=3Ddnslookup T=3Dremote_smtp H=3Dmail.onstor.com [66.201.51.107]
> 2008-04-07 10:42:48 1JiuqV-0008MR-Tk Completed
> 2008-04-07 10:42:48 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 10:42:48 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 10:42:48 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 10:42:48 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 10:42:48 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 10:42:48 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 10:42:48 1JiXYF-000627-LD Message is frozen
> 2008-04-07 10:42:48 1Jiuqa-0008OO-Ta =3D> =
sandrine.boulanger@onstor.com
> R=3Ddnslookup T=3Dremote_smtp H=3Dmail.onstor.com [66.201.51.107]
> 2008-04-07 10:42:48 1Jiuqa-0008OO-Ta Completed
> 2008-04-07 10:42:48 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 10:42:48 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 10:42:48 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 10:42:48 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 10:42:49 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 10:42:49 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 10:42:49 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 10:42:49 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 10:42:49 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 10:42:49 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 10:42:49 1Jis9f-000856-6E Message is frozen
> 2008-04-07 10:42:49 End queue run: pid=3D3341
> 2008-04-07 10:57:10 exim 4.63 daemon started: pid=3D795, -q30m,
> listening for SMTP on [127.0.0.1]:25
> 2008-04-07 10:57:10 Start queue run: pid=3D800
> 2008-04-07 10:57:10 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 10:57:10 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 10:57:10 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 10:57:10 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 10:57:10 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 10:57:10 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 10:57:10 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 10:57:12 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 10:57:12 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 10:57:12 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 10:57:12 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 10:57:12 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 10:57:12 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 10:57:12 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 10:57:12 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 10:57:13 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 10:57:13 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 10:57:13 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 10:57:13 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 10:57:13 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 10:57:13 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 10:57:13 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 10:57:13 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 10:57:13 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 10:57:13 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 10:57:13 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 10:57:13 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 10:57:13 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 10:57:13 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 10:57:13 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 10:57:13 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 10:57:13 1Jiuxr-0000lS-Ul Message is frozen
> 2008-04-07 10:57:13 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 10:57:13 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 10:57:13 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 10:57:13 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 10:57:13 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 10:57:13 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 10:57:13 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 10:57:13 1JiXYF-000627-LD Message is frozen
> 2008-04-07 10:57:13 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 10:57:13 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 10:57:13 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 10:57:13 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 10:57:13 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 10:57:13 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 10:57:13 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 10:57:13 1Jis9f-000856-6E Message is frozen
> 2008-04-07 10:57:13 End queue run: pid=3D800
> 2008-04-07 10:57:38 exim 4.63 daemon started: pid=3D1233, -q30m,
> listening for SMTP on [127.0.0.1]:25
> 2008-04-07 10:57:38 Start queue run: pid=3D1234
> 2008-04-07 10:57:38 1JiWcB-0005wX-BK Message is frozen
> 2008-04-07 10:57:38 1JicEZ-0006U5-Mi Message is frozen
> 2008-04-07 10:57:38 1Jigut-0006wJ-PM Message is frozen
> 2008-04-07 10:57:38 1JioPP-0007gM-Lw Message is frozen
> 2008-04-07 10:57:38 1Jikf9-0007J4-KQ Message is frozen
> 2008-04-07 10:57:38 1JiYUJ-00067h-2d Message is frozen
> 2008-04-07 10:57:38 1Jif2m-0006l1-0v Message is frozen
> 2008-04-07 10:57:38 1Jijj5-0007DF-64 Message is frozen
> 2008-04-07 10:57:38 1JibIV-0006OS-9h Message is frozen
> 2008-04-07 10:57:38 1JiqHX-0007re-CR Message is frozen
> 2008-04-07 10:57:38 1JimXH-0007V4-1v Message is frozen
> 2008-04-07 10:57:38 1JiUk3-0005gS-O3 Message is frozen
> 2008-04-07 10:57:38 1JiQzn-0005K8-1z Message is frozen
> 2008-04-07 10:57:38 1JiJVG-0004ab-Tk Message is frozen
> 2008-04-07 10:57:38 1JiIZD-0004V2-98 Message is frozen
> 2008-04-07 10:57:38 1JiGh5-0004Jx-MT Message is frozen
> 2008-04-07 10:57:38 1JilbD-0007P4-96 Message is frozen
> 2008-04-07 10:57:38 1JiMJT-0004rL-7v Message is frozen
> 2008-04-07 10:57:38 1Jifyp-0006qf-9n Message is frozen
> 2008-04-07 10:57:38 1JiP7f-00058c-Gm Message is frozen
> 2008-04-07 10:57:38 1JiLNP-0004ll-QE Message is frozen
> 2008-04-07 10:57:38 1Jit5j-0008Ap-0G Message is frozen
> 2008-04-07 10:57:38 1JipLT-0007m0-1e Message is frozen
> 2008-04-07 10:57:38 1JiCwq-0003xt-05 Message is frozen
> 2008-04-07 10:57:38 1JiTnz-0005as-8v Message is frozen
> 2008-04-07 10:57:38 1JinTL-0007ai-Bz Message is frozen
> 2008-04-07 10:57:38 1JidAd-0006Zj-1K Message is frozen
> 2008-04-07 10:57:38 1JiSru-0005VI-SW Message is frozen
> 2008-04-07 10:57:38 1JiaMR-0006Ir-Rg Message is frozen
> 2008-04-07 10:57:38 1JiKRL-0004gC-Cs Message is frozen
> 2008-04-07 10:57:38 1JiEox-00048v-T7 Message is frozen
> 2008-04-07 10:57:38 1Jiin1-00077b-Qa Message is frozen
> 2008-04-07 10:57:38 1JiVg7-0005qs-Ky Message is frozen
> 2008-04-07 10:57:38 1JiQ3j-0005EY-7x Message is frozen
> 2008-04-07 10:57:38 1JirDb-0007xJ-8A Message is frozen
> 2008-04-07 10:57:38 1Jiuxr-0000lS-Ul Message is frozen
> 2008-04-07 10:57:38 1JiFl1-0004ER-9T Message is frozen
> 2008-04-07 10:57:38 1JiZQN-0006DH-FX Message is frozen
> 2008-04-07 10:57:38 1JiHd9-0004PU-0J Message is frozen
> 2008-04-07 10:57:38 1Jiu1n-0008GT-9N Message is frozen
> 2008-04-07 10:57:38 1JiOQ9-00053v-4I Message is frozen
> 2008-04-07 10:57:38 1JiXYF-000627-LD Message is frozen
> 2008-04-07 10:57:38 1JiDst-00043P-Gj Message is frozen
> 2008-04-07 10:57:38 1JiNFX-0004wv-J2 Message is frozen
> 2008-04-07 10:57:38 1Jie6h-0006fN-E7 Message is frozen
> 2008-04-07 10:57:38 1JiRvr-0005Pi-HI Message is frozen
> 2008-04-07 10:57:38 1Jihqx-00071x-5v Message is frozen
> 2008-04-07 10:57:38 1Jis9f-000856-6E Message is frozen
> 2008-04-07 10:57:38 End queue run: pid=3D1234
> g14r10:~#
>=20
> _____________________________________________
> From: Rendell Fong=20
> Sent: Monday, April 07, 2008 10:58 AM
> To: Sandrine Boulanger; Andy Sharp; Larry Scheer
> Subject: RE: why eventd sends 'Node going down for reboot' 3 times in
> a row?
>=20
> The chassis daemon is checking every TXRX/FP cpu and sending the node
> event multiple times since multiple cpus are down.  I suppose once is
> enough.=20
>=20
> _____________________________________________
> From: Sandrine Boulanger=20
> Sent: Monday, April 07, 2008 10:51 AM
> To: Andy Sharp; Larry Scheer; Rendell Fong
> Subject: why eventd sends 'Node going down for reboot' 3 times in a
> row?
>=20
> I don't know yet if vol create triggered the crash, but I don't
> understand why eventd sends 3 times that the node is going down for
> reboot (I got 3 autosupport emails too)
>=20
> Apr  7 09:45:26 g14r10 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol_5', Id 0x000107540000012f, Event 'Create', State 'Down'
> Apr  7 09:45:30 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> 1, CPU 0, State Down
> Apr  7 09:45:30 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> 1, CPU 1, State Down
> Apr  7 09:45:30 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 2, slot 1, cpu 0
> Apr  7 09:45:30 g14r10 : 0:0:vtm:INFO: vtm_sendCardStateMsg: Sending
> card DOWN to g14r10
> Apr  7 09:45:31 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 2, slot 1, cpu 1
> Apr  7 09:45:31 g14r10 : 0:0:vtm:INFO: Vtm_ProcCardStateMsg: card
> state DOWN message from g14r10
> Apr  7 09:45:31 g14r10 : 0:0:auth_agent:INFO: uninitCifsd:
> Uninitializing CIFS daemon
> Apr  7 09:45:31 g14r10 : 0:0:auth_agent:INFO: killCifsd[213]: Killing
> CIFS daemon (pid=3D5204) for VS=3D2[G14R10-1] intentionally.
> alreadyRebooting=3D0, force=3D0
> Apr  7 09:45:31 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> 1, CPU 2, State Down
> Apr  7 09:45:32 g14r10 : 0:0:evm:ERROR: evm_sendCreateLvReq : Timeout
> creating run time instance of volume vol_5
> Apr  7 09:45:32 g14r10 : 0:0:evm:ERROR: evm_onlineEvmSrvr : Cannot
> create LV instance on FP for volume[vol_5].
> Apr  7 09:45:32 g14r10 : 0:0:evm:ERROR: evm_putVolOnline: Cannot
> create run time instance of volume[vol_5]
> Apr  7 09:45:32 g14r10 : 0:0:ea:ERROR: ea_evmCreateVol[145]: Internal
> EVM error. Operation failed on volume[vol_5].
> Apr  7 09:45:32 g14r10 : 0:0:sanm:ERROR: SANM: FP NIM down. Aborting
> all mirror sessions.
> Apr  7 09:45:32 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 2, slot 1, cpu 2
> Apr  7 09:45:32 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> 1, CPU 3, State Down
> Apr  7 09:45:32 g14r10 : 0:0:sanm:ERROR: SANM: FP NIM down. Aborting
> all mirror sessions.
> Apr  7 09:45:32 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 2, slot 1, cpu 3
> Apr  7 09:45:32 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> 1, CPU 4, State Down
> Apr  7 09:45:32 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 2, slot 1, cpu 4
> Apr  7 09:45:32 g14r10 : 0:0:eventd:WARNING: Process-EVENT CPU: Slot
> 1, CPU 5, State Down
> Apr  7 09:45:32 g14r10 : 0:0:vtm:NOTICE: Vtm_ProcEvntMsg:
> NFX_EVENT_CPU, state 2, slot 1, cpu 5
> Apr  7 09:45:32 g14r10 : 0:0:eventd:NOTICE: Process-EVENT IP i/f: IP
> 192.167.1.1, Port bp0, State Remove
> Apr  7 09:45:32 g14r10 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'vol_mgmt_67412', Id 0x000107540000006d, Event 'Offline', State 'Down'
> Apr  7 09:45:32 g14r10 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'sand', Id 0x0000075400000068, Event 'Offline', State 'Down'
> Apr  7 09:45:32 g14r10 : 0:0:eventd:WARNING: Process-EVENT Vsvr:
> Virtual server 'VS_MGMT_67412', Id 1, State 'Down'
> Apr  7 09:45:33 g14r10 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'sand2-m', Id 0x0001075400000099, Event 'Offline', State 'Down'
> Apr  7 09:45:33 g14r10 : 0:0:eventd:NOTICE: Process-EVENT Volume: name
> 'sand1', Id 0x000107540000006e, Event 'Offline', State 'Down'
> Apr  7 09:45:33 g14r10 : 0:0:eventd:NOTICE: Process-EVENT IP i/f: IP
> 192.167.2.1, Port bp0, State Remove
> Apr  7 09:45:33 g14r10 : 0:0:ea:INFO: Error string (Internal EVM
> error. Operation failed on volume[vol_5].) len (54)
> Apr  7 09:45:33 g14r10 : 0:0:eventd:WARNING: Process-EVENT Vsvr:
> Virtual server 'G14R10-1', Id 2, State 'Down'
> Apr  7 09:45:33 g14r10 : 0:0:nfxsh:NOTICE: cmd[16]: vol create vol_5
> 3PARdata_0565_000900565 : status[11]
> Apr  7 09:45:34 g14r10 : 0:0:vtm:ERROR: vtm_gatherInfo_proc: gather
> info failed for request from g14r10
> Apr  7 09:45:35 g14r10 : 0:0:vtm:ERROR: vtm_failover_vsvr_proc: no
> alternative filer found for vs 2 failover
> Apr  7 09:45:35 g14r10 : 0:0:vtm:ERROR: vtm_failover_vsvr_proc: no
> alternative filer found for vs 3 failover
> Apr  7 09:46:02 g14r10 : 0:0:eventd:NOTICE: handleMessageFailure: Msg
> failure (-19) for session 0x4add20 app  unused 0. Closing connection.
> Apr  7 09:46:02 g14r10 : 0:0:eventd:INFO: close_session: Lost
> connection. Close rmc session 0x4add20 for event session (nil) app
> unused 0.
> Apr  7 09:46:02 g14r10 : 0:0:eventd:NOTICE: handleMessageFailure: Msg
> failure (-19) for session 0x4ae278 app  unused 0. Closing connection.
> Apr  7 09:46:02 g14r10 : 0:0:eventd:INFO: close_session: Lost
> connection. Close rmc session 0x4ae278 for event session (nil) app
> unused 0.
> Apr  7 09:46:03 g14r10 : 0:0:sdm:INFO: sdm_rcvRmcMsg:RMC Msg type [4]
> {sdm:Agile1301}->{:scsi2.2.0} msg flags[1004] msg_id[221]
> msg_seq[11028] status[-19] rc[-20]
> Apr  7 09:48:36 g14r10 : 0:0:evm:ERROR: evm_readVolumeLabelReq: Unable
> to read primary label for LUN[3PARdata_0565_000900565]
> Apr  7 09:48:36 g14r10 : 0:0:evm:ERROR: evm_deleteVol: Cannot read
> label for LUN[3PARdata_0565_000900565]
> Apr  7 09:48:36 g14r10 : 0:0:evm:ERROR: evm_volCreateReqProc: Delete
> failed[4294961669] for volume[vol_5]
> Apr  7 10:09:24 g14r10 : 0:0:cm:WARNING: TXRX/FP module is down,
> rebooting system.
> Apr  7 10:09:24 g14r10 : 0:0:eventd:CRITICAL: Process-EVENT Node: Name
> 'local', State Down, Msg 'Node going down for reboot! (chassisd:
> because FP/TXRX module is down).'
> Apr  7 10:09:25 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent: Lost
> connect for
> Apr  7 10:09:25 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent:
> disconnected
>=20
> Broadcast message from root@g14r10 (Mon Apr  7 10:09:26 2008):
>=20
> The system is going down for reboot NOW!
> Apr  7 10:09:27 g14r10 : 0:0:eventd:CRITICAL: Process-EVENT Node: Name
> 'local', State Down, Msg 'Node going down for reboot! (chassisd:
> because FP/TXRX module is down).'
> Apr  7 10:09:28 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent: Lost
> connect for
> Apr  7 10:09:28 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent:
> disconnected Apr  7 10:09:28 g14r10 : 0:0:cm:WARNING: TXRX/FP module
> is down, rebooting system.
> Apr  7 10:09:28 g14r10 : 0:0:eventd:INFO: close_session: Lost
> connection. Close rmc session 0x4aa4d0 for event session (nil) app
> unused 0.
>=20
> Broadcast message from root@g14r10 (Mon Apr  7 10:09:29 2008):
>=20
> The system is going down for reboot NOW!
> Apr  7 10:09:32 g14r10 : 0:0:eventd:CRITICAL: Process-EVENT Node: Name
> 'local', State Down, Msg 'Node going down for reboot! (chassisd:
> because FP/TXRX module is down).'
> Apr  7 10:09:33 g14r10 : 0:0:spm:NOTICE: spm_ncmNodeEvent: Lost
> connect for
> Connection to 10.2.10.14 closed by remote host.
> Connection to 10.2.10.14 closed.
> [sandrineb@sandrineb tst-evm]$
