X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C71288.70C4C7F8@onstor-exch02.onstor.net>; Mon, 27 Nov 2006 16:59:22 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C71288.70C4C7F8"
References: <BB375AF679D4A34E9CA8DFA650E2B04E0116B38C@onstor-exch02.onstor.net><BB375AF679D4A34E9CA8DFA650E2B04E0A93D3@onstor-exch02.onstor.net> <20061127163126.5f2c4336@ripper.onstor.net>
Content-class: urn:content-classes:message
Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk
Date: Mon, 27 Nov 2006 16:57:53 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E013D25B3@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Had a lab system loose /var/log and /tmp/ramdisk
thread-index: AccShIoy808heYfsTDyTHUPtfIQ1SgAA7HWb
From: "Paul Hammer" <paul.hammer@onstor.com>
To: "Andrew Sharp" <andy.sharp@onstor.com>,
	"Larry Scheer" <larry.scheer@onstor.com>
Cc: "Sandrine Boulanger" <sandrine.boulanger@onstor.com>,
	"Raj Kumar" <raj.kumar@onstor.com>,
	"Ken Renshaw" <ken.renshaw@onstor.com>,
	"John Rogers" <john.rogers@onstor.com>,
	"Eric Barrett" <eric.barrett@onstor.com>,
	"John VanderWerf" <john.vanderwerf@onstor.com>,
	"Kevin Matthews" <kevin.matthews@onstor.com>,
	"Brian Baker" <brian.baker@onstor.com>,
	"dl-QA" <dl-qa@onstor.com>,
	"Tim Gardner" <tim.gardner@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C71288.70C4C7F8
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Thanks Andy. There are some excellent testing points mentioned here, =
Sandrine can you have some one turn several of this points into test =
cases?
=20
Thanks,
=20
=20
-Paul

________________________________

From: Andrew Sharp [mailto:andy.sharp@onstor.com]
Sent: Mon 11/27/2006 4:31 PM
To: Larry Scheer
Cc: Sandrine Boulanger; Raj Kumar; Ken Renshaw; Paul Hammer; John =
Rogers; Eric Barrett; John VanderWerf; Kevin Matthews; Brian Baker; =
dl-QA; Tim Gardner
Subject: Re: Had a lab system loose /var/log and /tmp/ramdisk



The CRON process is SOP for cron.  It starts a separate process for
each pipeline, or at least for each crontab entry, to manage that job,
collect the output and email the output and results.  It capitalizes it
to easily distinguish those threads/procs from the cron daemon itself.

cron was probably stuck waiting for an open/creat on a temporary file
that was never going to happen because your /tmp was bullocks.

It appears to me that there is some sauce in the way that /tmp/ramdisk
works.  If I'm right, /tmp/ramdisk is used by any process that
creates/utilizes a file on /tmp (or /tmp/ramdisk directly).
If /tmp/ramdisk is not mounted, then that special facility won't be
used and files will be on the flash filesystem, which makes just about
everything considerably slower.

/var/log/messages going away is probably some bug in the /etc/rc script.

Cheers,

a

On Mon, 27 Nov 2006 15:32:50 -0800 "Larry Scheer"
<larry.scheer@onstor.com> wrote:

> Ah, that reminds me... When I did a ps -ax (on system that was missing
> /var/log and /tmp/ramdisk) I saw a process called "CRON" as well as
> "cron" running. I never seen a process "CRON" (all caps) before. I am
> not sure if cron creates a thread or child process called CRON on
> openbsd. It might be worth investigating.
>
> -----Original Message-----
> From: Sandrine Boulanger
> Sent: Monday, November 27, 2006 3:25 PM
> To: Raj Kumar; Ken Renshaw; Paul Hammer; John Rogers; Larry Scheer;
> Eric Barrett
> Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk
>
> Check crontab. I've seen a couple of filers where the first 20 lines
> or so of crontab was missing, and for example messages was not
> recycled anymore.
>
> -----Original Message-----
> From: Raj Kumar
> Sent: Monday, November 27, 2006 3:21 PM
> To: Ken Renshaw; Paul Hammer; John Rogers; Larry Scheer; Eric Barrett
> Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk
>
> I just encountered (rather noticed) a case where /var/log/messages
> file vanished after a reboot and never recreated again, even after
> multiple reboots. Every time syslogd complains about it.
>
> PR#16553
>
> -----Original Message-----
> From: Ken Renshaw
> Sent: Monday, November 27, 2006 2:39 PM
> To: Raj Kumar; Paul Hammer; John Rogers; Larry Scheer; Eric Barrett
> Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Subject: Re: Had a lab system loose /var/log and /tmp/ramdisk
>
> Actually, now that you mention it
>
>=20
>
> -----Original Message-----
> From: Raj Kumar
> To: Ken Renshaw; Paul Hammer; John Rogers; Larry Scheer; Eric Barrett
> CC: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Sent: Mon Nov 27 14:29:06 2006
> Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk
>
> Isn't /tmp/ramdisk used by ssssscccc also (GUI's)? Not sure.
>
> -----Original Message-----
> From: Ken Renshaw
> Sent: Monday, November 27, 2006 2:26 PM
> To: Paul Hammer; John Rogers; Larry Scheer; Eric Barrett
> Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Subject: Re: Had a lab system loose /var/log and /tmp/ramdisk
>
> I'm pretty sure that /tmp/ramdisk ONLY gets used as a mount point for
> the memfs during system upgrade. The release tarfiles are un-tarred
> into the ramdisk after the memfs creation is finished.
>
> I do not believe the directory/mount point gets used at all during
> normal filer operations.
>
> Just a couple odd data points that may or may not mean anything here.
>
> -Ken
>
>=20
>
> -----Original Message-----
> From: Paul Hammer
> To: John Rogers; Larry Scheer; Eric Barrett
> CC: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Sent: Mon Nov 27 14:20:47 2006
> Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk
>
> Also, thes two directories would be the logical ones to get corrupted
> since we write to them
>
> ________________________________
>
> From: John Rogers
> Sent: Mon 11/27/2006 1:23 PM
> To: Larry Scheer; Eric Barrett
> Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk
>
>
>
> Although the disappearance of /var/log and /tmp/ramdisk is indictitive
> of being hacked, I don't believe this is the case. We should process
> due diligence even if we just suspect it a little. Kevin can you tell
> us if we've detected any intrusions recently?
>
>
>
>
>
> I am more concerned that the crash corrupted the flash card and fsk
> cleaned /var/log beyond recognition.
>
> _____________________________________________
> From: Larry Scheer
> Sent: Monday, November 27, 2006 11:56 AM
> To: Eric Barrett
> Cc: John Rogers; John VanderWerf
> Subject: Had a lab system loose /var/log and /tmp/ramdisk
>
> Eric,
>
>    Many months ago I remember a customer reporting /var/log
> disappearing on them and you mentioned that this is a classic sign
> of  a system being hacked.
>
> On Wednesday evening I had a filer (a development test system) crash
> because /var/log and /tmp/ramdisk were removed. It could have been a
> failure with the flash or a software bug. I am not sure.
>
> Do you recall if there were any other reported problems similar to
> this in the field or what was determined to be the root cause of the
> problem (with /var/log disappearing) reported by the customer?
>
> Larry
>



------_=_NextPart_001_01C71288.70C4C7F8
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<HTML dir=3Dltr><HEAD><TITLE>Re: Had a lab system loose /var/log and =
/tmp/ramdisk</TITLE>=0A=
<META http-equiv=3DContent-Type content=3D"text/html; charset=3Dunicode">=0A=
<META content=3D"MSHTML 6.00.2900.2995" name=3DGENERATOR></HEAD>=0A=
<BODY>=0A=
<DIV id=3DidOWAReplyText36395 dir=3Dltr>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 size=3D2></FONT>Thanks =
Andy. There are some excellent testing points mentioned here, Sandrine =
can you have some one turn several of this points into test cases?</DIV>=0A=
<DIV dir=3Dltr>&nbsp;</DIV>=0A=
<DIV dir=3Dltr>Thanks,</DIV>=0A=
<DIV dir=3Dltr>&nbsp;</DIV>=0A=
<DIV dir=3Dltr>&nbsp;</DIV>=0A=
<DIV dir=3Dltr>-Paul<BR></DIV>=0A=
<DIV dir=3Dltr>=0A=
<HR tabIndex=3D-1>=0A=
</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DTahoma size=3D2><B>From:</B> Andrew Sharp =
[mailto:andy.sharp@onstor.com]<BR><B>Sent:</B> Mon 11/27/2006 4:31 =
PM<BR><B>To:</B> Larry Scheer<BR><B>Cc:</B> Sandrine Boulanger; Raj =
Kumar; Ken Renshaw; Paul Hammer; John Rogers; Eric Barrett; John =
VanderWerf; Kevin Matthews; Brian Baker; dl-QA; Tim =
Gardner<BR><B>Subject:</B> Re: Had a lab system loose /var/log and =
/tmp/ramdisk<BR></FONT><BR></DIV></DIV>=0A=
<DIV>=0A=
<P><FONT size=3D2>The CRON process is SOP for cron.&nbsp; It starts a =
separate process for<BR>each pipeline, or at least for each crontab =
entry, to manage that job,<BR>collect the output and email the output =
and results.&nbsp; It capitalizes it<BR>to easily distinguish those =
threads/procs from the cron daemon itself.<BR><BR>cron was probably =
stuck waiting for an open/creat on a temporary file<BR>that was never =
going to happen because your /tmp was bullocks.<BR><BR>It appears to me =
that there is some sauce in the way that /tmp/ramdisk<BR>works.&nbsp; If =
I'm right, /tmp/ramdisk is used by any process that<BR>creates/utilizes =
a file on /tmp (or /tmp/ramdisk directly).<BR>If /tmp/ramdisk is not =
mounted, then that special facility won't be<BR>used and files will be =
on the flash filesystem, which makes just about<BR>everything =
considerably slower.<BR><BR>/var/log/messages going away is probably =
some bug in the /etc/rc script.<BR><BR>Cheers,<BR><BR>a<BR><BR>On Mon, =
27 Nov 2006 15:32:50 -0800 "Larry =
Scheer"<BR>&lt;larry.scheer@onstor.com&gt; wrote:<BR><BR>&gt; Ah, that =
reminds me... When I did a ps -ax (on system that was missing<BR>&gt; =
/var/log and /tmp/ramdisk) I saw a process called "CRON" as well =
as<BR>&gt; "cron" running. I never seen a process "CRON" (all caps) =
before. I am<BR>&gt; not sure if cron creates a thread or child process =
called CRON on<BR>&gt; openbsd. It might be worth =
investigating.<BR>&gt;<BR>&gt; -----Original Message-----<BR>&gt; From: =
Sandrine Boulanger<BR>&gt; Sent: Monday, November 27, 2006 3:25 =
PM<BR>&gt; To: Raj Kumar; Ken Renshaw; Paul Hammer; John Rogers; Larry =
Scheer;<BR>&gt; Eric Barrett<BR>&gt; Cc: John VanderWerf; Kevin =
Matthews; Brian Baker; dl-QA<BR>&gt; Subject: RE: Had a lab system loose =
/var/log and /tmp/ramdisk<BR>&gt;<BR>&gt; Check crontab. I've seen a =
couple of filers where the first 20 lines<BR>&gt; or so of crontab was =
missing, and for example messages was not<BR>&gt; recycled =
anymore.<BR>&gt;<BR>&gt; -----Original Message-----<BR>&gt; From: Raj =
Kumar<BR>&gt; Sent: Monday, November 27, 2006 3:21 PM<BR>&gt; To: Ken =
Renshaw; Paul Hammer; John Rogers; Larry Scheer; Eric Barrett<BR>&gt; =
Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA<BR>&gt; Subject: =
RE: Had a lab system loose /var/log and /tmp/ramdisk<BR>&gt;<BR>&gt; I =
just encountered (rather noticed) a case where /var/log/messages<BR>&gt; =
file vanished after a reboot and never recreated again, even =
after<BR>&gt; multiple reboots. Every time syslogd complains about =
it.<BR>&gt;<BR>&gt; PR#16553<BR>&gt;<BR>&gt; -----Original =
Message-----<BR>&gt; From: Ken Renshaw<BR>&gt; Sent: Monday, November =
27, 2006 2:39 PM<BR>&gt; To: Raj Kumar; Paul Hammer; John Rogers; Larry =
Scheer; Eric Barrett<BR>&gt; Cc: John VanderWerf; Kevin Matthews; Brian =
Baker; dl-QA<BR>&gt; Subject: Re: Had a lab system loose /var/log and =
/tmp/ramdisk<BR>&gt;<BR>&gt; Actually, now that you mention =
it<BR>&gt;<BR>&gt;&nbsp;<BR>&gt;<BR>&gt; -----Original =
Message-----<BR>&gt; From: Raj Kumar<BR>&gt; To: Ken Renshaw; Paul =
Hammer; John Rogers; Larry Scheer; Eric Barrett<BR>&gt; CC: John =
VanderWerf; Kevin Matthews; Brian Baker; dl-QA<BR>&gt; Sent: Mon Nov 27 =
14:29:06 2006<BR>&gt; Subject: RE: Had a lab system loose /var/log and =
/tmp/ramdisk<BR>&gt;<BR>&gt; Isn't /tmp/ramdisk used by ssssscccc also =
(GUI's)? Not sure.<BR>&gt;<BR>&gt; -----Original Message-----<BR>&gt; =
From: Ken Renshaw<BR>&gt; Sent: Monday, November 27, 2006 2:26 =
PM<BR>&gt; To: Paul Hammer; John Rogers; Larry Scheer; Eric =
Barrett<BR>&gt; Cc: John VanderWerf; Kevin Matthews; Brian Baker; =
dl-QA<BR>&gt; Subject: Re: Had a lab system loose /var/log and =
/tmp/ramdisk<BR>&gt;<BR>&gt; I'm pretty sure that /tmp/ramdisk ONLY gets =
used as a mount point for<BR>&gt; the memfs during system upgrade. The =
release tarfiles are un-tarred<BR>&gt; into the ramdisk after the memfs =
creation is finished.<BR>&gt;<BR>&gt; I do not believe the =
directory/mount point gets used at all during<BR>&gt; normal filer =
operations.<BR>&gt;<BR>&gt; Just a couple odd data points that may or =
may not mean anything here.<BR>&gt;<BR>&gt; =
-Ken<BR>&gt;<BR>&gt;&nbsp;<BR>&gt;<BR>&gt; -----Original =
Message-----<BR>&gt; From: Paul Hammer<BR>&gt; To: John Rogers; Larry =
Scheer; Eric Barrett<BR>&gt; CC: John VanderWerf; Kevin Matthews; Brian =
Baker; dl-QA<BR>&gt; Sent: Mon Nov 27 14:20:47 2006<BR>&gt; Subject: RE: =
Had a lab system loose /var/log and /tmp/ramdisk<BR>&gt;<BR>&gt; Also, =
thes two directories would be the logical ones to get corrupted<BR>&gt; =
since we write to them<BR>&gt;<BR>&gt; =
________________________________<BR>&gt;<BR>&gt; From: John =
Rogers<BR>&gt; Sent: Mon 11/27/2006 1:23 PM<BR>&gt; To: Larry Scheer; =
Eric Barrett<BR>&gt; Cc: John VanderWerf; Kevin Matthews; Brian Baker; =
dl-QA<BR>&gt; Subject: RE: Had a lab system loose /var/log and =
/tmp/ramdisk<BR>&gt;<BR>&gt;<BR>&gt;<BR>&gt; Although the disappearance =
of /var/log and /tmp/ramdisk is indictitive<BR>&gt; of being hacked, I =
don't believe this is the case. We should process<BR>&gt; due diligence =
even if we just suspect it a little. Kevin can you tell<BR>&gt; us if =
we've detected any intrusions =
recently?<BR>&gt;<BR>&gt;<BR>&gt;<BR>&gt;<BR>&gt;<BR>&gt; I am more =
concerned that the crash corrupted the flash card and fsk<BR>&gt; =
cleaned /var/log beyond recognition.<BR>&gt;<BR>&gt; =
_____________________________________________<BR>&gt; From: Larry =
Scheer<BR>&gt; Sent: Monday, November 27, 2006 11:56 AM<BR>&gt; To: Eric =
Barrett<BR>&gt; Cc: John Rogers; John VanderWerf<BR>&gt; Subject: Had a =
lab system loose /var/log and /tmp/ramdisk<BR>&gt;<BR>&gt; =
Eric,<BR>&gt;<BR>&gt;&nbsp;&nbsp;&nbsp; Many months ago I remember a =
customer reporting /var/log<BR>&gt; disappearing on them and you =
mentioned that this is a classic sign<BR>&gt; of&nbsp; a system being =
hacked.<BR>&gt;<BR>&gt; On Wednesday evening I had a filer (a =
development test system) crash<BR>&gt; because /var/log and /tmp/ramdisk =
were removed. It could have been a<BR>&gt; failure with the flash or a =
software bug. I am not sure.<BR>&gt;<BR>&gt; Do you recall if there were =
any other reported problems similar to<BR>&gt; this in the field or what =
was determined to be the root cause of the<BR>&gt; problem (with =
/var/log disappearing) reported by the customer?<BR>&gt;<BR>&gt; =
Larry<BR>&gt;<BR></FONT></P></DIV></BODY></HTML>
------_=_NextPart_001_01C71288.70C4C7F8--
