X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7127C.5A2ED0FD@onstor-exch02.onstor.net>; Mon, 27 Nov 2006 15:32:50 -0800
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk
Date: Mon, 27 Nov 2006 15:32:50 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E0A93D3@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E0116B38C@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Had a lab system loose /var/log and /tmp/ramdisk
thread-index: AccSXg5ixa9B7EZBRnu4OiZ+sTaUMAAC2NZgAAI2BEsAACt4XAAAGZGQAABfUm4AAV8YgAAAMpDQAAAozDA=
From: "Larry Scheer" <larry.scheer@onstor.com>
To: "Sandrine Boulanger" <sandrine.boulanger@onstor.com>,
	"Raj Kumar" <raj.kumar@onstor.com>,
	"Ken Renshaw" <ken.renshaw@onstor.com>,
	"Paul Hammer" <paul.hammer@onstor.com>,
	"John Rogers" <john.rogers@onstor.com>,
	"Eric Barrett" <eric.barrett@onstor.com>
Cc: "John VanderWerf" <john.vanderwerf@onstor.com>,
	"Kevin Matthews" <kevin.matthews@onstor.com>,
	"Brian Baker" <brian.baker@onstor.com>,
	"dl-QA" <dl-qa@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>,
	"Tim Gardner" <tim.gardner@onstor.com>

Ah, that reminds me... When I did a ps -ax (on system that was missing
/var/log and /tmp/ramdisk) I saw a process called "CRON" as well as
"cron" running. I never seen a process "CRON" (all caps) before. I am
not sure if cron creates a thread or child process called CRON on
openbsd. It might be worth investigating.

-----Original Message-----
From: Sandrine Boulanger=20
Sent: Monday, November 27, 2006 3:25 PM
To: Raj Kumar; Ken Renshaw; Paul Hammer; John Rogers; Larry Scheer; Eric
Barrett
Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk

Check crontab. I've seen a couple of filers where the first 20 lines or
so of crontab was missing, and for example messages was not recycled
anymore.

-----Original Message-----
From: Raj Kumar=20
Sent: Monday, November 27, 2006 3:21 PM
To: Ken Renshaw; Paul Hammer; John Rogers; Larry Scheer; Eric Barrett
Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk

I just encountered (rather noticed) a case where /var/log/messages file
vanished after a reboot and never recreated again, even after multiple
reboots. Every time syslogd complains about it.=20

PR#16553

-----Original Message-----
From: Ken Renshaw=20
Sent: Monday, November 27, 2006 2:39 PM
To: Raj Kumar; Paul Hammer; John Rogers; Larry Scheer; Eric Barrett
Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
Subject: Re: Had a lab system loose /var/log and /tmp/ramdisk

Actually, now that you mention it=20

=20

-----Original Message-----
From: Raj Kumar
To: Ken Renshaw; Paul Hammer; John Rogers; Larry Scheer; Eric Barrett
CC: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
Sent: Mon Nov 27 14:29:06 2006
Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk

Isn't /tmp/ramdisk used by ssssscccc also (GUI's)? Not sure.

-----Original Message-----
From: Ken Renshaw=20
Sent: Monday, November 27, 2006 2:26 PM
To: Paul Hammer; John Rogers; Larry Scheer; Eric Barrett
Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
Subject: Re: Had a lab system loose /var/log and /tmp/ramdisk

I'm pretty sure that /tmp/ramdisk ONLY gets used as a mount point for
the memfs during system upgrade. The release tarfiles are un-tarred into
the ramdisk after the memfs creation is finished.

I do not believe the directory/mount point gets used at all during
normal filer operations.

Just a couple odd data points that may or may not mean anything here.

-Ken

=20

-----Original Message-----
From: Paul Hammer
To: John Rogers; Larry Scheer; Eric Barrett
CC: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
Sent: Mon Nov 27 14:20:47 2006
Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk

Also, thes two directories would be the logical ones to get corrupted
since we write to them

________________________________

From: John Rogers
Sent: Mon 11/27/2006 1:23 PM
To: Larry Scheer; Eric Barrett
Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk



Although the disappearance of /var/log and /tmp/ramdisk is indictitive
of being hacked, I don't believe this is the case. We should process due
diligence even if we just suspect it a little. Kevin can you tell us if
we've detected any intrusions recently?





I am more concerned that the crash corrupted the flash card and fsk
cleaned /var/log beyond recognition.

_____________________________________________
From: Larry Scheer
Sent: Monday, November 27, 2006 11:56 AM
To: Eric Barrett
Cc: John Rogers; John VanderWerf
Subject: Had a lab system loose /var/log and /tmp/ramdisk

Eric,

   Many months ago I remember a customer reporting /var/log disappearing
on them and you mentioned that this is a classic sign of  a system being
hacked.

On Wednesday evening I had a filer (a development test system) crash
because /var/log and /tmp/ramdisk were removed. It could have been a
failure with the flash or a software bug. I am not sure.=20

Do you recall if there were any other reported problems similar to this
in the field or what was determined to be the root cause of the problem
(with /var/log disappearing) reported by the customer?

Larry

