AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20061127162549.762a2a93@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<larry.scheer@onstor.com>,<sandrine.boulanger@onstor.com>,<raj.kumar@onstor.com>,<ken.renshaw@onstor.com>,<paul.hammer@onstor.com>,<john.rogers@onstor.com>,<eric.barrett@onstor.com>,<john.vanderwerf@onstor.com>,<kevin.matthews@onstor.com>,<brian.baker@onstor.com>,<dl-qa@onstor.com>,<tim.gardner@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@onstor-exch02.onstor.net/INBOX	0	BB375AF679D4A34E9CA8DFA650E2B04E0A93D3@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Mon, 27 Nov 2006 16:31:26 -0800
From: Andrew Sharp <andy.sharp@onstor.com>
To: "Larry Scheer" <larry.scheer@onstor.com>
Cc: "Sandrine Boulanger" <sandrine.boulanger@onstor.com>, "Raj Kumar"
 <raj.kumar@onstor.com>, "Ken Renshaw" <ken.renshaw@onstor.com>, "Paul
 Hammer" <paul.hammer@onstor.com>, "John Rogers" <john.rogers@onstor.com>,
 "Eric Barrett" <eric.barrett@onstor.com>, "John VanderWerf"
 <john.vanderwerf@onstor.com>, "Kevin Matthews" <kevin.matthews@onstor.com>,
 "Brian Baker" <brian.baker@onstor.com>, "dl-QA" <dl-qa@onstor.com>, "Tim
 Gardner" <tim.gardner@onstor.com>
Subject: Re: Had a lab system loose /var/log and /tmp/ramdisk
Message-ID: <20061127163126.5f2c4336@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E0A93D3@onstor-exch02.onstor.net>
References: <BB375AF679D4A34E9CA8DFA650E2B04E0116B38C@onstor-exch02.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E0A93D3@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.5.6 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

The CRON process is SOP for cron.  It starts a separate process for
each pipeline, or at least for each crontab entry, to manage that job,
collect the output and email the output and results.  It capitalizes it
to easily distinguish those threads/procs from the cron daemon itself.

cron was probably stuck waiting for an open/creat on a temporary file
that was never going to happen because your /tmp was bullocks.

It appears to me that there is some sauce in the way that /tmp/ramdisk
works.  If I'm right, /tmp/ramdisk is used by any process that
creates/utilizes a file on /tmp (or /tmp/ramdisk directly).
If /tmp/ramdisk is not mounted, then that special facility won't be
used and files will be on the flash filesystem, which makes just about
everything considerably slower.

/var/log/messages going away is probably some bug in the /etc/rc script.

Cheers,

a

On Mon, 27 Nov 2006 15:32:50 -0800 "Larry Scheer"
<larry.scheer@onstor.com> wrote:

> Ah, that reminds me... When I did a ps -ax (on system that was missing
> /var/log and /tmp/ramdisk) I saw a process called "CRON" as well as
> "cron" running. I never seen a process "CRON" (all caps) before. I am
> not sure if cron creates a thread or child process called CRON on
> openbsd. It might be worth investigating.
> 
> -----Original Message-----
> From: Sandrine Boulanger 
> Sent: Monday, November 27, 2006 3:25 PM
> To: Raj Kumar; Ken Renshaw; Paul Hammer; John Rogers; Larry Scheer;
> Eric Barrett
> Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk
> 
> Check crontab. I've seen a couple of filers where the first 20 lines
> or so of crontab was missing, and for example messages was not
> recycled anymore.
> 
> -----Original Message-----
> From: Raj Kumar 
> Sent: Monday, November 27, 2006 3:21 PM
> To: Ken Renshaw; Paul Hammer; John Rogers; Larry Scheer; Eric Barrett
> Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk
> 
> I just encountered (rather noticed) a case where /var/log/messages
> file vanished after a reboot and never recreated again, even after
> multiple reboots. Every time syslogd complains about it. 
> 
> PR#16553
> 
> -----Original Message-----
> From: Ken Renshaw 
> Sent: Monday, November 27, 2006 2:39 PM
> To: Raj Kumar; Paul Hammer; John Rogers; Larry Scheer; Eric Barrett
> Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Subject: Re: Had a lab system loose /var/log and /tmp/ramdisk
> 
> Actually, now that you mention it 
> 
>  
> 
> -----Original Message-----
> From: Raj Kumar
> To: Ken Renshaw; Paul Hammer; John Rogers; Larry Scheer; Eric Barrett
> CC: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Sent: Mon Nov 27 14:29:06 2006
> Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk
> 
> Isn't /tmp/ramdisk used by ssssscccc also (GUI's)? Not sure.
> 
> -----Original Message-----
> From: Ken Renshaw 
> Sent: Monday, November 27, 2006 2:26 PM
> To: Paul Hammer; John Rogers; Larry Scheer; Eric Barrett
> Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Subject: Re: Had a lab system loose /var/log and /tmp/ramdisk
> 
> I'm pretty sure that /tmp/ramdisk ONLY gets used as a mount point for
> the memfs during system upgrade. The release tarfiles are un-tarred
> into the ramdisk after the memfs creation is finished.
> 
> I do not believe the directory/mount point gets used at all during
> normal filer operations.
> 
> Just a couple odd data points that may or may not mean anything here.
> 
> -Ken
> 
>  
> 
> -----Original Message-----
> From: Paul Hammer
> To: John Rogers; Larry Scheer; Eric Barrett
> CC: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Sent: Mon Nov 27 14:20:47 2006
> Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk
> 
> Also, thes two directories would be the logical ones to get corrupted
> since we write to them
> 
> ________________________________
> 
> From: John Rogers
> Sent: Mon 11/27/2006 1:23 PM
> To: Larry Scheer; Eric Barrett
> Cc: John VanderWerf; Kevin Matthews; Brian Baker; dl-QA
> Subject: RE: Had a lab system loose /var/log and /tmp/ramdisk
> 
> 
> 
> Although the disappearance of /var/log and /tmp/ramdisk is indictitive
> of being hacked, I don't believe this is the case. We should process
> due diligence even if we just suspect it a little. Kevin can you tell
> us if we've detected any intrusions recently?
> 
> 
> 
> 
> 
> I am more concerned that the crash corrupted the flash card and fsk
> cleaned /var/log beyond recognition.
> 
> _____________________________________________
> From: Larry Scheer
> Sent: Monday, November 27, 2006 11:56 AM
> To: Eric Barrett
> Cc: John Rogers; John VanderWerf
> Subject: Had a lab system loose /var/log and /tmp/ramdisk
> 
> Eric,
> 
>    Many months ago I remember a customer reporting /var/log
> disappearing on them and you mentioned that this is a classic sign
> of  a system being hacked.
> 
> On Wednesday evening I had a filer (a development test system) crash
> because /var/log and /tmp/ramdisk were removed. It could have been a
> failure with the flash or a software bug. I am not sure. 
> 
> Do you recall if there were any other reported problems similar to
> this in the field or what was determined to be the root cause of the
> problem (with /var/log disappearing) reported by the customer?
> 
> Larry
> 
