AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20080516175453.003898dc@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<warren.gale@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@onstor-exch02.onstor.net/INBOX	0	BB375AF679D4A34E9CA8DFA650E2B04E09FCF845@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Fri, 16 May 2008 17:56:33 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: "Warren Gale" <warren.gale@onstor.com>
Subject: Re: "dump not yet implemented" message at BSD crash
Message-ID: <20080516175633.3d7852c5@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E09FCF845@onstor-exch02.onstor.net>
References: <BB375AF679D4A34E9CA8DFA650E2B04E09FCF801@onstor-exch02.onstor.net>
	<20080516173239.176272ba@ripper.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E09FCF845@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

The whole puzzle of how do you write out an error message that says
"Out of memory" when the routine that outputs messages requires memory,
has been a difficult one for computer science since forever.

Linux has a little bundle of memory it scoops up at boot time and saves
just for the purpose of outputting a panic message no matter what the
conditions.  Guess what, not even that always works.  OOM, or, Out of
Memory condition is so difficult that often all that happens is the
system gets so constipated that it can't even crash properly.

On Fri, 16 May 2008 17:41:01 -0700 "Warren Gale"
<warren.gale@onstor.com> wrote:

> It all started with, there was a question about 
> I see this message on the console, why is there no "0.0" crash file.
> 
> I like the "if there isn't any, there isn't any"
> 
> Warren
> 
> -----Original Message-----
> From: Andy Sharp 
> Sent: Friday, May 16, 2008 5:33 PM
> To: Warren Gale
> Subject: Re: "dump not yet implemented" message at BSD crash
> 
> What's the question?  It looks like just about all info is there, so I
> don't know what <someone> wants to know.
> 
> A crash file can't always be created: after all, it's created by code
> in the kernel, and if the kernel crashes too badly, well, it ain't
> gonna be able to execute anything.  Being out of memory is a good
> example.  It probably requires memory to write the crash file, and if
> there isn't any, there isn't any.
> 
>  On Fri, 16 May 2008 16:42:18 -0700 "Warren Gale"
> <warren.gale@onstor.com> wrote:
> 
> > Andy,
> >    I thought that I should forward this to you also. 
> >  Know you don't really care about BSD but U the man :-)
> > 
> > Warren
> > 
> > _____________________________________________
> > From: Warren Gale 
> > Sent: Friday, May 16, 2008 4:39 PM
> > To: Shin Irie
> > Cc: Warren Gale
> > Subject: RE: "dump not yet implemented" message at BSD crash
> > 
> > Irie-san,
> > 
> > (please excuse me if I didn't get the greeting correct)
> > 
> > I have verified that if you see this message on the SSC console.
> > 
> > 		diag2280# panic: trap
> > 		Saved crashdump.
> > 
> > 		dumping to dev 401, offset 8
> > 		dump not yet implementedSystem restart.
> > 
> > It will write the data to PROM crash save area, and after a reboot
> > there will be a "0.0" file in /var/crash. (eventually)
> > 
> > I have seen it take up to 8 minutes, before the file shows up in
> > /var/crash. 
> > Seems to always take at least 1 minute before the file "0.0"
> > appears. Don't know why it takes so long.
> > 
> > BSD does not erase the SSC crash info in PROM after it retrieves it
> > and creates the "0.0" file,
> > but does over write the PROM crash save area with the new crash.
> > 
> > I have seen many reboots in my testing that does not produce any
> > error indication.
> > Just re-boots.
> > 
> > I have seen messages like:
> > 		 diag2280#
> > 		  Message from syslogd@diag2280 at Fri May 16
> > 13:37:01 2008 ...
> > 		  diag2280 /bsd: WARNING: system memory is running
> > low -- then reboots --
> > 
> > This does not produce a "0.0" file, nor is there any data saved to
> > PROM crash save area.
> > 
> > 
> > I have seen where chassis d is not responding, then reboots.
> > 
> > 		Message from syslogd@diag2280 at Fri May 16 14:28:25
> > 2008 ...
> > 		diag2280 /bsd: WARNING: system memory is running low
> > 		chassisd not responding...system restart
> > 		System restart.
> > This does not produce a "0.0" file, nor is there any data saved to
> > PROM crash save area either.
> > 
> > 
> > I don't think there is much more I can investigate here.
> > Sandrine also stated in a different e-mail thread,  that BSD does
> > not always produce a crash file.
> >         (RE: Case 8089/ NetOne/ Crashdump registered, but no crash
> > dump saved)
> > 
> > Thanks,
> > Warren
> > 
> > 
> > _____________________________________________
> > From: Warren Gale 
> > Sent: Wednesday, May 14, 2008 10:43 AM
> > To: Shin Irie
> > Subject: RE: "dump not yet implemented" message at BSD crash
> > 
> > HI Irie,
> > 
> > Yes.
> > 
> > I have not been able to make it create crashdump files all the time.
> > ( got 2 files out of 7 attempts)
> > 
> > The "0.0" file that is created is not readable like the crash files
> > from FP/TXRX
> > Here is the wiki link I got. 
> >      You'll find the information here:
> > http://wiki.onstor.net/wiki/Coredump_and_crashdump.
> > 
> > When the message is written to the console, "Saved crashdump.",
> >  it has saved a dump to PROM, but the next or subsequent reboot
> > does not always pull the info from PROM to create a "0.0" file.  
> > Not sure why yet.
> > 
> > Still looking into this.
> > 
> > Warren
> > 
> > 
> > _____________________________________________
> > From: Shin Irie 
> > Sent: Wednesday, May 14, 2008 4:04 AM
> > To: Warren Gale
> > Subject: FW: "dump not yet implemented" message at BSD crash
> > 
> > Warren,
> > 
> > I hear you are looking into this issue. Do you have any idea about
> > this message?
> > 
> > No crashdump was generated after BSD crash, and this appeared on the
> > serial console.
> > 
> > -------------------
> > bc01 panic: trap
> > Saved crashdump.
> > 
> > dumping to dev 401, offset 8
> > dump not yet implementedSystem restart.
> > -------------------
> > 
> > 
> > --
> > Irie
> > 
> > 
> > 
> > ______________________________________________ 
> > From: 	Larry Scheer  
> > Sent:	Saturday, May 10, 2008 8:34 AM
> > To:	Maxim Kozlovsky; Eric Barrett; Shin Irie; Ed Kwan; Jan
> > Seidel; dl-cstech
> > Subject:	RE: "dump not yet implemented" message at BSD crash
> > 
> > Max is right, when I was looking at the code I missed the routine,
> > located in nfx-tree not in openbsd/src, that writes the crash-dump
> > to PROM. The crashdump does get written to PROM. Warren is looking
> > into why /var/crash/0.0 may not be written completely when the
> > system reboots.
> > 
> > Larry
> > _____________________________________________
> > From: Maxim Kozlovsky 
> > Sent: Friday, May 09, 2008 4:16 PM
> > To: Eric Barrett; Larry Scheer; Shin Irie; Ed Kwan; Jan Seidel;
> > dl-cstech
> > Subject: RE: "dump not yet implemented" message at BSD crash
> > 
> > I gave the correct explanation. We save the crash dumps and don't
> > save the core dumps. Both messages are correct. "Saved crashdump"
> > means we saved crashdump without an error as far as we know, "Dump
> > not implemented" means core dump is not implemented.
> > 
> > _____________________________________________
> > From: Eric Barrett 
> > Sent: Friday, May 09, 2008 4:13 PM
> > To: Maxim Kozlovsky; Larry Scheer; Shin Irie; Ed Kwan; Jan Seidel;
> > dl-cstech
> > Subject: RE: "dump not yet implemented" message at BSD crash
> > 
> > But also useful to have a (truthful) explanation when a customer
> > asks!
> > 
> > 
> > _____________________________________________ 
> > From: 	Maxim Kozlovsky  
> > Sent:	Friday, May 09, 2008 4:05 PM
> > To:	Eric Barrett; Larry Scheer; Shin Irie; Ed Kwan; Jan
> > Seidel; dl-cstech
> > Subject:	RE: "dump not yet implemented" message at BSD crash
> > 
> > It is much safer to completely ignore the message than to try to
> > remember which portion of it was right and which was wrong. 
> > 
> > _____________________________________________
> > From: Eric Barrett 
> > Sent: Friday, May 09, 2008 4:01 PM
> > To: Maxim Kozlovsky; Larry Scheer; Shin Irie; Ed Kwan; Jan Seidel;
> > dl-cstech
> > Subject: RE: "dump not yet implemented" message at BSD crash
> > 
> > I think Larry's information about the "dump not implemented" message
> > is still quite informative.
> > 
> > _____________________________________________ 
> > From: 	Maxim Kozlovsky  
> > Sent:	Friday, May 09, 2008 3:59 PM
> > To:	Larry Scheer; Shin Irie; Ed Kwan; Jan Seidel; dl-cstech
> > Subject:	RE: "dump not yet implemented" message at BSD crash
> > 
> > Please disregard Larry's message.
> > 
> > Correct answer is we save BSD crash dumps but don't save the core
> > dumps.
> > 
> > _____________________________________________
> > From: Larry Scheer 
> > Sent: Friday, May 09, 2008 3:51 PM
> > To: Shin Irie; Ed Kwan; Jan Seidel; dl-cstech
> > Subject: RE: "dump not yet implemented" message at BSD crash
> > 
> > The portion of the BSD code that generates this message comes from a
> > routine that intended to write a crashdump to PROM. For some reason
> > lost in antiquity the routine never writes the information to PROM.
> > The routine was never finished in the version of OpenBSD that we use
> > (OpenBSD 2.8.) This is the original MIPS kernel code supplied by
> > OpenBSD foundation.
> > 
> > The message "Saved crashdump" is erroneous. The message "dump not
> > yet implemented" is accurate. The dump is never written and the
> > system reboots.
> > 
> > Ed is correct, we don't save BSD core dumps.
> > 
> > Larry
> > _____________________________________________
> > From: Shin Irie 
> > Sent: Friday, May 09, 2008 3:39 PM
> > To: Ed Kwan; Jan Seidel; dl-cstech
> > Subject: RE: "dump not yet implemented" message at BSD crash
> > 
> > We save BSD crashdump /var/crash/0.0. Looks it is not generated when
> > this message appeared on the serial console.  Do you know why?
> > 
> > 
> > --
> > Irie
> > 
> > 
> > 
> > _____________________________________________ 
> > From: 	Ed Kwan  
> > Sent:	Saturday, May 10, 2008 6:03 AM
> > To:	Jan Seidel; Shin Irie; dl-cstech
> > Subject:	RE: "dump not yet implemented" message at BSD crash
> > 
> > We don't save BSD core dumps.
> > 
> > _____________________________________________
> > From: Jan Seidel 
> > Sent: Friday, May 09, 2008 1:50 PM
> > To: Shin Irie; dl-cstech
> > Subject: RE: "dump not yet implemented" message at BSD crash
> > 
> > Yesterday I saw a very similar error on my filer (cheetah running on
> > sub20), but I don't know where it came from:
> > 
> > 		May  8 14:37:14 eng25 : 0:0:nfxsh:NOTICE: cmd[0]:
> > volume show  : status[0]
> > 		May  8 14:37:32 eng25 : 0:0:snmpd:NOTICE:
> > getEnvInfo: Failed to get PS/Fan info - rc=0 
> > 		panic: malloc: out of space in kmem_map
> > 		Saved crashdump.
> > 
> > 		dumping to dev 411, offset 8
> > 		dump not yet implementedSystem restart.
> > 
> > Regards,
> > Jan
> > 
> > _____________________________________________
> > From: Shin Irie 
> > Sent: Friday, May 09, 2008 1:49 AM
> > To: dl-cstech
> > Subject: "dump not yet implemented" message at BSD crash
> > 
> > I have a customer who saw these console messages when the filer
> > crashed with too many SSH sessions, that is a BSD crash.  No
> > crashdump was generated. Any idea about why no crashdump generated?
> > 
> > -------------------
> > bc01 panic: trap
> > Saved crashdump.
> > 
> > dumping to dev 401, offset 8
> > dump not yet implementedSystem restart.
> > -------------------
> > 
> > 
> > 
> > 
> > --
> > Irie
> > 
