X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C85FC3.9A69D2FF@onstor-exch02.onstor.net>; Fri, 25 Jan 2008 19:31:49 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C85FC3.9A69D2FF"
Content-class: urn:content-classes:message
Subject: RE: Defects related to dmalloc?
Date: Fri, 25 Jan 2008 19:31:49 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E03E9A57D@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E07E60255@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Defects related to dmalloc?
Thread-Index: AchfwXhiiwR5hTo3QHuWxn4YjvJ5CwAACHsgAABwMyA=
From: "Chris Vandever" <chris.vandever@onstor.com>
To: "Maxim Kozlovsky" <maxim.kozlovsky@onstor.com>,
	"dl-Cougar" <dl-Cougar@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C85FC3.9A69D2FF
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Thanks, Max.  I was just curious since it was complaining about the file
being out of bounds.

ChrisV

_____________________________________________
From: Maxim Kozlovsky=20
Sent: Friday, January 25, 2008 6:19 PM
To: Chris Vandever; dl-Cougar
Subject: RE: Defects related to dmalloc?

This is most certainly an issue with the cluster library and not with
dmalloc.=20

_____________________________________________
From: Chris Vandever=20
Sent: Friday, January 25, 2008 6:17 PM
To: dl-Cougar
Subject: Defects related to dmalloc?

I noticed the following stack trace (see the bold section) and wondered
if it may be an issue with dmalloc.  There are several defects with
similar traces.  It looks to me like perhaps an app (in this case vsd)
was built with dmalloc enabled, but a library it calls (libcluster)
wasn't.  Could someone familiar with dmalloc confirm or deny this
theory?  Thanks.

ChrisV
----------

Defect #21912:

vsd core dumped while system was coming up. below is the GDB trace.
elogs and console logs will be copied to /n/newcorevol/defect_21912
directory. submittal is sub4+ i.e. R4.0.0.DEV-CGDBG-011608 with dmalloc
enabled.

g12r10:/vikas# gdb vsd vsd.core
GNU gdb 6.4.90-debian
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "mipsel-linux-gnu"...Using host libthread_db
library "/lib/libthread_db.so.1".

Reading symbols from /lib/libldap-2.3.so.0...done.
Loaded symbols for /lib/libldap-2.3.so.0
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /onstor/lib/libipmd.so.1.0...done.
Loaded symbols for /onstor/lib/libipmd.so.1.0
Reading symbols from /onstor/lib/libfs.so.1.0...done.
Loaded symbols for /onstor/lib/libfs.so.1.0
Reading symbols from /onstor/lib/libevent.so.1.0...done.
Loaded symbols for /onstor/lib/libevent.so.1.0
Reading symbols from /onstor/lib/libcluster.so.1.0...done.
Loaded symbols for /onstor/lib/libcluster.so.1.0
Reading symbols from /onstor/lib/libncm.so.1.0...done.
Loaded symbols for /onstor/lib/libncm.so.1.0
Reading symbols from /onstor/lib/librmc.so.1.0...done.
Loaded symbols for /onstor/lib/librmc.so.1.0
Reading symbols from /onstor/lib/libanpssc.so.1.0...done.
Loaded symbols for /onstor/lib/libanpssc.so.1.0
Reading symbols from /onstor/lib/libe_log.so.1.0...done.
Loaded symbols for /onstor/lib/libe_log.so.1.0
Reading symbols from /onstor/lib/libgns.so.1.0...done.
Loaded symbols for /onstor/lib/libgns.so.1.0
Reading symbols from /onstor/lib/libutils.so.1.0...done.
Loaded symbols for /onstor/lib/libutils.so.1.0
Reading symbols from /onstor/lib/libauthen.so.1.0...done.
Loaded symbols for /onstor/lib/libauthen.so.1.0
Reading symbols from /onstor/lib/libnfxnis.so.1.0...done.
Loaded symbols for /onstor/lib/libnfxnis.so.1.0
Reading symbols from /onstor/lib/libicucommon.so.1.0...done.
Loaded symbols for /onstor/lib/libicucommon.so.1.0
Reading symbols from /onstor/lib/libicudata.so.1.0...done.
Loaded symbols for /onstor/lib/libicudata.so.1.0
Reading symbols from /onstor/lib/libacrypt.so.1.0...done.
Loaded symbols for /onstor/lib/libacrypt.so.1.0
Reading symbols from /onstor/lib/libtrace.so.1.0...done.
Loaded symbols for /onstor/lib/libtrace.so.1.0
Reading symbols from /onstor/lib/libbsdcompat.so.1.0...done.
Loaded symbols for /onstor/lib/libbsdcompat.so.1.0
Reading symbols from /onstor/lib/libgenlib.so.1.0...done.
Loaded symbols for /onstor/lib/libgenlib.so.1.0
Reading symbols from /onstor/lib/libifconfig.so.1.0...done.
Loaded symbols for /onstor/lib/libifconfig.so.1.0
Reading symbols from /usr/lib/libdmallocth.so.5...done.
Loaded symbols for /usr/lib/libdmallocth.so.5
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/liblber-2.3.so.0...done.
Loaded symbols for /lib/liblber-2.3.so.0
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Core was generated by `/onstor/bin/vsd -r'.
Program terminated with signal 6, Aborted.
#0  0x2b74db04 in kill () from /lib/libc.so.6
(gdb) bt
#0  0x2b74db04 in kill () from /lib/libc.so.6
#1  0x2b74f200 in abort () from /lib/libc.so.6
#2  0x2b614e50 in _dmalloc_die (silent_b=3D<value optimized out>) at
error.c:656
#3  0x2b614f70 in dmalloc_error (func=3D0x2b618fcc
"_dmalloc_chunk_heap_check") at error.c:703
#4  0x2b6120ec in log_error_info (now_file=3D0x0, now_line=3D0,
user_pnt=3D<value optimized out>, slot_p=3D0x2aac2308, =
reason=3D0x2b618fe8
"checking user pointer",
    where=3D0x2b618fcc "_dmalloc_chunk_heap_check") at chunk.c:1157
#5  0x2b6139f4 in _dmalloc_chunk_heap_check () at chunk.c:2229
#6  0x2b616020 in dmalloc_in (file=3D0x0, line=3D0, check_heap_b=3D1) at
malloc.c:565
#7  0x2b616b6c in dmalloc_free (file=3D0x751 <Address 0x751 out of
bounds>, line=3D6, pnt=3D0x2bdae808, func_id=3D0) at malloc.c:967
#8  0x2adeda9c in cluster_atomicUpdateRecord
(recType=3DCLUSDB_REC_TYPE_LOCK, keyLen=3D0, key=3D0x0, =
pRecId=3D0x7fcd7430,
dataLen=3D504, data=3D0x7fcd7218 "\022\006",
    recSize=3D0x7fcd7410, recDataBuf=3D0x7fcd7218 "\022\006") at
cluster-api.c:1512
#9  0x2adf2c48 in cluster_releaseLock (recId=3D12800, cookie=3D0) at
cluster-api.c:3890
#10 0x2ae312ec in cluster_releaseGnsLock (fWrite=3D0 '\0', =
recId=3D12800,
lockCookie=3D0) at cluster-gns-api.c:2075
#11 0x0041ff14 in vsd_createVsRunTime (vsdTxn=3D0x2bc88008) at
vs-daemon.c:6125
#12 0x004153c8 in vsd_enableVsProc (vsdTxn=3D0x2bc88008) at
vs-daemon.c:4046
#13 0x004558f4 in vsd_processTxnRunQueue (maxProcess=3D128) at
vs-util.c:1455
#14 0x0040e1cc in main (argc=3D2, argv=3D0x7fcd7de4) at vs-daemon.c:2734
(gdb) quit


------_=_NextPart_001_01C85FC3.9A69D2FF
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7653.38">
<TITLE>RE: Defects related to dmalloc?</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">Thanks, Max.&nbsp; I was just curious since it was =
complaining about the file</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" =
SIZE=3D2 FACE=3D"Arial"></FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"> <FONT COLOR=3D"#000080" SIZE=3D2 FACE=3D"Arial">being =
out of bounds.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">ChrisV</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Maxim Kozlovsky<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Friday, January 25, 2008 =
6:19 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Chris Vandever; =
dl-Cougar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> RE: Defects related to dmalloc?</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT COLOR=3D"#000080" SIZE=3D2 =
FACE=3D"Arial">This is most certainly an issue with the cluster library =
and not with dmalloc. </FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma">_____________________________________________<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">From:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Chris Vandever<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Sent:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> Friday, January 25, 2008 =
6:17 PM<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">To:</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Tahoma"> dl-Cougar<BR>
</FONT></SPAN><SPAN LANG=3D"en-us"><B></B></SPAN><SPAN =
LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Tahoma">Subject:</FONT></B></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Tahoma"> Defects related to dmalloc?</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I =
noticed the following stack trace (see the</FONT></SPAN><SPAN =
LANG=3D"en-us"><B></B></SPAN><SPAN LANG=3D"en-us"><B> <FONT SIZE=3D2 =
FACE=3D"Arial">bold</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial"> section) and wondered if =
it may be an issue with dmalloc.&nbsp; There are several defects with =
similar traces.&nbsp; It looks to me like perhaps an app (in this case =
vsd) was built with dmalloc enabled, but a library it calls (libcluster) =
wasn&#8217;t.&nbsp; Could someone familiar with dmalloc confirm or deny =
this theory?&nbsp; Thanks.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">ChrisV</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">----------</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Defect #21912:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">vsd =
core dumped while system was coming up. below is the GDB trace. elogs =
and console logs will be copied to /n/newcorevol/defect_21912 directory. =
submittal is sub4+ i.e. R4.0.0.DEV-CGDBG-011608 with dmalloc =
enabled.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">g12r10:/vikas# gdb vsd vsd.core</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">GNU =
gdb 6.4.90-debian</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Copyright (C) 2006 Free Software Foundation, =
Inc.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">GDB =
is free software, covered by the GNU General Public License, and you =
are</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">welcome to change it and/or distribute copies of it under =
certain conditions.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Type =
&quot;show copying&quot; to see the conditions.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">There =
is absolutely no warranty for GDB.&nbsp; Type &quot;show warranty&quot; =
for details.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">This =
GDB was configured as &quot;mipsel-linux-gnu&quot;...Using host =
libthread_db library =
&quot;/lib/libthread_db.so.1&quot;.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libldap-2.3.so.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/lib/libldap-2.3.so.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libm.so.6...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libm.so.6</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libnsl.so.1...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libnsl.so.1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libcrypt.so.1...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libcrypt.so.1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libipmd.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libipmd.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libfs.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libfs.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libevent.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libevent.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libcluster.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libcluster.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libncm.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libncm.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/librmc.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/librmc.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libanpssc.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libanpssc.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libe_log.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libe_log.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libgns.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libgns.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libutils.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libutils.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libauthen.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libauthen.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libnfxnis.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libnfxnis.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libicucommon.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libicucommon.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libicudata.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libicudata.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libacrypt.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libacrypt.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libtrace.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libtrace.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libbsdcompat.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libbsdcompat.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libgenlib.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libgenlib.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libifconfig.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libifconfig.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/usr/lib/libdmallocth.so.5...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/usr/lib/libdmallocth.so.5</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libgcc_s.so.1...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libgcc_s.so.1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libc.so.6...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libc.so.6</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/liblber-2.3.so.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/lib/liblber-2.3.so.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libresolv.so.2...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libresolv.so.2</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/usr/lib/libssl.so.0.9.8...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/usr/lib/libssl.so.0.9.8</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/usr/lib/libcrypto.so.0.9.8...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/usr/lib/libcrypto.so.0.9.8</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/ld.so.1...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/ld.so.1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libdl.so.2...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libdl.so.2</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/usr/lib/libz.so.1...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /usr/lib/libz.so.1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libnss_files.so.2...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/lib/libnss_files.so.2</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Core =
was generated by `/onstor/bin/vsd -r'.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Program terminated with signal 6, =
Aborted.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#0&nbsp; 0x2b74db04 in kill () from =
/lib/libc.so.6</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">(gdb) =
bt</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#0&nbsp; 0x2b74db04 in kill () from =
/lib/libc.so.6</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#1&nbsp; 0x2b74f200 in abort () from =
/lib/libc.so.6</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#2&nbsp; 0x2b614e50 in _dmalloc_die (silent_b=3D&lt;value =
optimized out&gt;) at error.c:656</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#3&nbsp; 0x2b614f70 in dmalloc_error (func=3D0x2b618fcc =
&quot;_dmalloc_chunk_heap_check&quot;) at error.c:703</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#4&nbsp; 0x2b6120ec in log_error_info (now_file=3D0x0, =
now_line=3D0, user_pnt=3D&lt;value optimized out&gt;, =
slot_p=3D0x2aac2308, reason=3D0x2b618fe8 &quot;checking user =
pointer&quot;,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp; where=3D0x2b618fcc =
&quot;_dmalloc_chunk_heap_check&quot;) at chunk.c:1157</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#5&nbsp; 0x2b6139f4 in _dmalloc_chunk_heap_check () at =
chunk.c:2229</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Arial">#6&nbsp; 0x2b616020 in dmalloc_in (file=3D0x0, line=3D0, =
check_heap_b=3D1) at malloc.c:565</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Arial">#7&nbsp; 0x2b616b6c in dmalloc_free (file=3D0x751 =
&lt;Address 0x751 out of bounds&gt;, line=3D6, pnt=3D0x2bdae808, =
func_id=3D0) at malloc.c:967</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Arial">#8&nbsp; 0x2adeda9c in cluster_atomicUpdateRecord =
(recType=3DCLUSDB_REC_TYPE_LOCK, keyLen=3D0, key=3D0x0, =
pRecId=3D0x7fcd7430, dataLen=3D504, data=3D0x7fcd7218 =
&quot;\022\006&quot;,</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp; recSize=3D0x7fcd7410, =
recDataBuf=3D0x7fcd7218 &quot;\022\006&quot;) at =
cluster-api.c:1512</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#9&nbsp; 0x2adf2c48 in cluster_releaseLock =
(recId=3D12800, cookie=3D0) at cluster-api.c:3890</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">#10 =
0x2ae312ec in cluster_releaseGnsLock (fWrite=3D0 '\0', recId=3D12800, =
lockCookie=3D0) at cluster-gns-api.c:2075</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">#11 =
0x0041ff14 in vsd_createVsRunTime (vsdTxn=3D0x2bc88008) at =
vs-daemon.c:6125</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">#12 =
0x004153c8 in vsd_enableVsProc (vsdTxn=3D0x2bc88008) at =
vs-daemon.c:4046</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">#13 =
0x004558f4 in vsd_processTxnRunQueue (maxProcess=3D128) at =
vs-util.c:1455</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">#14 =
0x0040e1cc in main (argc=3D2, argv=3D0x7fcd7de4) at =
vs-daemon.c:2734</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">(gdb) =
quit</FONT></SPAN></P>

</BODY>
</HTML>
------_=_NextPart_001_01C85FC3.9A69D2FF--
