X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C85FC1.7877A9AB@onstor-exch02.onstor.net>; Fri, 25 Jan 2008 19:16:33 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C85FC1.7877A9AB"
Content-class: urn:content-classes:message
Subject: Defects related to dmalloc?
Date: Fri, 25 Jan 2008 19:16:33 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E03E9A57C@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Defects related to dmalloc?
Thread-Index: AchfwXhiiwR5hTo3QHuWxn4YjvJ5Cw==
From: "Chris Vandever" <chris.vandever@onstor.com>
To: "dl-Cougar" <dl-Cougar@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C85FC1.7877A9AB
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I noticed the following stack trace (see the bold section) and wondered
if it may be an issue with dmalloc.  There are several defects with
similar traces.  It looks to me like perhaps an app (in this case vsd)
was built with dmalloc enabled, but a library it calls (libcluster)
wasn't.  Could someone familiar with dmalloc confirm or deny this
theory?  Thanks.

ChrisV
----------

Defect #21912:

vsd core dumped while system was coming up. below is the GDB trace.
elogs and console logs will be copied to /n/newcorevol/defect_21912
directory. submittal is sub4+ i.e. R4.0.0.DEV-CGDBG-011608 with dmalloc
enabled.

g12r10:/vikas# gdb vsd vsd.core
GNU gdb 6.4.90-debian
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "mipsel-linux-gnu"...Using host libthread_db
library "/lib/libthread_db.so.1".

Reading symbols from /lib/libldap-2.3.so.0...done.
Loaded symbols for /lib/libldap-2.3.so.0
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /onstor/lib/libipmd.so.1.0...done.
Loaded symbols for /onstor/lib/libipmd.so.1.0
Reading symbols from /onstor/lib/libfs.so.1.0...done.
Loaded symbols for /onstor/lib/libfs.so.1.0
Reading symbols from /onstor/lib/libevent.so.1.0...done.
Loaded symbols for /onstor/lib/libevent.so.1.0
Reading symbols from /onstor/lib/libcluster.so.1.0...done.
Loaded symbols for /onstor/lib/libcluster.so.1.0
Reading symbols from /onstor/lib/libncm.so.1.0...done.
Loaded symbols for /onstor/lib/libncm.so.1.0
Reading symbols from /onstor/lib/librmc.so.1.0...done.
Loaded symbols for /onstor/lib/librmc.so.1.0
Reading symbols from /onstor/lib/libanpssc.so.1.0...done.
Loaded symbols for /onstor/lib/libanpssc.so.1.0
Reading symbols from /onstor/lib/libe_log.so.1.0...done.
Loaded symbols for /onstor/lib/libe_log.so.1.0
Reading symbols from /onstor/lib/libgns.so.1.0...done.
Loaded symbols for /onstor/lib/libgns.so.1.0
Reading symbols from /onstor/lib/libutils.so.1.0...done.
Loaded symbols for /onstor/lib/libutils.so.1.0
Reading symbols from /onstor/lib/libauthen.so.1.0...done.
Loaded symbols for /onstor/lib/libauthen.so.1.0
Reading symbols from /onstor/lib/libnfxnis.so.1.0...done.
Loaded symbols for /onstor/lib/libnfxnis.so.1.0
Reading symbols from /onstor/lib/libicucommon.so.1.0...done.
Loaded symbols for /onstor/lib/libicucommon.so.1.0
Reading symbols from /onstor/lib/libicudata.so.1.0...done.
Loaded symbols for /onstor/lib/libicudata.so.1.0
Reading symbols from /onstor/lib/libacrypt.so.1.0...done.
Loaded symbols for /onstor/lib/libacrypt.so.1.0
Reading symbols from /onstor/lib/libtrace.so.1.0...done.
Loaded symbols for /onstor/lib/libtrace.so.1.0
Reading symbols from /onstor/lib/libbsdcompat.so.1.0...done.
Loaded symbols for /onstor/lib/libbsdcompat.so.1.0
Reading symbols from /onstor/lib/libgenlib.so.1.0...done.
Loaded symbols for /onstor/lib/libgenlib.so.1.0
Reading symbols from /onstor/lib/libifconfig.so.1.0...done.
Loaded symbols for /onstor/lib/libifconfig.so.1.0
Reading symbols from /usr/lib/libdmallocth.so.5...done.
Loaded symbols for /usr/lib/libdmallocth.so.5
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/liblber-2.3.so.0...done.
Loaded symbols for /lib/liblber-2.3.so.0
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Core was generated by `/onstor/bin/vsd -r'.
Program terminated with signal 6, Aborted.
#0  0x2b74db04 in kill () from /lib/libc.so.6
(gdb) bt
#0  0x2b74db04 in kill () from /lib/libc.so.6
#1  0x2b74f200 in abort () from /lib/libc.so.6
#2  0x2b614e50 in _dmalloc_die (silent_b=3D<value optimized out>) at
error.c:656
#3  0x2b614f70 in dmalloc_error (func=3D0x2b618fcc
"_dmalloc_chunk_heap_check") at error.c:703
#4  0x2b6120ec in log_error_info (now_file=3D0x0, now_line=3D0,
user_pnt=3D<value optimized out>, slot_p=3D0x2aac2308, =
reason=3D0x2b618fe8
"checking user pointer",
    where=3D0x2b618fcc "_dmalloc_chunk_heap_check") at chunk.c:1157
#5  0x2b6139f4 in _dmalloc_chunk_heap_check () at chunk.c:2229
#6  0x2b616020 in dmalloc_in (file=3D0x0, line=3D0, check_heap_b=3D1) at
malloc.c:565
#7  0x2b616b6c in dmalloc_free (file=3D0x751 <Address 0x751 out of
bounds>, line=3D6, pnt=3D0x2bdae808, func_id=3D0) at malloc.c:967
#8  0x2adeda9c in cluster_atomicUpdateRecord
(recType=3DCLUSDB_REC_TYPE_LOCK, keyLen=3D0, key=3D0x0, =
pRecId=3D0x7fcd7430,
dataLen=3D504, data=3D0x7fcd7218 "\022\006",
    recSize=3D0x7fcd7410, recDataBuf=3D0x7fcd7218 "\022\006") at
cluster-api.c:1512
#9  0x2adf2c48 in cluster_releaseLock (recId=3D12800, cookie=3D0) at
cluster-api.c:3890
#10 0x2ae312ec in cluster_releaseGnsLock (fWrite=3D0 '\0', =
recId=3D12800,
lockCookie=3D0) at cluster-gns-api.c:2075
#11 0x0041ff14 in vsd_createVsRunTime (vsdTxn=3D0x2bc88008) at
vs-daemon.c:6125
#12 0x004153c8 in vsd_enableVsProc (vsdTxn=3D0x2bc88008) at
vs-daemon.c:4046
#13 0x004558f4 in vsd_processTxnRunQueue (maxProcess=3D128) at
vs-util.c:1455
#14 0x0040e1cc in main (argc=3D2, argv=3D0x7fcd7de4) at vs-daemon.c:2734
(gdb) quit


------_=_NextPart_001_01C85FC1.7877A9AB
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7653.38">
<TITLE>Defects related to dmalloc?</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">I =
noticed the following stack trace</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial"> (see the</FONT></SPAN><SPAN =
LANG=3D"en-us"><B></B></SPAN><SPAN LANG=3D"en-us"><B> <FONT SIZE=3D2 =
FACE=3D"Arial">bold</FONT></B></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial"> =
section)</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial"> and =
wondered</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> =
<FONT SIZE=3D2 FACE=3D"Arial">if it may be an issue with dmalloc.&nbsp; =
There are several defects with similar traces.&nbsp; It looks to me like =
perhaps</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"> =
<FONT SIZE=3D2 FACE=3D"Arial">an app (in this case vsd) was built with =
dmalloc enabled, but a library</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial"> it calls</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial"> (libcluster) =
wasn</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT =
SIZE=3D2 FACE=3D"Arial">&#8217;</FONT></SPAN><SPAN =
LANG=3D"en-us"></SPAN><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">t.&nbsp; Could someone familiar with dmalloc confirm or =
deny this the</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">ory?&nbsp; =
Thanks.</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">ChrisV</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">----------</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Defect #21912:</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">vsd =
core dumped while system was coming up. below is the GDB trace. elogs =
and console logs will be copied to /n/newcorevol/defect_21912 directory. =
submittal is sub4+ i.e. R4.0.0.DEV-CGDBG-011608 with dmalloc =
enabled.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">g12r10:/vikas# gdb vsd vsd.core</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">GNU =
gdb 6.4.90-debian</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Copyright (C) 2006 Free Software Foundation, =
Inc.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">GDB =
is free software, covered by the GNU General Public License, and you =
are</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">welcome to change it and/or distribute copies of it under =
certain conditions.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Type =
&quot;show copying&quot; to see the conditions.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">There =
is absolutely no warranty for GDB.&nbsp; Type &quot;show warranty&quot; =
for details.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">This =
GDB was configured as &quot;mipsel-linux-gnu&quot;...Using host =
libthread_db library =
&quot;/lib/libthread_db.so.1&quot;.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libldap-2.3.so.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/lib/libldap-2.3.so.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libm.so.6...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libm.so.6</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libnsl.so.1...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libnsl.so.1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libcrypt.so.1...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libcrypt.so.1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libipmd.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libipmd.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libfs.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libfs.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libevent.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libevent.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libcluster.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libcluster.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libncm.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libncm.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/librmc.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/librmc.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libanpssc.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libanpssc.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libe_log.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libe_log.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libgns.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libgns.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libutils.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libutils.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libauthen.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libauthen.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libnfxnis.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libnfxnis.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libicucommon.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libicucommon.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libicudata.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libicudata.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libacrypt.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libacrypt.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libtrace.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libtrace.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libbsdcompat.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libbsdcompat.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libgenlib.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libgenlib.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/onstor/lib/libifconfig.so.1.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/onstor/lib/libifconfig.so.1.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/usr/lib/libdmallocth.so.5...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/usr/lib/libdmallocth.so.5</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libgcc_s.so.1...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libgcc_s.so.1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libc.so.6...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libc.so.6</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/liblber-2.3.so.0...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/lib/liblber-2.3.so.0</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libresolv.so.2...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libresolv.so.2</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/usr/lib/libssl.so.0.9.8...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/usr/lib/libssl.so.0.9.8</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/usr/lib/libcrypto.so.0.9.8...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/usr/lib/libcrypto.so.0.9.8</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/ld.so.1...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/ld.so.1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libdl.so.2...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /lib/libdl.so.2</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/usr/lib/libz.so.1...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for /usr/lib/libz.so.1</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Reading symbols from =
/lib/libnss_files.so.2...done.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Loaded symbols for =
/lib/libnss_files.so.2</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">Core =
was generated by `/onstor/bin/vsd -r'.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">Program terminated with signal 6, =
Aborted.</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#0&nbsp; 0x2b74db04 in kill () from =
/lib/libc.so.6</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">(gdb) =
bt</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#0&nbsp; 0x2b74db04 in kill () from =
/lib/libc.so.6</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#1&nbsp; 0x2b74f200 in abort () from =
/lib/libc.so.6</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#2&nbsp; 0x2b614e50 in _dmalloc_die (silent_b=3D&lt;value =
optimized out&gt;) at error.c:656</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#3&nbsp; 0x2b614f70 in dmalloc_error (func=3D0x2b618fcc =
&quot;_dmalloc_chunk_heap_check&quot;) at error.c:703</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#4&nbsp; 0x2b6120ec in log_error_info (now_file=3D0x0, =
now_line=3D0, user_pnt=3D&lt;value optimized out&gt;, =
slot_p=3D0x2aac2308, reason=3D0x2b618fe8 &quot;checking user =
pointer&quot;,</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp; where=3D0x2b618fcc =
&quot;_dmalloc_chunk_heap_check&quot;) at chunk.c:1157</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#5&nbsp; 0x2b6139f4 in _dmalloc_chunk_heap_check () at =
chunk.c:2229</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Arial">#6&nbsp; 0x2b616020 in dmalloc_in (file=3D0x0, line=3D0, =
check_heap_b=3D1) at malloc.c:565</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Arial">#7&nbsp; 0x2b616b6c in dmalloc_free (file=3D0x751 =
&lt;Address 0x751 out of bounds&gt;, line=3D6, pnt=3D0x2bdae808, =
func_id=3D0) at malloc.c:967</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Arial">#8&nbsp; 0x2adeda9c in cluster_atomicUpdateRecord =
(recType=3DCLUSDB_REC_TYPE_LOCK, keyLen=3D0, key=3D0x0, =
pRecId=3D0x7fcd7430, dataLen=3D504, data=3D0x7fcd7218 =
&quot;\022\006&quot;,</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><B><FONT SIZE=3D2 =
FACE=3D"Arial">&nbsp;&nbsp;&nbsp; recSize=3D0x7fcd7410, =
recDataBuf=3D0x7fcd7218 &quot;\022\006&quot;) at =
cluster-api.c:1512</FONT></B></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 =
FACE=3D"Arial">#9&nbsp; 0x2adf2c48 in cluster_releaseLock =
(recId=3D12800, cookie=3D0) at cluster-api.c:3890</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">#10 =
0x2ae312ec in cluster_releaseGnsLock (fWrite=3D0 '\0', recId=3D12800, =
lockCookie=3D0) at cluster-gns-api.c:2075</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">#11 =
0x0041ff14 in vsd_createVsRunTime (vsdTxn=3D0x2bc88008) at =
vs-daemon.c:6125</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">#12 =
0x004153c8 in vsd_enableVsProc (vsdTxn=3D0x2bc88008) at =
vs-daemon.c:4046</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">#13 =
0x004558f4 in vsd_processTxnRunQueue (maxProcess=3D128) at =
vs-util.c:1455</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">#14 =
0x0040e1cc in main (argc=3D2, argv=3D0x7fcd7de4) at =
vs-daemon.c:2734</FONT></SPAN></P>

<P ALIGN=3DLEFT><SPAN LANG=3D"en-us"><FONT SIZE=3D2 FACE=3D"Arial">(gdb) =
quit</FONT></SPAN><SPAN LANG=3D"en-us"></SPAN><SPAN =
LANG=3D"en-us"></SPAN></P>

</BODY>
</HTML>
------_=_NextPart_001_01C85FC1.7877A9AB--
