X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7CA17.667A41C7@onstor-exch02.onstor.net>; Thu, 19 Jul 2007 07:13:45 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7CA17.667A41C7"
Content-class: urn:content-classes:message
Subject: FW: BSD crashes: fs_abort
Date: Thu, 19 Jul 2007 07:13:45 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E04936D6D@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: BSD crashes: fs_abort
Thread-Index: AcfJgwweAhdG88wSQI2zIt4zxY5EpQAACQAgAADgfLAAAXgswAAAxxbQAAC79CAAIP1MgA==
From: "Charissa Willard" <charissa.willard@onstor.com>
To: "Andy Sharp" <andy.sharp@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7CA17.667A41C7
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Andy,

=20

When a function calls PANIC or ASSERT does that cause a BSD crash? In
particular, I see a lot of "Panic: fs_abort"s (which is coming from the
FS code) in the 1.2 files in /var/crash.

=20

Thanks,

Charissa

=20

________________________________

From: Jobi Ariyamannil=20
Sent: Wednesday, July 18, 2007 4:24 PM
To: Charissa Willard
Subject: RE: BSD crashes: fs_abort

=20

BSD crashes are not because of filesystem problems.  You should not be
looking at 1.2 crash file when BSD crashes.

1.2 keeps tracks of FP crashes and there should be another crash file
which keeps tracks of BSD crashes.

=20

Regards,

Jobi

=20

________________________________

From: Charissa Willard=20
Sent: Wednesday, July 18, 2007 4:20 PM
To: Jobi Ariyamannil
Subject: RE: BSD crashes: fs_abort

=20

We should be able to add code to distinguish where the panics are being
thrown. This will help us debug the root cause of why the panics are
being thrown. (I found that "failing now" is coming from rmc_util for
RMC_FAIL cases. That's the least helpful message I've found so far.)=20

=20

I got the information for the fs_abort from looking at /var/crash on
eng93, eng62 and g2r5 (see bugs below). These were all in triage this
morning. With regard to the one on g2r5 (number 3 below), Erik was
running a script that may not be a real work flow. I also noticed he has
other crash dumps on his system.=20

=20

1.) TED00019839 - BSD Crash: panic: malloc: out of space in kmem_map
(eng62)

=20

eng62# ls -l

total 213

-rw-r--r--  1 root  wheel  45850 Jul 13 10:05 0.0

-rw-r--r--  1 root  wheel  71241 Jul 13 09:48 1.2

-rw-r--r--  1 root  wheel  77679 Jul 13 10:05 2.0

-rw-r--r--  1 root  wheel  20735 Jul 13 09:55 locks

-rw-r--r--  1 root  wheel      5 Feb  7 21:53 minfree

eng62# more 1.2

"TERMCAP", line 0, col 15, terminal 'vt100': older tic versions may
treat the description field as an alias

=20

crashdump_begin: Thu Jul 12 09:01:19 2007

=20

Crashdump:

----------

Panic : fs_abort

Image Version : NFP_FP : R3.1.0.0BCDBG-070207 : Mon Jul  2 22:33:59 2007

PROM Version  : PROM_SIBYTE_BC : prom-2.0.0 : Mon May 16 15:58:00 2005

Boot Time  : Wed Jul 11 22:06:28 GMT 2007

Crash Time : Thu Jul 12 06:20:57 GMT 2007

=20

=20

=20

2.) TED00019839 - BSD crash followed by FP crash on eng93 two days in a
row (would probably repeat again) (eng93)

=20

# cd /var/crash


# ls -l

total 212

-rw-r--r--  1 root  wheel  18340 Jul 15 19:36 0.0

-rw-r--r--  1 root  wheel  82136 Jul 18 08:58 1.2

-rw-r--r--  1 root  wheel   5220 Jul 17 08:48 locks

-rw-r--r--  1 root  wheel      5 Jun  4 15:54 minfree

# more 1.2

"TERMCAP", line 0, col 15, terminal 'vt100': older tic versions may
treat the description field as an alias

crashdump_begin: Sat Jul 14 18:39:42 2007

=20

Crashdump:

----------

Panic : fs_abort

Image Version : NFP_FP : R3.1.0.0BCDBG-071207 : Thu Jul 12 22:11:17 2007

PROM Version  : PROM_SIBYTE_BC : prom-2.0.2 : Fri Apr  7 13:13:34 2006

Boot Time  : Fri Jul 13 11:09:40 GMT 2007

Crash Time : Fri Jul 13 23:08:41 GMT 2007

Regs  :

=20

=20

3.) TED00019861 - BSD crashed and filer rebooted when TTE tests were
started on TTE filer (g4r5)=20

=20

g2r5# more 1.2

"TERMCAP", line 0, col 15, terminal 'vt100': older tic versions may
treat the description field as an alias

Jul  6 11:00:02 g2r5 newsyslog[2967]: logfile turned over

crashdump_begin: Mon Jul  9 06:56:36 2007

=20

Crashdump:

----------

Panic : fs_abort

Image Version : NFP_FP : R3.1.0.0BCDBG-070207 : Mon Jul  2 22:33:59 2007

PROM Version  : PROM_SIBYTE_BC : prom-2.0.4 : Fri Sep 29 19:04:54 2006

=20

=20

________________________________

From: Jobi Ariyamannil=20
Sent: Wednesday, July 18, 2007 3:41 PM
To: Charissa Willard
Subject: RE: BSD crashes: fs_abort

=20

fs_abort can happen from dump and mirroring code as well, which can be
figured out by looking at the stack.

Do we know the stack traces of the assert hit ?  What are the defect
numbers ?

=20

Regards,

Jobi

=20

________________________________

From: Charissa Willard=20
Sent: Wednesday, July 18, 2007 2:59 PM
To: Jobi Ariyamannil
Subject: RE: BSD crashes: fs_abort

=20

=20

Should these be assigned to you then?

=20

________________________________

From: Jobi Ariyamannil=20
Sent: Wednesday, July 18, 2007 2:33 PM
To: Charissa Willard
Subject: RE: BSD crashes: fs_abort

=20

Yes Charissa.

=20

________________________________

From: Charissa Willard=20
Sent: Wednesday, July 18, 2007 2:32 PM
To: Jobi Ariyamannil
Subject: BSD crashes: fs_abort

=20

Jobi,

=20

John K. is seeing several BSD crashes on his system (eng93). Raj has one
of these, too, on eng62. Does the "Panic: fs_abort" mean that this is
coming from the FS code?

=20

Thanks,

Charissa

=20

# cd /var/crash                                                        =20

# ls -l

total 212

-rw-r--r--  1 root  wheel  18340 Jul 15 19:36 0.0

-rw-r--r--  1 root  wheel  82136 Jul 18 08:58 1.2

-rw-r--r--  1 root  wheel   5220 Jul 17 08:48 locks

-rw-r--r--  1 root  wheel      5 Jun  4 15:54 minfree

# more 1.2

"TERMCAP", line 0, col 15, terminal 'vt100': older tic versions may
treat the description field as an alias

=20

=20

=20

crashdump_begin: Sat Jul 14 18:39:42 2007

=20

=20

Crashdump:

----------

Panic : fs_abort

Image Version : NFP_FP : R3.1.0.0BCDBG-071207 : Thu Jul 12 22:11:17 2007

PROM Version  : PROM_SIBYTE_BC : prom-2.0.2 : Fri Apr  7 13:13:34 2006

Boot Time  : Fri Jul 13 11:09:40 GMT 2007

Crash Time : Fri Jul 13 23:08:41 GMT 2007

Regs  :

=20


------_=_NextPart_001_01C7CA17.667A41C7
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
 namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" =
name=3D"PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:10.0pt;
	font-family:"Courier New";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal;
	font-family:"Courier New";
	color:windowtext;
	font-weight:normal;
	font-style:normal;
	text-decoration:none none;}
span.EmailStyle18
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle19
	{mso-style-type:personal;
	font-family:"Courier New";
	color:windowtext;
	font-weight:normal;
	font-style:normal;
	text-decoration:none none;}
span.EmailStyle20
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle21
	{mso-style-type:personal;
	font-family:"Courier New";
	color:windowtext;
	font-weight:normal;
	font-style:normal;
	text-decoration:none none;}
span.EmailStyle22
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle23
	{mso-style-type:personal-reply;
	font-family:"Courier New";
	color:windowtext;
	font-weight:normal;
	font-style:normal;
	text-decoration:none none;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Andy,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>When
a function calls PANIC or ASSERT does that cause a BSD crash? In =
particular, I
see a lot of &#8220;Panic: fs_abort&#8221;s (which is coming from the FS =
code)
in the 1.2 files in /var/crash.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Thanks,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Charissa<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;font-family:"Times New Roman"'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
face=3DTahoma><span
style=3D'font-family:Tahoma'> <st1:PersonName w:st=3D"on">Jobi =
Ariyamannil</st1:PersonName>
<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Wednesday, July 18, =
2007
4:24 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Charissa
 Willard</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: BSD crashes: =
fs_abort</span></font><font
size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;font-family:"Times New =
Roman"'><o:p></o:p></span></font></p>

</div>

<p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:10.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>BSD crashes are not because of =
filesystem
problems.&nbsp; You should not be looking at 1.2 crash file when BSD =
crashes.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>1.2 keeps tracks of FP crashes and =
there should
be another crash file which keeps tracks of BSD =
crashes.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Regards,<o:p></o:p></span></font></p=
>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Jobi<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;font-family:"Times New Roman"'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
face=3DTahoma><span
style=3D'font-family:Tahoma'> <st1:PersonName w:st=3D"on">Charissa =
Willard</st1:PersonName>
<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Wednesday, July 18, =
2007
4:20 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Jobi
 Ariyamannil</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: BSD crashes: =
fs_abort</span></font><font
size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;font-family:"Times New =
Roman"'><o:p></o:p></span></font></p>

</div>

<p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:10.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:10.0pt'>We
should be able to add code to distinguish where the panics are being =
thrown.
This will help us debug the root cause of why the panics are being =
thrown. (I
found that &#8220;failing now&#8221; is coming from rmc_util for =
RMC_FAIL
cases. That&#8217;s the least helpful message I&#8217;ve found so far.) =
<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:10.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:10.0pt'>I
got the information for the fs_abort from looking at /var/crash on =
eng93, eng62
and g2r5 (see bugs below). These were all in triage this morning. With =
regard
to the one on g2r5 (number 3 below), Erik was running a script that may =
not be
a real work flow. I also noticed he has other crash dumps on his system. =
<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:10.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:10.0pt'>1.)
<u>TED00019839 - BSD Crash: panic: malloc: out of space in
kmem_map&nbsp;(eng62)</u><o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:10.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>eng62#
ls -l<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>total
213<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>-rw-r--r--&nbsp;
1 root&nbsp; wheel&nbsp; 45850 Jul 13 10:05 =
0.0<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>-rw-r--r--&nbsp;
1 root&nbsp; wheel&nbsp; 71241 Jul 13 09:48 =
1.2<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>-rw-r--r--&nbsp;
1 root&nbsp; wheel&nbsp; 77679 Jul 13 10:05 =
2.0<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>-rw-r--r--&nbsp;
1 root&nbsp; wheel&nbsp; 20735 Jul 13 09:55 =
locks<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>-rw-r--r--&nbsp;
1 root&nbsp; wheel&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 5 Feb&nbsp; 7 21:53 =
minfree<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>eng62#
more 1.2<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>&quot;TERMCAP&quot;,
line 0, col 15, terminal 'vt100': older tic versions may treat the =
description
field as an alias<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>crashdump_begin:
Thu Jul 12 09:01:19 2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Crashdump:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>----------<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 color=3Dred face=3D"Courier =
New"><span
style=3D'font-size:9.0pt;color:red'>Panic : =
fs_abort<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Image
Version : NFP_FP : R3.1.0.0BCDBG-070207 : Mon Jul&nbsp; 2 22:33:59 =
2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>PROM
Version&nbsp; : PROM_SIBYTE_BC : prom-2.0.0 : Mon May 16 15:58:00 =
2005<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Boot
Time&nbsp; : Wed Jul 11 22:06:28 GMT 2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Crash
Time : Thu Jul 12 06:20:57 GMT 2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>2.)
<u>TED00019839 - BSD crash followed by FP crash on eng93 two days in a =
row
(would probably repeat again) (eng93)<o:p></o:p></u></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>#
cd
/var/crash&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>#
ls -l<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>total
212<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>-rw-r--r--&nbsp;
1 root&nbsp; wheel&nbsp; 18340 Jul 15 19:36 =
0.0<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>-rw-r--r--&nbsp;
1 root&nbsp; wheel&nbsp; 82136 Jul 18 08:58 =
1.2<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>-rw-r--r--&nbsp;
1 root&nbsp; wheel&nbsp;&nbsp; 5220 Jul 17 08:48 =
locks<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>-rw-r--r--&nbsp;
1 root&nbsp; wheel&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 5 Jun&nbsp; 4 15:54 =
minfree<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>#
more 1.2<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>&quot;TERMCAP&quot;,
line 0, col 15, terminal 'vt100': older tic versions may treat the =
description
field as an alias<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>crashdump_begin:
Sat Jul 14 18:39:42 2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Crashdump:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>----------<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 color=3Dred face=3D"Courier =
New"><span
style=3D'font-size:9.0pt;color:red'>Panic : =
fs_abort<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Image
Version : NFP_FP : R3.1.0.0BCDBG-071207 : Thu Jul 12 22:11:17 =
2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>PROM
Version&nbsp; : PROM_SIBYTE_BC : prom-2.0.2 : Fri Apr&nbsp; 7 13:13:34 =
2006<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Boot
Time&nbsp; : Fri Jul 13 11:09:40 GMT 2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Crash
Time : Fri Jul 13 23:08:41 GMT 2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Regs&nbsp;
:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>3.)
<u>TED00019861 - BSD crashed and filer rebooted when TTE tests were =
started on
TTE filer (g4r5) <o:p></o:p></u></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>g2r5#
more 1.2<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>&quot;TERMCAP&quot;,
line 0, col 15, terminal 'vt100': older tic versions may treat the =
description
field as an alias<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Jul&nbsp;
6 11:00:02 g2r5 newsyslog[2967]: logfile turned =
over<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>crashdump_begin:
Mon Jul&nbsp; 9 06:56:36 2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Crashdump:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>----------<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 color=3Dred face=3D"Courier =
New"><span
style=3D'font-size:9.0pt;color:red'>Panic : =
fs_abort<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Image
Version : NFP_FP : R3.1.0.0BCDBG-070207 : Mon Jul&nbsp; 2 22:33:59 =
2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>PROM
Version&nbsp; : PROM_SIBYTE_BC : prom-2.0.4 : Fri Sep 29 19:04:54 =
2006<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;font-family:"Times New Roman"'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
face=3DTahoma><span
style=3D'font-family:Tahoma'> <st1:PersonName w:st=3D"on">Jobi =
Ariyamannil</st1:PersonName>
<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Wednesday, July 18, =
2007
3:41 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Charissa
 Willard</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: BSD crashes: =
fs_abort</span></font><font
size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;font-family:"Times New =
Roman"'><o:p></o:p></span></font></p>

</div>

<p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:10.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>fs_abort can happen from dump and
mirroring code as well, which can be figured out by looking at the =
stack.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Do we know the stack traces of the =
assert
hit ?&nbsp; What are the defect numbers ?<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Regards,<o:p></o:p></span></font></p=
>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Jobi<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;font-family:"Times New Roman"'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
face=3DTahoma><span
style=3D'font-family:Tahoma'> <st1:PersonName w:st=3D"on">Charissa =
Willard</st1:PersonName>
<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Wednesday, July 18, =
2007
2:59 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Jobi
 Ariyamannil</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: BSD crashes: =
fs_abort</span></font><font
size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;font-family:"Times New =
Roman"'><o:p></o:p></span></font></p>

</div>

<p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:10.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Should
these be assigned to you then?<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;font-family:"Times New Roman"'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
face=3DTahoma><span
style=3D'font-family:Tahoma'> <st1:PersonName w:st=3D"on">Jobi =
Ariyamannil</st1:PersonName>
<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Wednesday, July 18, =
2007 2:33
PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Charissa
 Willard</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: BSD crashes: =
fs_abort</span></font><font
size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;font-family:"Times New =
Roman"'><o:p></o:p></span></font></p>

</div>

<p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:10.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Yes =
Charissa.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;font-family:"Times New Roman"'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
face=3DTahoma><span
style=3D'font-family:Tahoma'> <st1:PersonName w:st=3D"on">Charissa =
Willard</st1:PersonName>
<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Wednesday, July 18, =
2007
2:32 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Jobi
 Ariyamannil</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> BSD crashes: =
fs_abort</span></font><font
size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;font-family:"Times New =
Roman"'><o:p></o:p></span></font></p>

</div>

<p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span =
style=3D'font-size:10.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Jobi,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>John
K. is seeing several BSD crashes on his system (eng93). Raj has one of =
these,
too, on eng62. Does the &#8220;Panic: fs_abort&#8221; mean that this is =
coming
from the FS code?<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Thanks,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Charissa<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>#
cd
/var/crash&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>#
ls -l<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>total
212<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>-rw-r--r--&nbsp;
1 root&nbsp; wheel&nbsp; 18340 Jul 15 19:36 =
0.0<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>-rw-r--r--&nbsp;
1 root&nbsp; wheel&nbsp; 82136 Jul 18 08:58 =
1.2<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>-rw-r--r--&nbsp;
1 root&nbsp; wheel&nbsp;&nbsp; 5220 Jul 17 08:48 =
locks<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>-rw-r--r--&nbsp;
1 root&nbsp; wheel&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 5 Jun&nbsp; 4 15:54 =
minfree<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>#
more 1.2<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>&quot;TERMCAP&quot;,
line 0, col 15, terminal 'vt100': older tic versions may treat the =
description
field as an alias<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>crashdump_begin:
Sat Jul 14 18:39:42 2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Crashdump:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>----------<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Panic
: fs_abort<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Image
Version : NFP_FP : R3.1.0.0BCDBG-071207 : Thu Jul 12 22:11:17 =
2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>PROM
Version&nbsp; : PROM_SIBYTE_BC : prom-2.0.2 : Fri Apr&nbsp; 7 13:13:34 =
2006<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Boot
Time&nbsp; : Fri Jul 13 11:09:40 GMT 2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Crash
Time : Fri Jul 13 23:08:41 GMT 2007<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'>Regs&nbsp;
:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 face=3D"Courier New"><span =
style=3D'font-size:9.0pt'><o:p>&nbsp;</o:p></span></font></p>

</div>

</body>

</html>

------_=_NextPart_001_01C7CA17.667A41C7--
