X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7FFBB.3F2A7779@onstor-exch02.onstor.net>; Tue, 25 Sep 2007 13:30:08 -0800
MIME-Version: 1.0
Content-Type: multipart/related;
	type="multipart/alternative";
	boundary="----_=_NextPart_001_01C7FFBB.3F2A7779"
Content-class: urn:content-classes:message
Subject: RE: Lab reproduction for UK cases - RE: British Red Cross Case 5940
Date: Tue, 25 Sep 2007 13:29:58 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E05B469F4@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E05B467CD@onstor-exch02.onstor.net>
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator: 
Thread-Topic: Lab reproduction for UK cases - RE: British Red Cross Case 5940
Thread-Index: Acf+9ejr/Ai3WViXTEqOj01eA2vKQwAg/jkgAAWJgIAAAQgGYAAAOCkAAABwmVQAABemsAAAQyoBAAAWJGAAAEWmoAAARPprAAAqkjAAABxF4AAGOYKw
References: <BB375AF679D4A34E9CA8DFA650E2B04E05B467CD@onstor-exch02.onstor.net>
From: "Tim O'Callaghan" <tim.ocallaghan@onstor.com>
To: "Carolyn Sims" <carolyn.sims@onstor.com>,
	"Huy Duong" <huy.duong@onstor.com>,
	"Neil Cook" <neil.cook@onstor.com>,
	"David Crispin" <david.crispin@onstor.com>,
	"Alan Cooke (Glasshouse)" <acooke@css.glasshouse.com>,
	"Brian Nguyen" <brian.nguyen@onstor.com>,
	"Fay Chong" <fay.chong@onstor.com>,
	"Paul Hammer" <paul.hammer@onstor.com>,
	"Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>,
	"Brian Montero" <brian.montero@onstor.com>,
	"John Rogers" <john.rogers@onstor.com>,
	"Brian Nguyen" <brian.nguyen@onstor.com>,
	"Brian Bliem" <brian.bliem@onstor.com>
Cc: "Sandrine Boulanger" <sandrine.boulanger@onstor.com>,
	"Paul Hammer" <paul.hammer@onstor.com>,
	"Jonathan Goldick" <jonathan.goldick@onstor.com>,
	"Caeli Collins" <caeli.collins@onstor.com>,
	"Gerald Sommariva" <gerald.sommariva@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7FFBB.3F2A7779
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_002_01C7FFBB.3F2A7779"


------_=_NextPart_002_01C7FFBB.3F2A7779
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

The main problem with this is that the symptoms take 5 days to reappear
within in the application.

=20

Seems pretty consistent too - the application reports file throughput
and the numbers seem to tally with the values reported by the port show
loadstat command.

=20

I have 2 cases that I can get detail for - 5940 and 5752 - both have
identical symptoms - slow performance after a period of time.

=20

As previously stated, both systems are providing disk for NFS client -
both Asigra DS-System servers

Both systems are;

Kernel 2.6.9-5.Elsmp

RedHat Enterprise 4.

The volumes are mounted with the following options; nfs
defaults,intr,rsize=3D32768,wsize=3D32768,async,nfsvers=3D3 - as =
specified in=20
http://wiki.onstor.net/wiki/Best_practices_for_NFS
<http://wiki.onstor.net/wiki/Best_practices_for_NFS>  - Backup
technology LAN connection is via dedicated unmanaged GigE switch. BRC
LAN connection is via managed switch.



Please bear in mind that the BRC box does not have any mirror or
snapshot processes defined or scheduled.

Rebooting the Bobcat causes performance to return to normal - also note
that mirror transfer speed on Backup Technology system was degraded to
7MB/s - this returned to 20MB/s after bobcat reboot.

Rebooting Linux client has same effect - returning system performance to
acceptable level. Asigra jobs that were hung/running slowly completed in
minutes after the reboot.

Asigra running on direct attached storage does not exhibit the symptom,
BRC also have an Asigra vault running on (I think) a Sun server as well.

DD timed transfer of 1024MB file - prior to reboot of BRC system=20

1m8.177s

After reboot

0m17.521s

Similar performance gains seen with Backup Technology system after
reboot of Bobcat - this is probably due to a process being restarted
because of NFS disconnect.

An SGA was created for case 5752 earlier today prior to system reboot. I
expected it to have uploaded - I will ftp the files tomorrow am - you
can have SGA from both devices.

I will try to collect all listed kpi stats, client logs etc in the
morning. If there is any other specific info required please email me
with the request.

from Linux client;

Vmstat=20

Top

from BobCat

Evm sh stats

System get all

stats txrx times show all

stats txrx kpi get_counters nfsv3

stats txrx kpi get_summary
=20
filesystem show stats cache

filesystem show stats performance

filesystem show stats resource

I will endeavour to arrange remote access to one or both systems for
engineering staff to facilitate our getting a more rapid resolution to
this issue.=20

Regards

Tim

=20

Tim O'Callaghan
Senior S.E.

ONStor, Inc.
office:  +44 1189 635866
mobile: +44 7767 435472=20

tim.ocallaghan@onstor.com <mailto:tim.ocallaghan@onstor.com>=20
http://www.onstor.com <http://www.onstor.com>=20

=20

=20

________________________________

From: Carolyn Sims=20
Sent: 25 September 2007 18:46
To: Huy Duong; Neil Cook; Tim O'Callaghan; David Crispin; Alan Cooke
(Glasshouse)
Cc: Carolyn Sims
Subject: FW: Lab reproduction for UK cases - RE: British Red Cross Case
5940

=20

More on exactly what and which stats and info

=20

________________________________

From: Ed Kwan=20
Sent: Tuesday, September 25, 2007 10:43 AM
To: Paul Hammer; Carolyn Sims
Cc: Brian Bliem; Brian Nguyen; Jobi Ariyamannil; Caeli Collins
Subject: RE: Lab reproduction for UK cases - RE: British Red Cross Case
5940

=20

Vmstat and top will give us that info.

=20

________________________________

From: Paul Hammer=20
Sent: Tuesday, September 25, 2007 10:37 AM
To: Ed Kwan; Carolyn Sims
Cc: Brian Bliem; Brian Nguyen; Jobi Ariyamannil; Caeli Collins
Subject: RE: Lab reproduction for UK cases - RE: British Red Cross Case
5940

=20

Also can we check out the clients and see if they are running out of
resources (processes and/or memory)?

=20

________________________________

From: Ed Kwan
Sent: Tue 9/25/2007 10:35 AM
To: Carolyn Sims; Paul Hammer
Cc: Brian Bliem; Brian Nguyen; Jobi Ariyamannil; Caeli Collins
Subject: RE: Lab reproduction for UK cases - RE: British Red Cross Case
5940

We need stats from the client (iostat -ktx, vmstat, top, etc), filer
(kpi stats) and the array, all with timestamps.

=20

________________________________

From: Carolyn Sims=20
Sent: Tuesday, September 25, 2007 10:24 AM
To: Paul Hammer; Ed Kwan
Cc: Brian Bliem; Brian Nguyen; Jobi Ariyamannil; Caeli Collins; Carolyn
Sims
Subject: RE: Lab reproduction for UK cases - RE: British Red Cross Case
5940

=20

I see if we can grab stats before UK disappears for their night.

=20

Re Asigra, apparently the Windows / CIFS / Asigra set up is not seeing
the same problems. I would guess that their app's algorithms in re
memory would be the same on both OS platforms. Yes?

=20

________________________________

From: Paul Hammer=20
Sent: Tuesday, September 25, 2007 10:19 AM
To: Ed Kwan; Carolyn Sims
Cc: Brian Bliem; Brian Nguyen; Jobi Ariyamannil
Subject: RE: Lab reproduction for UK cases - RE: British Red Cross Case
5940

=20

Can we get the stats today? Agree that rebooting the linux client to fix
this does not sound like an ONStor problem but more likely a client or
app problem. What does Asigra say about this issue (reboot the client),
could they have a mem leak?

=20

________________________________

From: Ed Kwan
Sent: Tue 9/25/2007 10:18 AM
To: Paul Hammer; Carolyn Sims
Cc: Brian Bliem; Brian Nguyen
Subject: RE: Lab reproduction for UK cases - RE: British Red Cross Case
5940

We have no stats.=20

From the SGA, I noticed a 17 MB spm.core file, so I recommended 3.0.1.x
which has Rendell's memory leak fix.

The description in the defects states "We know that rebooting the bobcat
clears the fault condition for a period, our strategy is to 1st reboot
the client ...".  Now we're saying rebooting the Linux client restores
performance?

=20

=20

________________________________

From: Paul Hammer=20
Sent: Tuesday, September 25, 2007 10:09 AM
To: Ed Kwan; Carolyn Sims
Cc: Brian Bliem; Brian Nguyen
Subject: FW: Lab reproduction for UK cases - RE: British Red Cross Case
5940

=20

Have we collected this info during the slow down? Jobi suggested the
same be collected.

=20

________________________________

From: Maxim Kozlovsky
Sent: Tue 9/25/2007 10:00 AM
To: Paul Hammer; Fay Chong
Subject: RE: Lab reproduction for UK cases - RE: British Red Cross Case
5940

There always is a chance. I can not tell anything based on this
description, they should get more data like traces and KPIs at high
verbosity level, both when the system behaves normally and when they see
the slow down. But since rebooting the Linux system fixes it there is a
good chance that this is Linux running out of memory and not us.

=20

________________________________

From: Paul Hammer=20
Sent: Tuesday, September 25, 2007 9:50 AM
To: Maxim Kozlovsky; Fay Chong
Subject: FW: Lab reproduction for UK cases - RE: British Red Cross Case
5940

=20

Any chance that what they are seeing is the same thing that you are
seeing with newer Linux kernals?

=20

________________________________

From: Carolyn Sims
Sent: Tue 9/25/2007 9:25 AM
To: Rich LaReau; Brian Nguyen
Cc: Gerald Sommariva; Caeli Collins; Neil Cook; David Crispin; Laurence
Armiger; Tim O'Callaghan; Sandrine Boulanger; Paul Hammer; Jonathan
Goldick
Subject: Lab reproduction for UK cases - RE: British Red Cross Case 5940

Hi Brian and Rich,

The slow downs are noticed on Linux / NFS / Asigra  and NOT on Windows /
CIFS / Asigra.

Specific Linux kernel version and the NFS mount options are shown below.

Please make this set up for recreating these slow down problems.

Thanks,

Carolyn

_____________________________________________
From: Tim O'Callaghan
Sent: Tuesday, September 25, 2007 8:43 AM
To: Laurence Armiger
Cc: Carolyn Sims; Gerald Sommariva; Caeli Collins; Neil Cook; David
Crispin
Subject: Re: British Red Cross Case 5940

Laurence,

HELP required.

I have finally managed to get the authority to reboot the Linux
DS-System server in British Red Cross.=20

Earlier today I rebooted the system in Backup Technology and their
performance levels resumed.

Rebooting the linux system has returned performance to its previous
levels as Gerald suggested it might.

All the systems supported by Backup Technology are;

Kernel 2.6.9-5.Elsmp

RedHat Enterprise 4.

The volumes are mounted with the following options; nfs
defaults,intr,rsize=3D32768,wsize=3D32768,async,nfsvers=3D3

Asigra recommend Red Hat Enterprise 4 which includes the 2.6.9-5.Elsmp
kernel.

Any Asigra vault of significance will require an NFS file system, for
the following reasons

1.  PostGreSql is 'better than MS SQL' in stability, efficiency and
cost.

2.  I believe Linux can manage a greater than number of files than W2K3
server.

A typical Asigra 10TB vault will contain > 100 million files. (10
million files per TB)

Backup Technology are now at a crossroad with respect to the ONstor
product - they feel that "since NFS is an open standard - it would never
cause a problem, ONStor should work" Simon has intimated that unless we
can get this fixed he has sold his last Bobcat - his retirement plan is
based on moving the Asigra CIFs user base to NFS .

You will not believe the crap I am now enduring about this - can we put
pressure on the engineering team to get the later kernel supported -
given the Asigra recommendation of RedHat 4 I consider it unrealistic to
think we can persuade Backup Technology to downgrade their DS-System
operating system patch revison level.

To retain any hopes of future deals with these guys we need to get this
fixed ASAP.

Regards

Tim

 http://www.onstor.com/images/footer/onstor_signature.gif

Tim O'Callaghan
Senior S.E.

ONStor, Inc.
office:  +44 1189 635866
mobile: +44 7767 435472

tim.ocallaghan@onstor.com <mailto:tim.ocallaghan@onstor.com>=20
http://www.onstor.com <http://www.onstor.com/>=20
 http://www.onstor.com/images/footer/storage_mag_product_of_year.gif   =20


------_=_NextPart_002_01C7FFBB.3F2A7779
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<title>Lab reproduction for UK cases - RE: British Red Cross Case =
5940</title>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"country-region"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"place"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
font-face=0A=
	{font-family:Tahoma;}
font-face=0A=
	{font-family:Verdana;}
font-face=0A=
	{font-family:Tahoma;}
font-face=0A=
	{font-family:Verdana;}
font-face=0A=
	{font-family:Tahoma;}
font-face=0A=
	{font-family:Verdana;}

 /* Font Definitions */
 @font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
	{font-family:Verdana;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:blue;
	text-decoration:underline;}
p
	{mso-margin-top-alt:auto;
	margin-right:0cm;
	mso-margin-bottom-alt:auto;
	margin-left:0cm;
	font-size:12.0pt;
	font-family:"Times New Roman";}
pre
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:10.0pt;
	font-family:"Courier New";}
tt
	{font-family:"Courier New";}
span.emailstyle18
	{font-family:Arial;
	color:navy;}
span.emailstyle19
	{font-family:Arial;
	color:navy;}
span.emailstyle20
	{font-family:Arial;
	color:navy;}
span.emailstyle21
	{font-family:Arial;
	color:navy;}
span.EmailStyle22
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle23
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle24
	{mso-style-type:personal-reply;
	font-family:Arial;
	color:navy;}
@page Section1
	{size:612.0pt 792.0pt;
	margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1027" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-GB link=3Dblue vlink=3Dblue>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 color=3D"#666699" face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial;color:#666699'>The main =
problem with
this is that the symptoms take 5 days to reappear within in the =
application.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3D"#666699" face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial;color:#666699'><o:p>&nbsp;</o=
:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3D"#666699" face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial;color:#666699'>Seems pretty
consistent too &#8211; the application reports file throughput and the =
numbers seem
to tally with the values reported by the port show loadstat =
command.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3D"#666699" face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial;color:#666699'><o:p>&nbsp;</o=
:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3D"#666699" face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial;color:#666699'>I have 2 =
cases that I
can get detail for - 5940 and 5752 &#8211; both have identical symptoms =
&#8211;
slow performance after a period of time.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3D"#666699" face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial;color:#666699'><o:p>&nbsp;</o=
:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3D"#666699" face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial;color:#666699'>As previously =
stated, both
systems are providing disk for NFS client &#8211; both Asigra DS-System =
servers<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3D"#666699" face=3DArial><span
style=3D'font-size:10.0pt;font-family:Arial;color:#666699'>Both systems =
are;<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>Kernel =
2.6.9-5.Elsmp<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>RedHat Enterprise =
4.<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>The volumes are mounted with the
following options; nfs =
defaults,intr,rsize=3D32768,wsize=3D32768,async,nfsvers=3D3 &#8211;
as specified in <a =
href=3D"http://wiki.onstor.net/wiki/Best_practices_for_NFS"><font
color=3D"#666699"><span =
style=3D'color:#666699'>http://wiki.onstor.net/wiki/Best_practices_for_NF=
S</span></font></a>
- Backup technology LAN connection is via dedicated unmanaged GigE =
switch. BRC
LAN connection is via managed switch.<br>
<br>
<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>Please bear in mind that the BRC =
box
does not have any mirror or snapshot processes defined or =
scheduled.<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>Rebooting the Bobcat causes =
performance
to return to normal &#8211; also note that mirror transfer speed on =
Backup
Technology system was degraded to 7MB/s &#8211; this returned to 20MB/s =
after
bobcat reboot.<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>Rebooting Linux client has same =
effect &#8211;
returning system performance to acceptable level. Asigra jobs that were
hung/running slowly completed in minutes after the =
reboot.<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>Asigra running on direct =
attached
storage does not exhibit the symptom, BRC also have an Asigra vault =
running on (I
think) a Sun server as well.<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>DD timed transfer of 1024MB file =
&#8211;
prior to reboot of BRC system <o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>1m8.177s<o:p></o:p></span></font>=
</p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>After =
reboot<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>0m17.521s<o:p></o:p></span></font=
></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>Similar performance gains seen =
with
Backup Technology system after reboot of Bobcat &#8211; this is probably =
due to
a process being restarted because of NFS =
disconnect.<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>An SGA was created for case 5752
earlier today prior to system reboot. I expected it to have uploaded =
&#8211; I
will ftp the files tomorrow am &#8211; you can have SGA from both =
devices.<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>I will try to collect all listed =
kpi
stats, client logs etc in the morning. If there is any other specific =
info
required please email me with the request.<o:p></o:p></span></font></p>

<p><font size=3D2 color=3Dblue face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:blue'>from Linux =
client;<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#003300" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#003300'>Vmstat =
<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#003300" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#003300'>Top<o:p></o:p></span></font></p>

<p><font size=3D2 color=3Dblue face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:blue'>from BobCat<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#003300" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#003300'>Evm sh =
stats<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#003300" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#003300'>System get =
all<o:p></o:p></span></font></p>

<p><tt><font size=3D2 color=3D"#003300" face=3DArial><span lang=3DEN =
style=3D'font-size:
10.0pt;font-family:Arial;color:#003300'>stats txrx times show =
all<o:p></o:p></span></font></tt></p>

<p><font size=3D2 color=3D"#003300" face=3DArial><span lang=3DEN =
style=3D'font-size:10.0pt;
font-family:Arial;color:#003300'>stats txrx kpi get_counters =
nfsv3<o:p></o:p></span></font></p>

<pre><font size=3D2 color=3D"#003300" face=3DArial><span lang=3DEN =
style=3D'font-size:
10.0pt;font-family:Arial;color:#003300'>stats txrx kpi =
get_summary<o:p></o:p></span></font></pre><pre><font
size=3D2 color=3D"#003300" face=3DArial><span lang=3DEN =
style=3D'font-size:10.0pt;
font-family:Arial;color:#003300'><o:p>&nbsp;</o:p></span></font></pre><pr=
e><font
size=3D2 color=3D"#003300" face=3DArial><span lang=3DEN =
style=3D'font-size:10.0pt;
font-family:Arial;color:#003300'>filesystem show stats =
cache<o:p></o:p></span></font></pre>

<p><font size=3D2 color=3D"#003300" face=3DArial><span lang=3DEN =
style=3D'font-size:10.0pt;
font-family:Arial;color:#003300'>filesystem show stats =
performance<o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#003300" face=3DArial><span lang=3DEN =
style=3D'font-size:10.0pt;
font-family:Arial;color:#003300'>filesystem show stats =
resource</span></font><font
size=3D2 color=3D"#003300" face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:#003300'><o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>I will endeavour to arrange =
remote
access to one or both systems for engineering staff to facilitate our =
getting a
more rapid resolution to this issue. <o:p></o:p></span></font></p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>Regards<o:p></o:p></span></font><=
/p>

<p><font size=3D2 color=3D"#666699" face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial;color:#666699'>Tim<o:p></o:p></span></font></p>

<div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><font size=3D3 =
color=3D"#666699"
face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;color:#666699'><img
border=3D0 width=3D332 height=3D38 id=3D"_x0000_i1036"
src=3D"cid:image001.gif@01C7FFC3.9A9B67D0"><o:p></o:p></span></font></p>

</div>

<table class=3DMsoNormalTable border=3D0 cellspacing=3D0 cellpadding=3D0 =
width=3D332
 style=3D'width:249.0pt'>
 <tr>
  <td width=3D150 valign=3Dtop style=3D'width:112.5pt;padding:0cm 0cm =
0cm 0cm'>
  <div>
  <p class=3DMsoNormal><strong><b><font size=3D1 color=3Dnavy =
face=3DVerdana><span
  style=3D'font-size:7.5pt;font-family:Verdana;color:navy'>Tim =
O'Callaghan</span></font></b></strong><font
  size=3D1 color=3Dnavy face=3DVerdana><span =
style=3D'font-size:7.5pt;font-family:Verdana;
  color:navy'><br>
  Senior S.E.<br>
  <br>
  ONStor, Inc.<br>
  <b><span style=3D'font-weight:bold'>office:</span></b>&nbsp; +44 1189 =
635866<br>
  <b><span style=3D'font-weight:bold'>mobile:</span></b> +44 7767 435472 =
<o:p></o:p></span></font></p>
  </div>
  <p class=3DMsoNormal><font size=3D1 color=3Dnavy face=3DVerdana><span
  style=3D'font-size:7.5pt;font-family:Verdana;color:navy'><a
  href=3D"mailto:tim.ocallaghan@onstor.com"><b><font =
color=3D"#005c8f"><span
  =
style=3D'color:#005C8F;font-weight:bold;text-decoration:none'>tim.ocallag=
han@onstor.com</span></font></b></a><br>
  <a href=3D"http://www.onstor.com"><b><font color=3D"#005c8f"><span
  =
style=3D'color:#005C8F;font-weight:bold;text-decoration:none'>http://www.=
onstor.com</span></font></b></a></span></font><font
  color=3Dnavy><span style=3D'color:navy'><o:p></o:p></span></font></p>
  </td>
  <td width=3D150 valign=3Dtop style=3D'width:112.5pt;padding:0cm 0cm =
0cm 0cm'>
  <p class=3DMsoNormal><!--[if gte vml 1]><v:shapetype id=3D"_x0000_t75" =

   coordsize=3D"21600,21600" o:spt=3D"75" o:preferrelative=3D"t" =
path=3D"m@4@5l@4@11@9@11@9@5xe"=20
   filled=3D"f" stroked=3D"f">
   <v:stroke joinstyle=3D"miter" />
   <v:formulas>
    <v:f eqn=3D"if lineDrawn pixelLineWidth 0" />
    <v:f eqn=3D"sum @0 1 0" />
    <v:f eqn=3D"sum 0 0 @1" />
    <v:f eqn=3D"prod @2 1 2" />
    <v:f eqn=3D"prod @3 21600 pixelWidth" />
    <v:f eqn=3D"prod @3 21600 pixelHeight" />
    <v:f eqn=3D"sum @0 0 1" />
    <v:f eqn=3D"prod @6 1 2" />
    <v:f eqn=3D"prod @7 21600 pixelWidth" />
    <v:f eqn=3D"sum @8 21600 0" />
    <v:f eqn=3D"prod @7 21600 pixelHeight" />
    <v:f eqn=3D"sum @10 21600 0" />
   </v:formulas>
   <v:path o:extrusionok=3D"f" gradientshapeok=3D"t" =
o:connecttype=3D"rect" />
   <o:lock v:ext=3D"edit" aspectratio=3D"t" />
  </v:shapetype><v:shape id=3D"_x0000_s1026" type=3D"#_x0000_t75" =
alt=3D"" style=3D'position:absolute;
   =
margin-left:38.75pt;margin-top:0;width:78.75pt;height:99.75pt;z-index:1;
   =
mso-wrap-distance-left:0;mso-wrap-distance-top:0;mso-wrap-distance-right:=
0;
   mso-wrap-distance-bottom:0;mso-position-horizontal:right;
   =
mso-position-horizontal-relative:text;mso-position-vertical-relative:line=
'=20
   o:allowoverlap=3D"f">
   <v:imagedata src=3D"cid:image002.gif@01C7FFC3.9A9B67D0" =
o:href=3D"http://www.onstor.com/images/footer/storage_mag_product_of_year=
.gif" />
   <w:wrap type=3D"square"/>
  </v:shape><![endif]--><![if !vml]><img width=3D105 height=3D133
  src=3D"cid:image002.gif@01C7FFC3.9A9B67D0" align=3Dright =
v:shapes=3D"_x0000_s1026"><![endif]><font
  size=3D3 color=3Dnavy face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt;
  color:navy'><o:p></o:p></span></font></p>
  </td>
 </tr>
</table>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span lang=3DEN-US style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma'>
Carolyn Sims <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> 25 September 2007 =
18:46<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Huy Duong; Neil Cook; =
Tim
O'Callaghan; David Crispin; Alan Cooke (Glasshouse)<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Carolyn Sims<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> FW: Lab =
reproduction for <st1:country-region
w:st=3D"on"><st1:place w:st=3D"on">UK</st1:place></st1:country-region> =
cases - RE:
British Red Cross Case 5940</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:navy'>More on exactly =
what and
which stats and info<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p>=
</span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span lang=3DEN-US style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma'>
<st1:PersonName w:st=3D"on">Ed Kwan</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, September =
25, 2007
10:43 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Paul Hammer; Carolyn =
Sims<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Brian Bliem; Brian =
Nguyen; <st1:PersonName
w:st=3D"on">Jobi Ariyamannil</st1:PersonName>; <st1:PersonName =
w:st=3D"on">Caeli
 Collins</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: Lab =
reproduction for <st1:place
w:st=3D"on"><st1:country-region =
w:st=3D"on">UK</st1:country-region></st1:place>
cases - RE: British Red Cross Case 5940</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:navy'>Vmstat and top =
will give
us that info.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:navy'>&nbsp;<o:p></o:p>=
</span></font></p>

<div style=3D'border:none;border-left:solid blue 1.5pt;padding:0cm 0cm =
0cm 4.0pt'>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span lang=3DEN-US style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma'>
<st1:PersonName w:st=3D"on">Paul Hammer</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, September =
25, 2007
10:37 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Ed
 Kwan</st1:PersonName>; <st1:PersonName w:st=3D"on">Carolyn =
Sims</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Brian Bliem; Brian =
Nguyen; <st1:PersonName
w:st=3D"on">Jobi Ariyamannil</st1:PersonName>; <st1:PersonName =
w:st=3D"on">Caeli
 Collins</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: Lab =
reproduction for <st1:place
w:st=3D"on"><st1:country-region =
w:st=3D"on">UK</st1:country-region></st1:place>
cases - RE: British Red Cross Case 5940</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div id=3DidOWAReplyText90206>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dblack face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:black'>Also can we =
check out
the clients and see if they are running out of resources (processes =
and/or
memory)?</span></font><span lang=3DEN-US><o:p></o:p></span></p>

</div>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span lang=3DEN-US style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma'>
<st1:PersonName w:st=3D"on">Ed Kwan</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tue 9/25/2007 10:35 =
AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Carolyn
 Sims</st1:PersonName>; <st1:PersonName w:st=3D"on">Paul =
Hammer</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Brian Bliem; Brian =
Nguyen; <st1:PersonName
w:st=3D"on">Jobi Ariyamannil</st1:PersonName>; <st1:PersonName =
w:st=3D"on">Caeli
 Collins</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: Lab =
reproduction for <st1:place
w:st=3D"on"><st1:country-region =
w:st=3D"on">UK</st1:country-region></st1:place>
cases - RE: British Red Cross Case 5940</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:navy'>We need stats =
from the
client (iostat &#8211;ktx, vmstat, top, etc), filer (kpi stats) and the =
array,
all with timestamps.</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<div style=3D'border:none;border-left:solid blue 1.5pt;padding:0cm 0cm =
0cm 4.0pt'>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span lang=3DEN-US style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma'>
<st1:PersonName w:st=3D"on">Carolyn Sims</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, September =
25, 2007
10:24 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Paul
 Hammer</st1:PersonName>; <st1:PersonName w:st=3D"on">Ed =
Kwan</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Brian Bliem; Brian =
Nguyen; <st1:PersonName
w:st=3D"on">Jobi Ariyamannil</st1:PersonName>; <st1:PersonName =
w:st=3D"on">Caeli
 Collins</st1:PersonName>; <st1:PersonName w:st=3D"on">Carolyn =
Sims</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: Lab =
reproduction for <st1:place
w:st=3D"on"><st1:country-region =
w:st=3D"on">UK</st1:country-region></st1:place>
cases - RE: British Red Cross Case 5940</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:navy'>I see if we can =
grab
stats before <st1:place w:st=3D"on"><st1:country-region =
w:st=3D"on">UK</st1:country-region></st1:place>
disappears for their night.</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:navy'>Re Asigra, =
apparently the
Windows / CIFS / Asigra set up is not seeing the same problems. I would =
guess
that their app&#8217;s algorithms in re memory would be the same on both =
OS
platforms. Yes?</span></font><span lang=3DEN-US><o:p></o:p></span></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span lang=3DEN-US style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma'>
<st1:PersonName w:st=3D"on">Paul Hammer</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, September =
25, 2007
10:19 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Ed
 Kwan</st1:PersonName>; <st1:PersonName w:st=3D"on">Carolyn =
Sims</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Brian Bliem; Brian =
Nguyen; <st1:PersonName
w:st=3D"on">Jobi Ariyamannil</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: Lab =
reproduction for <st1:place
w:st=3D"on"><st1:country-region =
w:st=3D"on">UK</st1:country-region></st1:place>
cases - RE: British Red Cross Case 5940</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<div id=3DidOWAReplyText96623>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dblack face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:black'>Can we get the =
stats
today? Agree that rebooting the linux client to fix this does not sound =
like an
ONStor problem but more likely a client or app problem. What does Asigra =
say
about this issue (reboot the client), could they have a mem =
leak?</span></font><span
lang=3DEN-US><o:p></o:p></span></p>

</div>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span lang=3DEN-US style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma'>
<st1:PersonName w:st=3D"on">Ed Kwan</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tue 9/25/2007 10:18 =
AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Paul
 Hammer</st1:PersonName>; <st1:PersonName w:st=3D"on">Carolyn =
Sims</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Brian Bliem; Brian =
Nguyen<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: Lab =
reproduction for <st1:place
w:st=3D"on"><st1:country-region =
w:st=3D"on">UK</st1:country-region></st1:place>
cases - RE: British Red Cross Case 5940</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:navy'>We have no =
stats. </span></font><span
lang=3DEN-US><o:p></o:p></span></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:navy'>From the SGA, I =
noticed a
17 MB spm.core file, so I recommended 3.0.1.x which has Rendell&#8217;s =
memory
leak fix.</span></font><span lang=3DEN-US><o:p></o:p></span></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:navy'>The description =
in the
defects states &#8220;We know that <b><span =
style=3D'font-weight:bold'>rebooting
the bobcat</span></b> clears the fault condition for a period, our =
strategy is
to 1st reboot the client &#8230;&#8221;.&nbsp; Now we&#8217;re saying =
rebooting
the Linux client restores performance?</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<div style=3D'border:none;border-left:solid blue 1.5pt;padding:0cm 0cm =
0cm 4.0pt'>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span lang=3DEN-US style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma'>
<st1:PersonName w:st=3D"on">Paul Hammer</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, September =
25, 2007
10:09 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Ed
 Kwan</st1:PersonName>; <st1:PersonName w:st=3D"on">Carolyn =
Sims</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Brian Bliem; Brian =
Nguyen<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> FW: Lab =
reproduction for <st1:place
w:st=3D"on"><st1:country-region =
w:st=3D"on">UK</st1:country-region></st1:place>
cases - RE: British Red Cross Case 5940</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<div id=3DidOWAReplyText41329>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dblack face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:black'>Have we =
collected this
info during the slow down? Jobi suggested the same be =
collected.</span></font><span
lang=3DEN-US><o:p></o:p></span></p>

</div>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span lang=3DEN-US style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma'>
<st1:PersonName w:st=3D"on">Maxim Kozlovsky</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tue 9/25/2007 10:00 =
AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Paul
 Hammer</st1:PersonName>; <st1:PersonName w:st=3D"on">Fay =
Chong</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: Lab =
reproduction for <st1:place
w:st=3D"on"><st1:country-region =
w:st=3D"on">UK</st1:country-region></st1:place>
cases - RE: British Red Cross Case 5940</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:navy'>There always is =
a chance.
I can not tell anything based on this description, they should get more =
data
like traces and KPIs at high verbosity level, both when the system =
behaves
normally and when they see the slow down. But since rebooting the Linux =
system
fixes it there is a good chance that this is Linux running out of memory =
and
not us.</span></font><span lang=3DEN-US><o:p></o:p></span></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<div style=3D'border:none;border-left:solid blue 1.5pt;padding:0cm 0cm =
0cm 4.0pt'>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span lang=3DEN-US style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma'>
<st1:PersonName w:st=3D"on">Paul Hammer</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, September =
25, 2007
9:50 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Maxim
 Kozlovsky</st1:PersonName>; <st1:PersonName w:st=3D"on">Fay =
Chong</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> FW: Lab =
reproduction for <st1:place
w:st=3D"on"><st1:country-region =
w:st=3D"on">UK</st1:country-region></st1:place>
cases - RE: British Red Cross Case 5940</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<div id=3DidOWAReplyText3282>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dblack face=3DArial><span =
lang=3DEN-US
style=3D'font-size:10.0pt;font-family:Arial;color:black'>Any chance that =
what
they are seeing is the same thing that you are seeing with newer Linux =
kernals?</span></font><span
lang=3DEN-US><o:p></o:p></span></p>

</div>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
lang=3DEN-US
style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span lang=3DEN-US style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span lang=3DEN-US =
style=3D'font-size:10.0pt;font-family:Tahoma'>
<st1:PersonName w:st=3D"on">Carolyn Sims</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tue 9/25/2007 9:25 =
AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Rich
 LaReau</st1:PersonName>; Brian Nguyen<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Gerald
 Sommariva</st1:PersonName>; <st1:PersonName w:st=3D"on">Caeli =
Collins</st1:PersonName>;
<st1:PersonName w:st=3D"on">Neil Cook</st1:PersonName>; <st1:PersonName =
w:st=3D"on">David
 Crispin</st1:PersonName>; <st1:PersonName w:st=3D"on">Laurence =
Armiger</st1:PersonName>;
<st1:PersonName w:st=3D"on">Tim O'Callaghan</st1:PersonName>; =
<st1:PersonName
w:st=3D"on">Sandrine Boulanger</st1:PersonName>; <st1:PersonName =
w:st=3D"on">Paul
 Hammer</st1:PersonName>; <st1:PersonName w:st=3D"on">Jonathan =
Goldick</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> Lab reproduction =
for <st1:place
w:st=3D"on"><st1:country-region =
w:st=3D"on">UK</st1:country-region></st1:place>
cases - RE: British Red Cross Case 5940</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

</div>

<div>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Hi Brian and Rich,</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>The slow downs are noticed on Linux / NFS =
/
Asigra&nbsp; and NOT on Windows / CIFS / Asigra.</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Specific Linux kernel version and the NFS =
mount
options are shown below.</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Please make this set up for recreating =
these slow
down problems.</span></font><span lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Thanks,</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Carolyn</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 face=3DTahoma><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Tahoma'>_____________________________________________<br>
<b><span style=3D'font-weight:bold'>From:</span></b> <st1:PersonName =
w:st=3D"on">Tim
 O'Callaghan</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, September =
25, 2007
8:43 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">Laurence
 Armiger</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> <st1:PersonName =
w:st=3D"on">Carolyn
 Sims</st1:PersonName>; <st1:PersonName w:st=3D"on">Gerald =
Sommariva</st1:PersonName>;
<st1:PersonName w:st=3D"on">Caeli Collins</st1:PersonName>; =
<st1:PersonName
w:st=3D"on">Neil Cook</st1:PersonName>; <st1:PersonName =
w:st=3D"on">David Crispin</st1:PersonName><br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> Re: British Red =
Cross
Case 5940</span></font><span lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Laurence,</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>HELP required.</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>I have finally managed to get the =
authority to
reboot the Linux DS-System server in British Red Cross. =
</span></font><span
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Earlier today I rebooted the system in =
Backup
Technology and their performance levels resumed.</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Rebooting the linux system has returned
performance to its previous levels as Gerald suggested it =
might.</span></font><span
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>All the systems supported by Backup =
Technology
are;</span></font><span lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Kernel 2.6.9-5.Elsmp</span></font><span
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>RedHat Enterprise 4.</span></font><span
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>The volumes are mounted with the following
options; nfs =
defaults,intr,rsize=3D32768,wsize=3D32768,async,nfsvers=3D3</span></font>=
<span
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Asigra recommend Red Hat Enterprise 4 =
which
includes the 2.6.9-5.Elsmp kernel.</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Any Asigra vault of significance will =
require an
NFS file system, for the following reasons</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>1.&nbsp; PostGreSql is &#8216;better than =
MS
SQL&#8217; in stability, efficiency and cost.</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>2.&nbsp; I believe Linux can manage a =
greater
than number of files than W2K3 server.</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>A typical Asigra 10TB vault will contain =
&gt; 100
million files. (10 million files per TB)</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Backup Technology are now at a crossroad =
with
respect to the ONstor product &#8211; they feel that &#8220;since NFS is =
an
open standard &#8211; it would never cause a problem, ONStor should =
work&#8221;
Simon has intimated that unless we can get this fixed he has sold his =
last
Bobcat &#8211; his retirement plan is based on moving the Asigra CIFs =
user base
to NFS .</span></font><span lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>You will not believe the crap I am now =
enduring
about this &#8211; can we put pressure on the engineering team to get =
the later
kernel supported &#8211; given the Asigra recommendation of RedHat 4 I =
consider
it unrealistic to think we can persuade Backup Technology to downgrade =
their
DS-System operating system patch revison level.</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>To retain any hopes of future deals with =
these
guys we need to get this fixed ASAP.</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Regards</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D2 color=3Dnavy face=3DArial><span lang=3DEN-US =
style=3D'font-size:10.0pt;
font-family:Arial;color:navy'>Tim</span></font><span =
lang=3DEN-US><o:p></o:p></span></p>

<p style=3D'margin-bottom:12.0pt'><a name=3D""></a><font size=3D3
face=3D"Times New Roman"><span lang=3DEN-US =
style=3D'font-size:12.0pt'>&nbsp;<a
href=3D"http://www.onstor.com/images/footer/onstor_signature.gif">http://=
www.onstor.com/images/footer/onstor_signature.gif</a><o:p></o:p></span></=
font></p>

<p><st1:PersonName w:st=3D"on"><b><font size=3D1 color=3Dnavy =
face=3DVerdana><span
 lang=3DEN-US =
style=3D'font-size:7.5pt;font-family:Verdana;color:navy;font-weight:
 bold'>Tim O'Callaghan</span></font></b></st1:PersonName><span =
lang=3DEN-US><br>
</span><font size=3D1 color=3Dnavy face=3DVerdana><span lang=3DEN-US =
style=3D'font-size:
7.5pt;font-family:Verdana;color:navy'>Senior S.E.<br>
<br>
ONStor, Inc.<br>
<b><span style=3D'font-weight:bold'>office:</span></b>&nbsp; +44 1189 =
635866<br>
<b><span style=3D'font-weight:bold'>mobile:</span></b> +44 7767 =
435472</span></font><span
lang=3DEN-US><o:p></o:p></span></p>

<p><font size=3D3 face=3D"Times New Roman"><span lang=3DEN-US =
style=3D'font-size:12.0pt'><a
href=3D"mailto:tim.ocallaghan@onstor.com"><font size=3D1 =
face=3DVerdana><span
style=3D'font-size:7.5pt;font-family:Verdana'>tim.ocallaghan@onstor.com</=
span></font></a><br>
<a href=3D"http://www.onstor.com/"><font size=3D1 face=3DVerdana><span
style=3D'font-size:7.5pt;font-family:Verdana'>http://www.onstor.com</span=
></font></a><br>
&nbsp;<a
href=3D"http://www.onstor.com/images/footer/storage_mag_product_of_year.g=
if">http://www.onstor.com/images/footer/storage_mag_product_of_year.gif</=
a>&nbsp;&nbsp;&nbsp;
<o:p></o:p></span></font></p>

</div>

</div>

</div>

</div>

</div>

</div>

</div>

</div>

</div>

</body>

</html>

------_=_NextPart_002_01C7FFBB.3F2A7779--

------_=_NextPart_001_01C7FFBB.3F2A7779
Content-Type: image/gif;
	name="image001.gif"
Content-Transfer-Encoding: base64
Content-ID: <image001.gif@01C7FFC3.9A9B67D0>
Content-Description: image001.gif
Content-Location: image001.gif

R0lGODlhTAEmAMQGAABQgBhQcICwwCBgiODg/2hwWP+wMEBwkP///8DQ0LDA0OD//9Da4HCgwDNc
cDV1mticMGCKqpCouJWARQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACH5BAAA
AAAALAAAAABMASYAAAX/ICCOZGmeaKqubOu+cCzPdG3feK7vfO//wKBwSCwaj8ikcslsOp/QqHRK
rVqv2Kx2y+16v+CweEwGDx5oR3nNdjrSuID86FAgEAzToc3v9x4SBHcMCg8jDwoJigOHigIjDhMQ
k5MTagAOmQE+dneMJAMCEX6kpTIRd6mpowAPqgmHgyIOEAa2twYQmLUGlzuoCAmPJA57Anumycp6
qQwREQypew8LqsiueAABvBCWkrnatRC+OQ53AuSYCrMKm8vvpgPRCAq+EncEb9XNIth5E7YguJs1
IcC3bgUKuAtQYIJDXwMSqgmgycEcAPdgifiEERmAZ/BC+jmQiiOAAauo/w26M8ofAIC5Bo4ocItX
r124DEwQUUBcpJgOEp6M5vHAqAMSQEkwKbKpGAGD0s0TMKAaAWAMzsiiKW5CAYuzKImVc8shr51c
JwUs4SorCQkREqSTYMip3TCd1pXIqxKBg3sIIpjLti0XLggFwtbyBXNnOFuZajpMTAKVRkgMPI6Q
oPmuZy0J7uglwVeQX3ksZT3OaSAxrVyXCgsc0dhBQJmVg5lgKoLz599a8poIjWDpPkbA+ImgKMmm
ATk+Z92mbcvSWhQoGaQ7UYc38O9QgC0g52BehL6f5qkuYfBWpuiYAjKubhtcimjeSwxICr6/lMGB
5eYJeiKQxI8l7PHynv8tC8H3ki1fXYcCKvypIMF2/mWoRHIRnJFcUgSKQJxulVg0wE8xFdbLHI1t
QsskFE2Xgh35iRBBhRrmuIR6C5iGxybYIMCRgdnYRIl7IsDUzTgqLlkdADTZlwJqnY1wo45YLvHX
PncssFQ/ppkEWB49OTeJL68h+aJkPFGCmwnlFVcXJoCMluWdRzwQQQMNnEcMGnNuBCgkCUlkAkWG
LheUV75QlEkLDsSFhwKJKMAKnphm+uczfmrq6aeghsrEKAMwckY/DXwSygqXmsBnjSsEKuqsSDAw
aAPDNIBJb4acWuosVKVa1wO+6HrSqXueZIgDJ55hKiOGoNEKrLRWi0OuA3QZIkCqAPC5GRp8PtDn
LEv1maq5IyRw7p4D7OmAKOehAxc64zYg7hmvWqtvD/Zmq8C2qIqgq7HejjBwK4AU3K3ACAMqgb3S
6gouw+bmu+/F1wIgwAPoYNStvQzvFywJ9kqspyjplozwfhgJa8jDSRE8wKsPY2yzH8berPM71O7s
889ABy300EQXXYoqSCet9NJMN+3001BHLfXUVFdt9dVYZ6311lxDbfTXX4QAADs=

------_=_NextPart_001_01C7FFBB.3F2A7779
Content-Type: image/gif;
	name="image002.gif"
Content-Transfer-Encoding: base64
Content-ID: <image002.gif@01C7FFC3.9A9B67D0>
Content-Description: image002.gif
Content-Location: image002.gif

R0lGODlhaQCFAMQVAAAAAP9wYJCwgDBAUP/////AgJCwwDAwMMy0TIiIWNTUtPCwaPDcoP/QkEBA
QLC4mFBAIGBgYICAgCAgEFhgUKDQ/6WaZYB1KmB8hgAAAAAAAAAAAAAAAAAAAAAAAAAAACH5BAAA
AAAALAAAAABpAIUAAAX/ICGOZGmeaKqubOu+cCzPdG3feK7vfM8rjKBwGFT4jjogo0FcNp0MI3Lq
UgobWKZiy11ir0Gq+ARkerkPhHq9fjy2UOd4bC0q0mqBXp+w7P8Ib1BMc0hOWFsIexgYewgUDg4J
CBcUFH8CCFtmYYVJV3d7CQIYkZN6DhYXFwIOGKN6r3qaCpyeOIeJAhYUBxcIpRQWeZcUFxa+jr3C
ipqcUrcyoIp+vK5qwZmtqhQJrn6ZpRgWCbBvTA3Q0S5BiHyRFg8YFIoXGAfDj6bzD8oUDxYk7cpU
S8g6dkL6KXL1aiEGfY0yIbAwjOIfb/EuHBBgL0+dgyvaMehXjtKFBxQe/1pS84aLEpd45iHwBsnB
TFYCrKgDSaKdgmbIHkpa1QzOIDCI7tR7eKCPA0t+CkbhWcJKP1UCEuC7J8nZlzMupaLbkskYAgdb
wUmlSgKdQko2vQ0s41NBhbt483Yp8mCUJUaKtK1l6zNrykf2Up0LYgFD3rt2HyvAMIGek5+VtPEZ
NhhkYV6lfl2QRBcBBAAHKmxRvRoyFwwAAExI4BPlw0wB8/nkaabfvUlEcwpJMCH2ANWPWXOpADs2
agtLQs2qVC6nGc/RJd57lUpJAwrOARxPjne5gdgHDsiGnq4Z5b+KyuwspKUevYBNH7Q7feCe+PLk
QXYeI4zERlt73oyjyP8o+nVyS2HGjDbOJGXwR+ABFLgEWXl2KWBAfxcaGB1FipTSlDNTPRiFQ31d
UxB4ADDC3AAZRiZZawZM4Bgj6sUGXRSGVYYBKw9c50lhMz2VQEoFWeAcBgMwoqONrSm3GmXpUebc
BEo8sMqSWelRhooMLDhMaJpEURx6lE1wgAF62eiaXYyktKZzF0S3JBvxGTlHfd4oOcqL4cnGSJSr
yRlZa+oVd8AAdwLw40yvEDiMQXPIxxEjDlSSJgJrCtljjBrqpVwFAxg6QJQwAuCAOxr1kkqaDBSS
EEdoXSJAkQzA2N8BOjaHAZVUWgkeBatCCqVz0P0UUKeCaTFHEc2QYoz/BQUVp+MEUUZZ3LCnkrcF
bFDWGSOeS7SIAR67YiqGOwlUUpFwTqKmbIgGdNjavpDBlhKB58bGZRTlLLnKpQ5OURBmiUkURALo
gUdgc/myJu6Vsf0rXnM+ArlKf4cpQYcWrXSTyq5MnBYbbJVRoC2cASLHxYfhbRsebT/t8pSLS4xx
65BokcbEqP3d2Z/MHG7ImgGHRqlleBQEIY8v8grXgM/pSrRLH0AoECmIh6am6L76svzoqOFBgEVf
o+RhXYpT3AqIcKAWihqwbl4MoGptTuB3pLGp3YBCmCwshtya9EM34ABAwOfjkLNBlEZ2vzq4Hm8E
ZjgVt2riuXBe2914/+SkQ65yoYID5EYgYkrLedaepwE644GXbjsCaKed9eqKv3S41LPI/nbuAFRi
1u2QXwCB8nZHPVLwe4wpxkvBB8br6Rlr1RQDC3S/APJqQOAm8zeviHlgwtU6PckSBYZAEK3KBnFs
F3jfPfiUCHxBoc064rbIWFPAHt7QOwbUCz3zmwAC7Pc90tkvf/S708AEuCvhvS9hcTMf62RXi9AJ
bH4OYGADI/fAuk1gf+i63Ooo2CD1/a5MBHED5pgQPwqcJmoMLF0JT0O+juUsEDJ823yOoKnzRa9M
EkwABCwgQtvdTw3KQyF63DEL9AHwT0DCnOLeFr8EiPCJ+INgx2BIEP/3uQtrZGRJUZAosAV+MYxQ
dA4EspMJdr3NhVhLR/RWhy3GbMmLTQyj+NoYBEW4IYhWi0YRZiFA4REqY9zLoSCZRUfezRCDeaRg
4hxRBgf46I1hROGBKIi5nAkxGgtDH+suqQBPMjGQgUweAEbpPuG1C5PTouOugCimIgjjiwvIgjC5
l7wEIMJ6adji5tZhBvQdsoBOYKAXvCDML4xQDQuIjuYCYUU/raMw50OkfqzZPTDksABeaGA2jxk9
bjKyZ1QRgiaVokqpDKEB2AyAPvUZTHSsk52zyIkyvQmS3vzhJ6u0GjoKoIDuFWCf2SwTAtARHcVx
E5F3ZIsIhuA+liD/Mk1MWEABFhAABgRApPiEqBlMmZPYRW+lGt2oNgeYzIMugaTcK+n7GkDSfQag
AFSsY0tfWoSYjsAnzkymIZfw0AA0tKRY6KlP+8lCNbD0I0Y9qhO2SJDWpTQAbsRCU32qTycUpXC2
yGpP5Ok/r0oVpTwl6z6BmsaDplWta52pXXO6z4nyVKQFQIBPgXq5PyyGEHg1ARSk49bBhrScDBgr
XblahyEmlgBWSMdVahHXfTJVpV+l60RocdfLouAQmuXsWKG62s+u8yeLkYNpV5DZKETTsw1YLVAf
OtEimAGxbHHJE4bgEhLUoR2STYdjRxoH2Rp3L8Mtbg6aO1wiZKET/8ftbDBNOtfc0hUKIoDDdasb
3U/YAQ3PbAlC16COOgj2p3zd7iHgRgCJsvcO6VUvVnlwmYFggnTzucJIRRpJuo5FsZBj6R44Ex0k
vISru0xwrbqwiXZ4l5pECMtGS1e4MxqCWue7bxd+S2J0DPML1cRwdLbA0PW6DaR45Bwi+rGXaq40
LDiGSY6FS83ruiR99F1fe3aqhbCszm1VbOt/0VdFIIbFwgK0wi3qEw8m2BETWDZMRQ5GiorM4xRZ
/u85BBgPDzsYxJlozzAgjOUkTS5C25iJL2YVZqHuokikbGEAndkegbIZEEPiBUfKUYwzXaIrdb4D
C/tcSlzuQJ5hGf8yjescjgtgxNJguoQ8jkHnLGtilyiLQqQdjYsrpBiGmUvofxPUlAhxWhh/6TQm
kikdXtl4CD7gQnh1rVfo1Xk0wlDFMQTwlyVdI8y2xNx5dQ2HaDA6yXWultbapw0mL3nWpGamgv3n
hh17e8dHRna2nf1D/WbWxsO0cI9PLeotsKtBWS2yujOs613PZwvPNco9zztuT3gH0oU4d7+PFOSD
XHG2CE+4whfO8IY7/OFVkIAELJsDBUiABBO3gcXFIIF/vIEAGe/BAybwABE8AAAlr4ECKCAGB6Sc
ACOPQMktnvI3TPwBEng5zjeOWYmTACUsJ0AEInBxi4ec5hcHecb/jS7xjU/c5+HNOQ5GnnQFHGDi
rcR5BEDugAjQnOggl3kEHIBZl4NdBBLnktV9nnMJkD3rEZhAzx9wAALSPedkfwpKLv4AmdcdBwqI
uxFcLvSkc0kCW+/52AlwACM8gOxEd4Pc0Y54iVve5BIAgNBnrvmeOyDpHYd53kv+eMa7IfQ5CD3h
CU8ABxgdsxOQONk7X/quXx7tbohAhnDO+JxrnvWap3vfi84l0bee9LO3fNJVjvatE97nCtA82y/u
dgL8g+tKN4HMCTABsqOk9NHPfvgJD/a/G5/wpb9+Dsb+FMfHXuiRMAL0K1P9VlJg8fD//AjIz/e8
X13uK/d5wVcZ/0SXeZEgc6NnfAGofhqFeElwABBnApXXeDcwfKwXgcb1cjegaBjYgRHXdC5QejgQ
fyUAdSoQcm5wBMInAcXHAiKoAryXAg44AolngjJoeBRnAyJoeRtndNAgcaWXdCnYcxMXCcvXc0bQ
SvoHcgDwdDwYdUfoNRa3da8Hc1IHcjwHAyL4D09RhLJnBEP3eLNHeTBXd28wdD/4DxQwcWgYXihn
cSQXenB3hBLXeHMIcsPXemsoAyPXhccHc4lHdF5jcmOYfRRIhiNAgYNog50HdZoXeSNXAu8HiQCI
eAnIh2S3fyUHdUCYiaXXiBfXeYgoAqIod4yIiL9XgEf4h7bXdP+xV30XqIWZKAKE10oiUHzFV33F
t4d7OIrWt4ksd4rZRwDSt4qaOIzZB4sa+AIv+Idct4R0t4ZkJwEYAn01sQUTkHiYBQkZgoxC13iO
KALst4qsN45eI43O6IHquI7s2I7u+I48MISYVYIxcAcjwIGYpQ4fl4EnYIxjAIS6h4UkMIshKIRi
13jyiIcYl3PL2HrMqANXyJDyd3HeJwU0h1k4l3JMh3mYZ49sR0C0+IMY2XoW53hPh3ZKR3VEiJFY
t3Qh1481B4Qg6HrauIl410qvd3YxeHxPmHNeF15XR4tWyH13kCFeF3hbUICDJwIZ0okrN3EQ6HoU
d4U4l3OtRJGfh6h4bYeHniiEVSdzXghyQXmPPud9FBmSa3gHkXeVDnmVbsd7Ugd5P4kCV+h6V5gh
LjeLZgmXfWeSHJmSSdeU1LgTZAeBvAeBDpl9TzECeIlZNUl9JZeJKyeDRLd9DIl4tgh1lbmVVkl0
vfh4lWeFWkdztqh0GYd4YPd5MjeFE2mFXueAlnd6CjmNNviO3QiPOFCbuLmbvNmbvvmbDxcCADs=

------_=_NextPart_001_01C7FFBB.3F2A7779--
