X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C89B49.EDAB81E8@onstor-exch02.onstor.net>; Thu, 10 Apr 2008 13:32:00 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C89B49.EDAB81E8"
Content-class: urn:content-classes:message
Subject: RE: ls take very long time to return
Date: Thu, 10 Apr 2008 13:31:59 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E09596822@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E09596819@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: ls take very long time to return
Thread-Index: AciVlMqALLP+//Y3S8ifzeA6ieKGvgFtJeRQAAAQp7A=
From: "Paul Hammer" <paul.hammer@onstor.com>
To: "Narayan Venkat" <narayan.venkat@onstor.com>,
	"John Culp" <john.culp@onstor.com>,
	"dl-cstech" <dl-cstech@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C89B49.EDAB81E8
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi All,

=20

I had asked that a bug be filed for this along with details about the
client machine and if they could include the script that they are using
to create this structure that would be very helpful. I am not aware of
this one as a known issue.

=20

-Paul

=20

=20

=20

________________________________

From: Narayan Venkat=20
Sent: 2008-04-10 13:29
To: John Culp; dl-cstech
Subject: RE: ls take very long time to return

=20

Folks,

=20

Is this a known issue? Can somebody acknowledge?  We have deal pending
on this one!

=20

Thanks.

=20

Narayan Venkat

Vice President, Marketing

ONStor Inc. (www.onstor.com)

Tel: (408) 963-2404

Cell: (408) 221-4297

________________________________

From: John Culp=20
Sent: Thursday, April 03, 2008 7:13 AM
To: dl-cstech
Subject: ls take very long time to return

=20

IHAP that  is running an eval currently and part of their work load is
to load directories with 64,000 - 300k - 500k files.  These files are
then randomly read back out by their application, processed and the out
put is written some where else.

=20

The issue is that while copying the files into the directory the
customer will do a "ls | wc" or similar to check status.  As more and
more files are being copied into the directory the response takes longer
and longer.  At one point, 22 min to return.  If the file copy finishes
or is killed, the "ls" will return as expected.

=20

I tested this very scenario in our lab to our filer and the same thing
happened.  I tested it between two NFS servers (not our filer) and it
did not happen.  I used the following script


count=3D1
while [ $count !=3D 32768 ]
do
dd if=3D/dev/zero of=3D./testdir/bob$count bs=3D8k count=3D50 > =
/dev/null
((count=3Dcount+1))
done
exit=20

Then in a separate ssh shell I ran "time ls | wc" the response was ok at
first but ended up taking a longer and longer time until the following:

[root@cspc01 testdir]# ls | wc
[root@cspc01 testdir]# time ls | wc
30148 30148 260222

real 3m19.319s
user 0m0.380s
sys 0m16.120s

[root@cspc01 testdir]# time ls | wc
32058 32058 277412

real 3m48.024s
user 0m0.230s
sys 0m17.340s

[root@cspc01 testdir]# time ls | wc
32768 32768 283802

real 0m0.692s
user 0m0.160s
sys 0m0.130s

I also ran vsv stats on the filer and noticed nfs IO at about 4500 in
and out the whole time. If I was not running the ls | wc, NFS in was 30+
MB/PS in and 250k out. When I was running the ls | wc it was 1.3MB/PS in
and 15MB/PS out.

=20

=20

Here is an email and output from the customer.

Here's a little more info. I've attached the output of "vsvr stats agg
-i 3". At the point in the file it "<--- HERE" is when I executed the
command below. You can see how long it took to complete. I also went in
as root and did some poking around using iostat and vmstat. There wasn't
anything particularly useful or interesting, so I didn't bother with
output. Let me know if you'd like me to check something else.

[root@garrison scratch2]# time ls &> /dev/null=20

real 22m40.984s
user 0m0.018s
sys 1m2.885s

=20

John Culp=20
Regional SE Manager
940-239-7489 Office=20
972-523-4287 Cell=20
john.culp@onstor.com=20

=20


------_=_NextPart_001_01C89B49.EDAB81E8
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
 namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" =
name=3D"PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:"MS Mincho";
	panose-1:2 2 6 9 4 2 5 8 3 4;}
@font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
	{font-family:"\@MS Mincho";
	panose-1:0 0 0 0 0 0 0 0 0 0;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:purple;
	text-decoration:underline;}
p
	{mso-margin-top-alt:auto;
	margin-right:0in;
	mso-margin-bottom-alt:auto;
	margin-left:0in;
	font-size:12.0pt;
	font-family:"Times New Roman";}
span.EmailStyle18
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle19
	{mso-style-type:personal-reply;
	font-family:Arial;
	color:navy;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Hi =
All,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>I had asked that a bug be filed for =
this along
with details about the client machine and if they could include the =
script that
they are using to create this structure that would be very helpful. I am =
not
aware of this one as a known issue.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>-Paul<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">Narayan Venkat</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> 2008-04-10 =
13:29<br>
<b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName =
w:st=3D"on">John
 Culp</st1:PersonName>; dl-cstech<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: ls take very =
long
time to return</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Folks,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Is this a known issue? Can somebody
acknowledge?&nbsp; We have deal pending on this =
one!<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Thanks.<o:p></o:p></span></font></p>=


<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<p class=3DMsoNormal><st1:PersonName w:st=3D"on"><font size=3D2 =
color=3Dnavy
 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial;color:navy'>Narayan
 Venkat</span></font></st1:PersonName><font color=3Dnavy><span =
style=3D'color:navy'><o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Vice President, =
Marketing</span></font><font
color=3Dnavy><span style=3D'color:navy'><o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>ONStor Inc. (<a
href=3D"http://www.onstor.com">www.onstor.com</a>)</span></font><font =
color=3Dnavy><span
style=3D'color:navy'><o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Tel: (408) =
963-2404</span></font><font
color=3Dnavy><span style=3D'color:navy'><o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Cell: (408) =
221-4297</span></font><o:p></o:p></p>

</div>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
<st1:PersonName
w:st=3D"on">John Culp</st1:PersonName> <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Thursday, April 03, =
2008
7:13 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> dl-cstech<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> ls take very =
long time to
return</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>IHAP that&nbsp; is running an eval currently and part =
of
their work load is to load directories with 64,000 - 300k - 500k =
files.&nbsp; These
files are then randomly read back out by their application, processed =
and the
out put is written some where else.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>The issue is that while copying the files into the =
directory
the customer will do a &quot;ls | wc&quot; or similar to check =
status.&nbsp; As
more and more files are being copied into the directory the response =
takes
longer and longer.&nbsp; At one point, 22 min to return.&nbsp; If the =
file copy
finishes or is killed, the &quot;ls&quot; will return as =
expected.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>I tested this very scenario in our lab to our filer =
and the same
thing happened.&nbsp; I tested it between two NFS servers (not our =
filer) and
it did not happen.&nbsp; I used the following =
script</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><br>
count=3D1<br>
while [ $count !=3D 32768 ]<br>
do<br>
dd if=3D/dev/zero of=3D./testdir/bob$count bs=3D8k count=3D50 &gt; =
/dev/null<br>
((count=3Dcount+1))<br>
done<br>
exit </span></font><font size=3D2><span =
style=3D'font-size:10.0pt'><o:p></o:p></span></font></p>

</div>

<div>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>Then
in a separate ssh shell I ran &quot;time ls | wc&quot; the response was =
ok at
first but ended up taking a longer and longer time until the =
following:</span></font><font
size=3D2><span style=3D'font-size:10.0pt'><o:p></o:p></span></font></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>[root@cspc01
testdir]# ls | wc<br>
[root@cspc01 testdir]# time ls | wc<br>
30148 30148 260222<br>
<br>
real 3m19.319s<br>
user 0m0.380s<br>
sys 0m16.120s</span></font><font size=3D2><span =
style=3D'font-size:10.0pt'><o:p></o:p></span></font></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>[root@cspc01
testdir]# time ls | wc<br>
32058 32058 277412</span></font><font size=3D2><span =
style=3D'font-size:10.0pt'><o:p></o:p></span></font></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>real
3m48.024s<br>
user 0m0.230s<br>
sys 0m17.340s</span></font><font size=3D2><span =
style=3D'font-size:10.0pt'><o:p></o:p></span></font></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>[root@cspc01
testdir]# time ls | wc<br>
32768 32768 283802</span></font><font size=3D2><span =
style=3D'font-size:10.0pt'><o:p></o:p></span></font></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>real
0m0.692s<br>
user 0m0.160s<br>
sys 0m0.130s</span></font><font size=3D2><span =
style=3D'font-size:10.0pt'><o:p></o:p></span></font></p>

<p><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>I
also ran vsv stats on the filer and noticed nfs IO at about 4500 in and =
out the
whole time. If I was not running the ls | wc, NFS in was 30+ MB/PS in =
and 250k
out. When I was running the ls | wc it was 1.3MB/PS in and 15MB/PS =
out.</span></font><font
size=3D2><span style=3D'font-size:10.0pt'><o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Here is an email and output from the =
customer.</span></font><o:p></o:p></p>

</div>

<div>

<p><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt'>Here's a
little more info. I've attached the output of &quot;vsvr stats agg -i =
3&quot;.
At the point in the file it &quot;&lt;--- HERE&quot; is when I executed =
the
command below. You can see how long it took to complete. I also went in =
as root
and did some poking around using iostat and vmstat. There wasn't =
anything
particularly useful or interesting, so I didn't bother with output. Let =
me know
if you'd like me to check something else.</span></font><font =
size=3D2><span
style=3D'font-size:10.0pt'><o:p></o:p></span></font></p>

<p><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt'>[root@garrison
scratch2]# time ls &amp;&gt; /dev/null </span></font><font =
size=3D2><span
style=3D'font-size:10.0pt'><o:p></o:p></span></font></p>

<p><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:12.0pt'>real
22m40.984s<br>
user 0m0.018s<br>
sys 1m2.885s</span></font><font size=3D2><span =
style=3D'font-size:10.0pt'><o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<p><!-- Converted from text/rtf format --><st1:PersonName =
w:st=3D"on"><font
 size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>John =
Culp</span></font></st1:PersonName>
<br>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>Regional
SE Manager</span></font><br>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>940-239-7489
Office</span></font> <br>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>972-523-4287
Cell</span></font> <br>
<font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'>john.culp@onstor.com</span><=
/font>
<o:p></o:p></p>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

</div>

</body>

</html>

------_=_NextPart_001_01C89B49.EDAB81E8--
