X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7196A.6D6DC7A9@onstor-exch02.onstor.net>; Wed, 6 Dec 2006 11:12:09 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7196A.6D6DC7A9"
Content-class: urn:content-classes:message
Subject: FW: Perforce update - Was: RE: any problem with p4 ?
Date: Wed, 6 Dec 2006 11:12:07 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E019F5693@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Perforce update - Was: RE: any problem with p4 ?
thread-index: AccX0RZ13rmGOcxyTwyBkSUy5ZKXfQAAw2H1AAN/xQAARbVgMAAAmz+6ABt/EXA=
X-Priority: 1
Priority: Urgent
Importance: high
From: "Ken Renshaw" <ken.renshaw@onstor.com>
To: "dl-Engineering" <dl-engineering@onstor.com>
Cc: "Kevin Matthews" <kevin.matthews@onstor.com>,
	"Brian Baker" <brian.baker@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7196A.6D6DC7A9
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Perforce is up at the moment so quick sync your trees! :-)

=20

The tact now is that we've disabled icmp responses and brought up the
services and are watching it to see if there's any slowdown in normal
operations. The interactions between liszt and mightydog vsvrs does not
occur in normal day-to-day perforce usage, but rather a) when something
is posted to /n/Build-Trees and the BUILD vsvr is automounted and more
importantly b) during backup operations. The database and depot backup
images and db checkpoint and journal data are stored off under
/n/QA/Perforce/BACKUPS, another mightydog vsvr.=20

=20

I can just simply not use liszt for anything that posts to
/n/Build-Trees easy enough, but the dependence on doing nightly backups
means we need at least one vsvr, and thus we can't get away from the
problem.

=20

Thanks, especially to Jeseem for jumping in to help us all out.

=20

-Ken

=20

________________________________

From: Ken Renshaw=20
Sent: Tuesday, December 05, 2006 10:11 PM
To: Jobi Ariyamannil
Cc: dl-Software; dl-QA
Subject: Perforce update - Was: RE: any problem with p4 ?
Importance: High

=20

Yeah, I know, but thanks Jobi. Jeseem has been helping on it all
afternoon and we have a working theory that CQ 11072 or something like
it on mightydog is causing the problem. We have been trying to stabilize
the Perforce service by adding static MAC entries on mightydog's vsvrs
for liszt, the server running p4. There's basically not much I can do
from the server-side, everything is running fine locally.

=20

Jeseem's explanation of what's going on:

=20

When a virtual server wants to know the IP to mac translation for a
client ( like say the perforce server) it sends an arp request.
The client sends back an arp reply.

The arp reply is sent to the filer's mac address. Since all virtual
servers share the same mac address, it reaches all the virtual servers.
The virtual server which did not issue the request return the invalid
ICMP error message ( since they hadn't requested that info.) .

=20

We started out by adding liszt's MAC address to the BUILD vsvr context,
and it seemed to fix things for a time this afternoon, but after a while
it started again and Jeseem added the MAC to all the vsvrs. Again,
seemed to run fine for a while and now this evening the system's network
interface is totally whacked again.

=20

There is lots of traffic between the build vsvr on mightydog and liszt,
the Perforce server, and I've seen tons of these errors. Neil has
brought up a like question in the context of dl-cstech today from a
customer site today.

=20

Anyone's insight into what may be happening or how to fix it are most
welcome, on behalf of all Perforce users and myself.

=20

Thanks,

=20

-Ken

=20

________________________________

From: Jobi Ariyamannil
Sent: Tue 12/5/2006 9:40 PM
To: Ken Renshaw
Cc: dl-Software
Subject: RE: any problem with p4 ?

Hi Ken,

I see same problem again.

Regards,
Jobi

-----Original Message-----
From: Ken Renshaw
Sent: Monday, December 04, 2006 12:25 PM
To: Ken Renshaw; Jobi Ariyamannil
Cc: dl-Software
Subject: RE: any problem with p4 ?
Importance: High

The Perforce service is again available, I'm still looking into what
happened. If there are any issues please let me know ASAP, thanks.

-Ken

-----Original Message-----
From: Ken Renshaw
Sent: Monday, December 04, 2006 10:44 AM
To: Jobi Ariyamannil
Cc: dl-Software
Subject: RE: any problem with p4 ?

Yep, it appears to be having issues; Liszt responds to pings but no ssh
or p4 commands. I'll have it back up just as quick as I can, thanks
Jobi.

Happy Monday!

-Ken

P.S. The checkpoint and backup of Perforce ran perfectly last evening,
which is good, but I don't yet know why the system is nonresponsive this
morning.


-----Original Message-----
From: Jobi Ariyamannil
Sent: Mon 12/4/2006 10:21 AM
To: Ken Renshaw
Subject: any problem with p4 ?

Hi Ken,

I am seeing p4 commands hanging for long time.

jobia@localhost:Linux 2.6.9-11.ELsmp:/homes/jobia/p4/R21 > p4 sync ...
Perforce client error:
        TCP receive failed.
        read: socket: Connection reset by peer


Regards,
Jobi


------_=_NextPart_001_01C7196A.6D6DC7A9
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<title>RE: any problem with p4 ?</title>
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:Wingdings;
	panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:purple;
	text-decoration:underline;}
p
	{mso-margin-top-alt:auto;
	margin-right:0in;
	mso-margin-bottom-alt:auto;
	margin-left:0in;
	font-size:12.0pt;
	font-family:"Times New Roman";}
span.EmailStyle18
	{mso-style-type:personal-reply;
	font-family:Arial;
	color:navy;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
	{page:Section1;}
-->
</style>

</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Perforce is up at the moment so =
quick sync
your trees! </span></font><font size=3D2 color=3Dnavy =
face=3DWingdings><span
style=3D'font-size:10.0pt;font-family:Wingdings;color:navy'>J</span></fon=
t><font
size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial;
color:navy'><o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>The tact now is that we&#8217;ve =
disabled
icmp responses and brought up the services and are watching it to see if =
there&#8217;s
any slowdown in normal operations. The interactions between liszt and =
mightydog
vsvrs does not occur in normal day-to-day perforce usage, but rather a) =
when
something is posted to /n/Build-Trees and the BUILD vsvr is automounted =
and
more importantly b) during backup operations. The database and depot =
backup images
and db checkpoint and journal data are stored off under =
/n/QA/Perforce/BACKUPS,
another mightydog vsvr. <o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>I can just simply not use liszt for
anything that posts to /n/Build-Trees easy enough, but the dependence on =
doing
nightly backups means we need at least one vsvr, and thus we can&#8217;t =
get
away from the problem.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Thanks, especially to Jeseem for =
jumping
in to help us all out.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>-Ken<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> Ken =
Renshaw <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, December =
05, 2006
10:11 PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Jobi Ariyamannil<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> dl-Software; =
dl-QA<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> Perforce update =
- Was:
RE: any problem with p4 ?<br>
<b><span style=3D'font-weight:bold'>Importance:</span></b> =
High</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div id=3DidOWAReplyText50096>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dblack face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:black'>Yeah, I know, but thanks Jobi. =
Jeseem has
been helping on it all afternoon and we have a working theory that CQ =
11072 or
something like it on mightydog is causing the problem. We have been =
trying to
stabilize the Perforce service by adding static MAC entries on =
mightydog's
vsvrs for liszt, the server running p4. There's basically not much I can =
do
from the server-side, everything is running fine =
locally.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dblack face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:black'>Jeseem's explanation of what's =
going on:</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3D"Times New Roman"><span =
style=3D'font-size:
10.0pt'>When a virtual server wants to know the IP to mac translation =
for a<br>
client ( like say the perforce server) it sends an arp request.<br>
The client sends back an arp reply.<br>
<br>
The arp reply is sent to the filer's mac address. Since all virtual<br>
servers share the same mac address, it reaches all the virtual =
servers.<br>
The virtual server which did not issue the request return the =
invalid<br>
ICMP error message ( since they hadn't requested that info.) =
.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>We started out by adding liszt's MAC address to the =
BUILD
vsvr context, and it seemed to fix things for a time this afternoon, but =
after
a while it started again and Jeseem added the MAC to all the vsvrs. =
Again,
seemed to run fine for a while and now this evening the system's network
interface is totally whacked again.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>There is lots of traffic between the build vsvr on =
mightydog
and liszt, the Perforce server, and I've seen tons of&nbsp;these errors. =
Neil
has brought up a like question in the context of dl-cstech today from a
customer site today.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Anyone's insight into what may be happening or how to =
fix it
are most welcome, on behalf of all Perforce users and =
myself.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Thanks,</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>-Ken</span></font><o:p></o:p></p>

</div>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;font-family:Tahoma'> Jobi
Ariyamannil<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Tue 12/5/2006 9:40 =
PM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Ken Renshaw<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> dl-Software<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: any problem =
with p4 ?</span></font><o:p></o:p></p>

</div>

<div>

<p style=3D'margin-bottom:12.0pt'><font size=3D2 face=3D"Times New =
Roman"><span
style=3D'font-size:10.0pt'>Hi Ken,<br>
<br>
I see same problem again.<br>
<br>
Regards,<br>
Jobi<br>
<br>
-----Original Message-----<br>
From: Ken Renshaw<br>
Sent: Monday, December 04, 2006 12:25 PM<br>
To: Ken Renshaw; Jobi Ariyamannil<br>
Cc: dl-Software<br>
Subject: RE: any problem with p4 ?<br>
Importance: High<br>
<br>
The Perforce service is again available, I'm still looking into what =
happened.
If there are any issues please let me know ASAP, thanks.<br>
<br>
-Ken<br>
<br>
-----Original Message-----<br>
From: Ken Renshaw<br>
Sent: Monday, December 04, 2006 10:44 AM<br>
To: Jobi Ariyamannil<br>
Cc: dl-Software<br>
Subject: RE: any problem with p4 ?<br>
<br>
Yep, it appears to be having issues; Liszt responds to pings but no ssh =
or p4
commands. I'll have it back up just as quick as I can, thanks Jobi.<br>
<br>
Happy Monday!<br>
<br>
-Ken<br>
<br>
P.S. The checkpoint and backup of Perforce ran perfectly last evening, =
which is
good, but I don't yet know why the system is nonresponsive this =
morning.<br>
<br>
<br>
-----Original Message-----<br>
From: Jobi Ariyamannil<br>
Sent: Mon 12/4/2006 10:21 AM<br>
To: Ken Renshaw<br>
Subject: any problem with p4 ?<br>
<br>
Hi Ken,<br>
<br>
I am seeing p4 commands hanging for long time.<br>
<br>
jobia@localhost:Linux 2.6.9-11.ELsmp:/homes/jobia/p4/R21 &gt; p4 sync =
...<br>
Perforce client error:<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; TCP receive failed.<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; read: socket: Connection =
reset by
peer<br>
<br>
<br>
Regards,<br>
Jobi</span></font><o:p></o:p></p>

</div>

</div>

</body>

</html>

------_=_NextPart_001_01C7196A.6D6DC7A9--
