X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C718FD.4314DDC1@onstor-exch02.onstor.net>; Tue, 5 Dec 2006 22:10:43 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C718FD.4314DDC1"
Content-class: urn:content-classes:message
Subject: Perforce update - Was: RE: any problem with p4 ?
Date: Tue, 5 Dec 2006 22:10:43 -0800
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E01335FE7@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Perforce update - Was: RE: any problem with p4 ?
thread-index: AccX0RZ13rmGOcxyTwyBkSUy5ZKXfQAAw2H1AAN/xQAARbVgMAAAmz+6
X-Priority: 1
Priority: Urgent
Importance: high
References: <BB375AF679D4A34E9CA8DFA650E2B04E019F5459@onstor-exch02.onstor.net>
From: "Ken Renshaw" <ken.renshaw@onstor.com>
To: "Jobi Ariyamannil" <jobi.ariyamannil@onstor.com>
Cc: "dl-Software" <dl-software@onstor.com>,
	"dl-QA" <dl-qa@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C718FD.4314DDC1
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Yeah, I know, but thanks Jobi. Jeseem has been helping on it all =
afternoon and we have a working theory that CQ 11072 or something like =
it on mightydog is causing the problem. We have been trying to stabilize =
the Perforce service by adding static MAC entries on mightydog's vsvrs =
for liszt, the server running p4. There's basically not much I can do =
from the server-side, everything is running fine locally.
=20
Jeseem's explanation of what's going on:
=20
When a virtual server wants to know the IP to mac translation for a
client ( like say the perforce server) it sends an arp request.
The client sends back an arp reply.

The arp reply is sent to the filer's mac address. Since all virtual
servers share the same mac address, it reaches all the virtual servers.
The virtual server which did not issue the request return the invalid
ICMP error message ( since they hadn't requested that info.) .
=20
We started out by adding liszt's MAC address to the BUILD vsvr context, =
and it seemed to fix things for a time this afternoon, but after a while =
it started again and Jeseem added the MAC to all the vsvrs. Again, =
seemed to run fine for a while and now this evening the system's network =
interface is totally whacked again.
=20
There is lots of traffic between the build vsvr on mightydog and liszt, =
the Perforce server, and I've seen tons of these errors. Neil has =
brought up a like question in the context of dl-cstech today from a =
customer site today.
=20
Anyone's insight into what may be happening or how to fix it are most =
welcome, on behalf of all Perforce users and myself.
=20
Thanks,
=20
-Ken

________________________________

From: Jobi Ariyamannil
Sent: Tue 12/5/2006 9:40 PM
To: Ken Renshaw
Cc: dl-Software
Subject: RE: any problem with p4 ?



Hi Ken,

I see same problem again.

Regards,
Jobi

-----Original Message-----
From: Ken Renshaw
Sent: Monday, December 04, 2006 12:25 PM
To: Ken Renshaw; Jobi Ariyamannil
Cc: dl-Software
Subject: RE: any problem with p4 ?
Importance: High

The Perforce service is again available, I'm still looking into what =
happened. If there are any issues please let me know ASAP, thanks.

-Ken

-----Original Message-----
From: Ken Renshaw
Sent: Monday, December 04, 2006 10:44 AM
To: Jobi Ariyamannil
Cc: dl-Software
Subject: RE: any problem with p4 ?

Yep, it appears to be having issues; Liszt responds to pings but no ssh =
or p4 commands. I'll have it back up just as quick as I can, thanks =
Jobi.

Happy Monday!

-Ken

P.S. The checkpoint and backup of Perforce ran perfectly last evening, =
which is good, but I don't yet know why the system is nonresponsive this =
morning.


-----Original Message-----
From: Jobi Ariyamannil
Sent: Mon 12/4/2006 10:21 AM
To: Ken Renshaw
Subject: any problem with p4 ?

Hi Ken,

I am seeing p4 commands hanging for long time.

jobia@localhost:Linux 2.6.9-11.ELsmp:/homes/jobia/p4/R21 > p4 sync ...
Perforce client error:
        TCP receive failed.
        read: socket: Connection reset by peer


Regards,
Jobi




------_=_NextPart_001_01C718FD.4314DDC1
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<HTML dir=3Dltr><HEAD><TITLE>RE: any problem with p4 ?</TITLE>=0A=
<META http-equiv=3DContent-Type content=3D"text/html; charset=3Dunicode">=0A=
<META content=3D"MSHTML 6.00.5730.11" name=3DGENERATOR></HEAD>=0A=
<BODY>=0A=
<DIV id=3DidOWAReplyText50096 dir=3Dltr>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 size=3D2>Yeah, I know, =
but thanks Jobi. Jese</FONT><FONT face=3DArial color=3D#000000 =
size=3D2>em has been helping on it all afternoon and </FONT><FONT =
face=3DArial color=3D#000000 size=3D2>we have a working theory that CQ =
11072 or something like it </FONT><FONT face=3DArial color=3D#000000 =
size=3D2>on mightydog is causing the problem. We </FONT><FONT =
face=3DArial color=3D#000000 size=3D2>have been trying to stabilize the =
Perforce service by adding static MAC entries on mightydog's vsvrs for =
liszt, the server running p4</FONT><FONT face=3DArial color=3D#000000 =
size=3D2>. There's basically not much I can do from the server-side, =
everything is running fine locally.</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 =
size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 size=3D2>Jeseem's =
explanation of what's going on:</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT size=3D2>When a virtual server wants to know the IP =
to mac translation for a<BR>client ( like say the perforce server) it =
sends an arp request.<BR>The client sends back an arp reply.<BR><BR>The =
arp reply is sent to the filer's mac address. Since all =
virtual<BR>servers share the same mac address, it reaches all the =
virtual servers.<BR>The virtual server which did not issue the request =
return the invalid<BR>ICMP error message ( since they hadn't requested =
that info.) .</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>We started out by adding =
liszt's MAC address to the BUILD vsvr context, and it seemed to fix =
things for a time this afternoon, but after a while it started again and =
Jeseem added the MAC to all the vsvrs. Again, seemed to run fine for a =
while and now this evening the system's network interface is totally =
whacked again.</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>There is lots of traffic =
between the build vsvr on mightydog and liszt, the Perforce server, and =
I've seen tons of</FONT><FONT face=3DArial size=3D2>&nbsp;these errors. =
Neil has brought up a like question in the c</FONT><FONT face=3DArial =
size=3D2>ontext of dl-cstech today from a customer site =
today.</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>Anyone's insight into what =
may be happening or how to fix it are most welcome, on behalf of all =
Perforce users and myself.</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>Thanks,</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>-Ken</FONT></DIV></DIV>=0A=
<DIV dir=3Dltr><BR>=0A=
<HR tabIndex=3D-1>=0A=
<FONT face=3DTahoma size=3D2><B>From:</B> Jobi =
Ariyamannil<BR><B>Sent:</B> Tue 12/5/2006 9:40 PM<BR><B>To:</B> Ken =
Renshaw<BR><B>Cc:</B> dl-Software<BR><B>Subject:</B> RE: any problem =
with p4 ?<BR></FONT><BR></DIV>=0A=
<DIV>=0A=
<P><FONT size=3D2>Hi Ken,<BR><BR>I see same problem =
again.<BR><BR>Regards,<BR>Jobi<BR><BR>-----Original =
Message-----<BR>From: Ken Renshaw<BR>Sent: Monday, December 04, 2006 =
12:25 PM<BR>To: Ken Renshaw; Jobi Ariyamannil<BR>Cc: =
dl-Software<BR>Subject: RE: any problem with p4 ?<BR>Importance: =
High<BR><BR>The Perforce service is again available, I'm still looking =
into what happened. If there are any issues please let me know ASAP, =
thanks.<BR><BR>-Ken<BR><BR>-----Original Message-----<BR>From: Ken =
Renshaw<BR>Sent: Monday, December 04, 2006 10:44 AM<BR>To: Jobi =
Ariyamannil<BR>Cc: dl-Software<BR>Subject: RE: any problem with p4 =
?<BR><BR>Yep, it appears to be having issues; Liszt responds to pings =
but no ssh or p4 commands. I'll have it back up just as quick as I can, =
thanks Jobi.<BR><BR>Happy Monday!<BR><BR>-Ken<BR><BR>P.S. The checkpoint =
and backup of Perforce ran perfectly last evening, which is good, but I =
don't yet know why the system is nonresponsive this =
morning.<BR><BR><BR>-----Original Message-----<BR>From: Jobi =
Ariyamannil<BR>Sent: Mon 12/4/2006 10:21 AM<BR>To: Ken =
Renshaw<BR>Subject: any problem with p4 ?<BR><BR>Hi Ken,<BR><BR>I am =
seeing p4 commands hanging for long time.<BR><BR>jobia@localhost:Linux =
2.6.9-11.ELsmp:/homes/jobia/p4/R21 &gt; p4 sync ...<BR>Perforce client =
error:<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; TCP receive =
failed.<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; read: socket: =
Connection reset by =
peer<BR><BR><BR>Regards,<BR>Jobi<BR><BR></FONT></P></DIV></BODY></HTML>
------_=_NextPart_001_01C718FD.4314DDC1--
