X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C7963B.661E7620@onstor-exch02.onstor.net>; Mon, 14 May 2007 08:20:26 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C7963B.661E7620"
Content-class: urn:content-classes:message
Subject: RE: ssh configuration (Defect 18513)
Date: Mon, 14 May 2007 08:20:26 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E03B1BD8B@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E03B1BD8A@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: ssh configuration (Defect 18513)
Thread-Index: AceV/BMaMWqJvdKlTQW3ZDYeAODqCQAAKOSOAA95iFAAAC1VkA==
From: "Rendell Fong" <rendell.fong@onstor.com>
To: "Mike Lee" <mike.lee@onstor.com>
Cc: "Andy Sharp" <andy.sharp@onstor.com>,
	"Larry Scheer" <larry.scheer@onstor.com>,
	"Brian DeForest" <brian.deforest@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C7963B.661E7620
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

=20

________________________________

From: Rendell Fong=20
Sent: Monday, May 14, 2007 8:19 AM
To: Mike Lee
Subject: RE: ssh configuration (Defect 18513)

=20

The QA expect script is just using ssh to login as root. nfxsh is not
being used at all.

Because of a bug in their script it doesn't logout after killing off a
bunch of processes.=20

So each iteration of their their test adds an additional ssh login.=20

=20

________________________________

From: Mike Lee=20
Sent: Monday, May 14, 2007 12:52 AM
To: Andy Sharp; Larry Scheer
Cc: Brian DeForest; Rendell Fong; Sandrine Boulanger; Tim Gardner
Subject: RE: ssh configuration (Defect 18513)

=20

Andy:

=20

I'll read over the QA script more carefully tomorrow, but I don't recall
it kicking off nfxsh.

I think all it does is start an ssh session, in which a process ID
search is done, followed by a kill.

However, I could be wrong...

=20

Got a bad headache at the moment, so will have to call it a night now.

=20

Thanks.

=20

-Mike

=20

________________________________

From: Andy Sharp
Sent: Mon 5/14/2007 12:47 AM
To: Larry Scheer
Cc: Mike Lee; Brian DeForest; Rendell Fong; Sandrine Boulanger; Tim
Gardner
Subject: Re: ssh configuration (Defect 18513)

Something just doesn't add up here.  40KB per ssh login?  I guarantee
it's a lot more than that.  But Larry's point still persists, even if
it's 10 times that, it doesn't make sense that there are 1000 ssh
sessions going.  Are there?  The amount of memory used by each instance
of nfxsh is going to be in the megabytes per, and that's before you run
any commands.  Even if you add that to the ssh usage, it still sounds
wonky.

Something is amiss or missing with this analysis.

Cheers,

a


On Sun, 13 May 2007 21:36:12 -0700 "Larry Scheer"
<larry.scheer@onstor.com> wrote:

> How many concurrent SSH connections were there?
> At 40Kbytes of memory each session, you would need 6554 sessions
> running to exhaust 256Mbytes of memory. Are you saying the real
> problem is a runaway process spawning SSH connections?
>
> When do we ever have dozens of SSH processes running running on the
> SSC? I can't imagine hundreds much less thousands of SSH processes.
> What are seeing that I am missing here?
>
> Larry
>
> -----Original Message-----
> From: Mike Lee
> Sent: Sun 5/13/2007 8:29 PM
> To: Andy Sharp; Larry Scheer
> Cc: Brian DeForest; Rendell Fong; Sandrine Boulanger; Tim Gardner
> Subject: ssh configuration (Defect 18513)
>=20
> Gentlemen:
>
> Concerning that BSD panic due to kernel memory exhaustion, Rendell
> figured out that it was due to too many concurrent ssh connections to
> our filer, where each connection ate up 40K of memory.=20
>
> As such, I think we need to configure our ssh daemon to limit the
> maximum number of concurrent connections.  I searched a bit online
> and the only thing I found was the MaxStartups setting, but it is for
> "concurrent unauthenticated connections".=20
>
> Do you know of a way to limit number of connections, authenticated or
> unauthenticated?
>
> Thanks!
>
> -Mike
>
>
> MaxStartups
> Specifies the maximum number of concurrent unauthenticated
> connections to the sshd daemon. Additional connections will be
> dropped until authentication succeeds or the LoginGraceTime expires
> for a connection. The default is 10. Alternatively, random early drop
> can be enabled by specifying the three colon separated values
> ``start:rate:full'' (e.g., "10:30:60"). sshd will refuse connection
> attempts with a probability of ``rate/100'' (30%) if there are
> currently ``start'' (10) unauthenticated connections. The probability
> increases linearly and all connection attempts are refused if the
> number of unauthenticated connections reaches ``full'' (60).
>


------_=_NextPart_001_01C7963B.661E7620
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<title>Re: ssh configuration (Defect 18513)</title>
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:navy;
	text-decoration:underline;}
p
	{mso-margin-top-alt:auto;
	margin-right:0in;
	mso-margin-bottom-alt:auto;
	margin-left:0in;
	font-size:12.0pt;
	font-family:"Times New Roman";}
span.EmailStyle18
	{mso-style-type:personal;
	font-family:Arial;
	color:navy;}
span.EmailStyle19
	{mso-style-type:personal-reply;
	font-family:Arial;
	color:navy;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
	{page:Section1;}
-->
</style>

</head>

<body lang=3DEN-US link=3Dblue vlink=3Dnavy>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> =
Rendell Fong <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, May 14, =
2007 8:19 AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Mike Lee<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: ssh =
configuration
(Defect 18513)</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>The QA expect script is just using =
ssh to
login as root. nfxsh is not being used at =
all.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Because of a bug in their script it
doesn&#8217;t logout after killing off a bunch of processes. =
<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>So each iteration of their their =
test adds
an additional ssh login. <o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p>

<div>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1>

</span></font></div>

<p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font =
size=3D2
face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> Mike =
Lee <br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, May 14, =
2007 12:52
AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Andy Sharp; Larry =
Scheer<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Brian DeForest; =
Rendell Fong;
Sandrine Boulanger; Tim Gardner<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> RE: ssh =
configuration
(Defect 18513)</span></font><o:p></o:p></p>

</div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div id=3DidOWAReplyText70864>

<div>

<p class=3DMsoNormal><font size=3D2 color=3Dblack face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:black'>Andy:</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>I'll read over the QA script more carefully tomorrow, =
but I
don't recall it kicking off nfxsh.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>I think all it does is start an ssh session, in which =
a
process ID search is done, followed by a =
kill.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>However, I could be =
wrong...</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Got a bad headache at the moment, so will have to =
call it a
night now.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Thanks.</span></font><o:p></o:p></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'>&nbsp;<o:p></o:p></span></font></p>

</div>

<div>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>-Mike</span></font><o:p></o:p></p>

</div>

</div>

<div>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

<div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font =
size=3D3
face=3D"Times New Roman"><span style=3D'font-size:12.0pt'>

<hr size=3D2 width=3D"100%" align=3Dcenter tabIndex=3D-1>

</span></font></div>

<p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><b><font size=3D2 =
face=3DTahoma><span
style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa=
n></font></b><font
size=3D2 face=3DTahoma><span =
style=3D'font-size:10.0pt;font-family:Tahoma'> Andy
Sharp<br>
<b><span style=3D'font-weight:bold'>Sent:</span></b> Mon 5/14/2007 12:47 =
AM<br>
<b><span style=3D'font-weight:bold'>To:</span></b> Larry Scheer<br>
<b><span style=3D'font-weight:bold'>Cc:</span></b> Mike Lee; Brian =
DeForest;
Rendell Fong; Sandrine Boulanger; Tim Gardner<br>
<b><span style=3D'font-weight:bold'>Subject:</span></b> Re: ssh =
configuration
(Defect 18513)</span></font><o:p></o:p></p>

</div>

<div>

<p><font size=3D2 face=3D"Times New Roman"><span =
style=3D'font-size:10.0pt'>Something
just doesn't add up here.&nbsp; 40KB per ssh login?&nbsp; I =
guarantee<br>
it's a lot more than that.&nbsp; But Larry's point still persists, even =
if<br>
it's 10 times that, it doesn't make sense that there are 1000 ssh<br>
sessions going.&nbsp; Are there?&nbsp; The amount of memory used by each
instance<br>
of nfxsh is going to be in the megabytes per, and that's before you =
run<br>
any commands.&nbsp; Even if you add that to the ssh usage, it still =
sounds<br>
wonky.<br>
<br>
Something is amiss or missing with this analysis.<br>
<br>
Cheers,<br>
<br>
a<br>
<br>
<br>
On Sun, 13 May 2007 21:36:12 -0700 &quot;Larry Scheer&quot;<br>
&lt;larry.scheer@onstor.com&gt; wrote:<br>
<br>
&gt; How many concurrent SSH connections were there?<br>
&gt; At 40Kbytes of memory each session, you would need 6554 =
sessions<br>
&gt; running to exhaust 256Mbytes of memory. Are you saying the real<br>
&gt; problem is a runaway process spawning SSH connections?<br>
&gt;<br>
&gt; When do we ever have dozens of SSH processes running running on =
the<br>
&gt; SSC? I can't imagine hundreds much less thousands of SSH =
processes.<br>
&gt; What are seeing that I am missing here?<br>
&gt;<br>
&gt; Larry<br>
&gt;<br>
&gt; -----Original Message-----<br>
&gt; From: Mike Lee<br>
&gt; Sent: Sun 5/13/2007 8:29 PM<br>
&gt; To: Andy Sharp; Larry Scheer<br>
&gt; Cc: Brian DeForest; Rendell Fong; Sandrine Boulanger; Tim =
Gardner<br>
&gt; Subject: ssh configuration (Defect 18513)<br>
&gt;&nbsp;<br>
&gt; Gentlemen:<br>
&gt;<br>
&gt; Concerning that BSD panic due to kernel memory exhaustion, =
Rendell<br>
&gt; figured out that it was due to too many concurrent ssh connections =
to<br>
&gt; our filer, where each connection ate up 40K of memory.&nbsp;<br>
&gt;<br>
&gt; As such, I think we need to configure our ssh daemon to limit =
the<br>
&gt; maximum number of concurrent connections.&nbsp; I searched a bit =
online<br>
&gt; and the only thing I found was the MaxStartups setting, but it is =
for<br>
&gt; &quot;concurrent unauthenticated connections&quot;.&nbsp;<br>
&gt;<br>
&gt; Do you know of a way to limit number of connections, authenticated =
or<br>
&gt; unauthenticated?<br>
&gt;<br>
&gt; Thanks!<br>
&gt;<br>
&gt; -Mike<br>
&gt;<br>
&gt;<br>
&gt; MaxStartups<br>
&gt; Specifies the maximum number of concurrent unauthenticated<br>
&gt; connections to the sshd daemon. Additional connections will be<br>
&gt; dropped until authentication succeeds or the LoginGraceTime =
expires<br>
&gt; for a connection. The default is 10. Alternatively, random early =
drop<br>
&gt; can be enabled by specifying the three colon separated values<br>
&gt; ``start:rate:full'' (e.g., &quot;10:30:60&quot;). sshd will refuse
connection<br>
&gt; attempts with a probability of ``rate/100'' (30%) if there are<br>
&gt; currently ``start'' (10) unauthenticated connections. The =
probability<br>
&gt; increases linearly and all connection attempts are refused if =
the<br>
&gt; number of unauthenticated connections reaches ``full'' (60).<br>
&gt;</span></font><o:p></o:p></p>

</div>

</div>

</body>

</html>

------_=_NextPart_001_01C7963B.661E7620--
