X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C795FC.1C84859B@onstor-exch02.onstor.net>; Mon, 14 May 2007 00:47:24 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C795FC.1C84859B"
References: <BB375AF679D4A34E9CA8DFA650E2B04E028FB43E@onstor-exch02.onstor.net> <BB375AF679D4A34E9CA8DFA650E2B04E0A91F0@onstor-exch02.onstor.net>
Content-class: urn:content-classes:message
Subject: RE: ssh configuration (Defect 18513)
Date: Mon, 14 May 2007 00:47:24 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E028FB441@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: ssh configuration (Defect 18513)
Thread-Index: AceV2BIgLc8ojXfxQBSg1eKJfl/8lAAB0bnwAAb6wGs=
From: "Mike Lee" <mike.lee@onstor.com>
To: "Larry Scheer" <larry.scheer@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>
Cc: "Brian DeForest" <brian.deforest@onstor.com>,
	"Rendell Fong" <rendell.fong@onstor.com>,
	"Sandrine Boulanger" <sandrine.boulanger@onstor.com>,
	"Tim Gardner" <tim.gardner@onstor.com>

This is a multi-part message in MIME format.

------_=_NextPart_001_01C795FC.1C84859B
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Larry:
=20
There were probably a few hundred concurrent SSH sessions at the most =
when the filer crashed.
I am basing my guess on the fact that the script spawn a new SSH session =
every 5 seconds, and it took less than 10 minutes of the script running =
to kill the filer.
=20
Perhaps "vmstat -m" is not the correct way to measure free kernel =
memory; some of the websites I searched suggested this command.  =
However, when reboot my filer and run "vmstat -m" the amount of free =
memory reported usually is only 200K - 300K, which seems awfully low, =
based on your insights.
=20
I agree on the unlikeliness of the test scenario occuring in real life.  =

Thus, I have raised the issue of sending this defect back to triage.=20
=20
I'll have to brainstorm with you some more tomorrow.
=20
Thanks!
=20
-Mike

________________________________

From: Larry Scheer
Sent: Sun 5/13/2007 9:36 PM
To: Mike Lee; Andy Sharp
Cc: Brian DeForest; Rendell Fong; Sandrine Boulanger; Tim Gardner
Subject: RE: ssh configuration (Defect 18513)



How many concurrent SSH connections were there?
At 40Kbytes of memory each session, you would need 6554 sessions running =
to exhaust 256Mbytes of memory. Are you saying the real problem is a =
runaway process spawning SSH connections?

When do we ever have dozens of SSH processes running running on the SSC? =
I can't imagine hundreds much less thousands of SSH processes. What are =
seeing that I am missing here?

Larry

-----Original Message-----
From: Mike Lee
Sent: Sun 5/13/2007 8:29 PM
To: Andy Sharp; Larry Scheer
Cc: Brian DeForest; Rendell Fong; Sandrine Boulanger; Tim Gardner
Subject: ssh configuration (Defect 18513)

Gentlemen:

Concerning that BSD panic due to kernel memory exhaustion, Rendell =
figured out that it was due to too many concurrent ssh connections to =
our filer, where each connection ate up 40K of memory.=20

As such, I think we need to configure our ssh daemon to limit the =
maximum number of concurrent connections.  I searched a bit online and =
the only thing I found was the MaxStartups setting, but it is for =
"concurrent unauthenticated connections".=20

Do you know of a way to limit number of connections, authenticated or =
unauthenticated?

Thanks!

-Mike


MaxStartups
Specifies the maximum number of concurrent unauthenticated connections =
to the sshd daemon. Additional connections will be dropped until =
authentication succeeds or the LoginGraceTime expires for a connection. =
The default is 10.
Alternatively, random early drop can be enabled by specifying the three =
colon separated values ``start:rate:full'' (e.g., "10:30:60"). sshd will =
refuse connection attempts with a probability of ``rate/100'' (30%) if =
there are currently ``start'' (10) unauthenticated connections. The =
probability increases linearly and all connection attempts are refused =
if the number of unauthenticated connections reaches ``full'' (60).




------_=_NextPart_001_01C795FC.1C84859B
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<HTML dir=3Dltr><HEAD><TITLE>RE: ssh configuration (Defect 18513)</TITLE>=0A=
<META http-equiv=3DContent-Type content=3D"text/html; charset=3Dunicode">=0A=
<META content=3D"MSHTML 6.00.2800.1589" name=3DGENERATOR></HEAD>=0A=
<BODY>=0A=
<DIV id=3DidOWAReplyText55801 dir=3Dltr>=0A=
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 =
size=3D2>Larry:</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>There were probably a few =
hundred concurrent SSH sessions at the most when the filer =
crashed.</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>I am basing my guess on the =
fact that the script spawn a new SSH session every 5 seconds, and it =
took less than 10 minutes of the script running to kill the =
filer.</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>Perhaps "vmstat -m" is not =
the correct way to measure free kernel memory; some of the websites I =
searched suggested this command.&nbsp; However, when reboot my filer and =
run "vmstat -m" the amount of free memory reported usually is only 200K =
- 300K, which seems awfully low, based on your insights.</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>I agree on the unlikeliness =
of the test scenario occuring in real life.&nbsp; </FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>Thus, I&nbsp;have&nbsp;raised =
the issue of sending this defect back to triage.</FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>I'll have to brainstorm with =
you some more tomorrow.</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>Thanks!</FONT></DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>=0A=
<DIV dir=3Dltr><FONT face=3DArial size=3D2>-Mike</FONT></DIV></DIV>=0A=
<DIV dir=3Dltr><BR>=0A=
<HR tabIndex=3D-1>=0A=
<FONT face=3DTahoma size=3D2><B>From:</B> Larry Scheer<BR><B>Sent:</B> =
Sun 5/13/2007 9:36 PM<BR><B>To:</B> Mike Lee; Andy Sharp<BR><B>Cc:</B> =
Brian DeForest; Rendell Fong; Sandrine Boulanger; Tim =
Gardner<BR><B>Subject:</B> RE: ssh configuration (Defect =
18513)<BR></FONT><BR></DIV>=0A=
<DIV>=0A=
<P><FONT size=3D2>How many concurrent SSH connections were there?<BR>At =
40Kbytes of memory each session, you would need 6554 sessions running to =
exhaust 256Mbytes of memory. Are you saying the real problem is a =
runaway process spawning SSH connections?<BR><BR>When do we ever have =
dozens of SSH processes running running on the SSC? I can't imagine =
hundreds much less thousands of SSH processes. What are seeing that I am =
missing here?<BR><BR>Larry<BR><BR>-----Original Message-----<BR>From: =
Mike Lee<BR>Sent: Sun 5/13/2007 8:29 PM<BR>To: Andy Sharp; Larry =
Scheer<BR>Cc: Brian DeForest; Rendell Fong; Sandrine Boulanger; Tim =
Gardner<BR>Subject: ssh configuration (Defect =
18513)<BR><BR>Gentlemen:<BR><BR>Concerning that BSD panic due to kernel =
memory exhaustion, Rendell figured out that it was due to too many =
concurrent ssh connections to our filer, where each connection ate up =
40K of memory.&nbsp;<BR><BR>As such, I think we need to configure our =
ssh daemon to limit the maximum number of concurrent connections.&nbsp; =
I searched a bit online and the only thing I found was the MaxStartups =
setting, but it is for "concurrent unauthenticated =
connections".&nbsp;<BR><BR>Do you know of a way to limit number of =
connections, authenticated or =
unauthenticated?<BR><BR>Thanks!<BR><BR>-Mike<BR><BR><BR>MaxStartups<BR>Sp=
ecifies the maximum number of concurrent unauthenticated connections to =
the sshd daemon. Additional connections will be dropped until =
authentication succeeds or the LoginGraceTime expires for a connection. =
The default is 10.<BR>Alternatively, random early drop can be enabled by =
specifying the three colon separated values ``start:rate:full'' (e.g., =
"10:30:60"). sshd will refuse connection attempts with a probability of =
``rate/100'' (30%) if there are currently ``start'' (10) unauthenticated =
connections. The probability increases linearly and all connection =
attempts are refused if the number of unauthenticated connections =
reaches ``full'' (60).<BR><BR></FONT></P></DIV></BODY></HTML>
------_=_NextPart_001_01C795FC.1C84859B--
