X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: by onstor-exch02.onstor.net 
	id <01C8B60A.8FBE937A@onstor-exch02.onstor.net>; Wed, 14 May 2008 14:36:25 -0700
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
Subject: RE: Defect  SW-BSD Opened TED00023791
Date: Wed, 14 May 2008 14:36:25 -0700
Message-ID: <BB375AF679D4A34E9CA8DFA650E2B04E03E9A865@onstor-exch02.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E09EE845A@onstor-exch02.onstor.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Defect  SW-BSD Opened TED00023791
Thread-Index: Aci2BtTJKjsjhoXeRyWXATcxyCFOOQAAmpOQAAAXbJAAABq34AAAD4Cw
From: "Chris Vandever" <chris.vandever@onstor.com>
To: "Raj Kumar" <raj.kumar@onstor.com>,
	"Andy Sharp" <andy.sharp@onstor.com>

Andy, is there some way we can find out what processes are running?  Can
we access /dev/proc or whatever directly?  It looks to me like we're out
of memory, so how is an administrator supposed to recover without
rebooting?

ChrisV

-----Original Message-----
From: Raj Kumar=20
Sent: Wednesday, May 14, 2008 2:33 PM
To: Chris Vandever
Subject: RE: Defect SW-BSD Opened TED00023791

Even ps fails.

# ps
sh: cannot fork - try again




-----Original Message-----
From: Chris Vandever=20
Sent: Wednesday, May 14, 2008 2:32 PM
To: Raj Kumar
Subject: RE: Defect SW-BSD Opened TED00023791

We're out of memory.  There are too many processes running.  Do a ps and
see, but I suspect we have a bunch of emrs processes wedged.  'sh' will
fail when we're out of memory.  -13 error from pm is RMC_NOMEM.

ChrisV

-----Original Message-----
From: Raj Kumar=20
Sent: Wednesday, May 14, 2008 2:28 PM
To: Chris Vandever
Subject: FW: Defect SW-BSD Opened TED00023791

Since you are looking at the elogs, there is already a defect for the pm
errors that you will see on g8r9.

-----Original Message-----
From: raj.kumar@onstor.com [mailto:raj.kumar@onstor.com]=20
Sent: Wednesday, May 14, 2008 2:10 PM
To: Andy Sharp
Cc: Raj Kumar
Subject: Defect SW-BSD Opened TED00023791

id: TED00023791
Headline: S-Soak (G8R9): BSD can not fork any more processes (May 14
14:07:59 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs)
Severity: 2-Major
Build: Submittal 20 Beta
Description: Submittal : 20_BETA
Setup: SS
Node: G8r9
Elog at /n/newcorevol/defect_23791

BSD on thsi particular node is not able to fork any more processes.=20

I was trying to get a SGA on this node and the CLI failed. Then I
noticed several pm related messages on the elog. When I tried to look at
process list using ps, ps failed.

I wonder whether this is due to the fact that I have startedusing NCM on
this node or not.

# ps ax | grep onstor
sh: cannot fork - try again
# Connection to g8r9 closed.

g8r9 diag> system get all
% Command failure.

# nfxsh

sh: cannot fork - try again

************** Elog*********

May 14 14:07:59 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:07:59 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:00 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:00 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:01 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:01 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:02 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:02 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:03 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:03 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:04 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:04 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:05 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:05 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:06 g2r5-2280.onstor.lab : 0:0:cluster2:INFO:
Cluster_SendMsgSock: sendto to 10.4.1.1 failed, msgId 10452, code 64
(Host is down)
May 14 14:08:06 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:06 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:07 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:07 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:09 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:09 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:10 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:10 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:11 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:11 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:12 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:12 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:13 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:13 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:14 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:14 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13
May 14 14:08:16 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_get_procs: not
enough pid entries, got(512) need(521)
May 14 14:08:16 g8r9-2260.onstor.lab : 0:0:pm:WARNING: pm_timeout_work:
pm_get_procs failed, -13


Release_Project: Cougar


