AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20090205140648.798fe666@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:exch1.onstor.net
NSV:
SSH:
R:<david.crispin@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@exch1.onstor.net/INBOX	0	2779531E7C760D4491C96305019FEEB51851E65C79@exch1.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Thu, 5 Feb 2009 14:07:31 -0800
From: Andrew Sharp <andy.sharp@onstor.com>
To: David Crispin <david.crispin@onstor.com>
Subject: Re: Bobcat slow to initialize and some system commands run slowly
Message-ID: <20090205140731.6ae3e9e0@ripper.onstor.net>
In-Reply-To: <2779531E7C760D4491C96305019FEEB51851E65C79@exch1.onstor.net>
References: <2779531E7C760D4491C96305019FEEB51851E65C79@exch1.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

On Thu, 5 Feb 2009 06:42:49 -0800 David Crispin
<david.crispin@onstor.com> wrote:

> A customer called me because they lost access to their NFS share. On
> a Webex session I logged into the Bobcat with Putty and it took a
> long time for the prompt to come up after entering the password. All
> vsvrs and volume were online. I ran a system get all -c 11557 and it
> took a long time to run and came up with some messages. This was on
> 3.3.2.0. When it finished I rebooted the gateway expecting it to come
> up as usual but again login was slow. The NFS share was mountable
> after the reboot.
> 
> 
> I ran the top command and there were 65 processes, 1 running and 64
> idle. The CPU % was low. I decided to upgrade to 3.3.2.6. The first
> attempt hung and I lost my network connection. I logged back in and
> ran system upgrade again but it hung. I had to open another session
> and try a system reboot but that gave me the rmc_pm_dispatch():
> sess{:pm.0.0} down and I had to reboor from the BSD prompt. I got it
> upgraded to 3.3.2.6 but it still takes ages to initialize and login
> and a SGA gives these messages:-

The first thing I would look for is one or more of our daemons that are
taking up a large amount of memory.  If that is slowing the system
down, then you should see a "large" load average reported in top,
something above 1.0.  You would also see free memory reported as very
low, below 1MB maybe.

The messages you report below, I thought those were cleared up in 3.3.0
timeframe.  While overly chatty, they don't indicate any problem.

> 
> 
> nas02> system get all -c 11557
> 
>     Looking for virtual servers and volumes..
> 
>     Querying the
> system....................................................................................................................................................................................................................................................................................................................
> OK
> 
>     Uploading file sfinfo-20090205_05032522.xml.gz.. [-1] + Done
> (143)           ( sleep $TO ; gkill "$MRHUNG" && kill -USR1 $MO
> 
>  OK
> 
>     Uploading elog files..                  [-1] + Done
> (143)           ( sleep $TO ; gkill "$MRHUNG" && kill -USR1 $MO
> 
>  OK
> 
>     Uploading elog files from secondary flash.. [-1] + Done
> (143)           ( sleep $TO ; gkill "$MRHUNG" && kill -USR1 $MO
> 
>  OK
> 
>     Uploading syslog files..                [-1] + Done
> (143)           ( sleep $TO ; gkill "$MRHUNG" && kill -USR1 $MO
> 
>  OK
> 
>     Uploading auth log files..              [-1] + Done
> (143)           ( sleep $TO ; gkill "$MRHUNG" && kill -USR1 $MO
> 
>  OK
> 
>     Uploading ndmp log files..              [-1] + Done
> (143)           ( sleep $TO ; gkill "$MRHUNG" && kill -USR1 $MO
> 
>  OK
> 
>     Uploading quota log file..              [-1] + Done
> (143)           ( sleep $TO ; gkill "$MRHUNG" && kill -USR1 $MO
> 
>  OK
> 
>     Uploading cron log files..              [-1] + Done
> (143)           ( sleep $TO ; gkill "$MRHUNG" && kill -USR1 $MO
> 
>  OK
> 
>     Uploading kpi stats..                   [-1] + Done
> (143)           ( sleep $TO ; gkill "$MRHUNG" && kill -USR1 $MO
> 
>  OK
> 
>     Uploading cron tabs..                   [-1] + Done
> (143)           ( sleep $TO ; gkill "$MRHUNG" && kill -USR1 $MO
> 
>  OK
> 
>     Uploading crash files..                 [-1] + Done
> (143)           ( sleep $TO ; gkill "$MRHUNG" && kill -USR1 $MO
> 
>  OK
> 
>     Uploading core files..                   SKIPPED (No files to
> copy)
> 
>     Uploading cluster DB..                  [-1] + Done
> (143)           ( sleep $TO ; gkill "$MRHUNG" && kill -USR1 $MO
> 
>  OK
> 
> nas02>
> 
> 
> 
> 
> 
> Does anyone have an idea why this system seems "slow"
> 
> Regards,
> 
> [cid:image001.jpg@01C9879F.E0AAF5A0]
> David Crispin
> Technical Services Manager
> ONStor UK
> 
> office: +44 (0)1483 804822
> mobile: +44 (0)7940 547895
> 
> david.crispin@onstor.com<mailto:q@onstor.com>
> http://www.onstor.com<http://www.onstor.com/>
> 
