AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20070830192206.6fa99b08@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<ken.renshaw@onstor.com>,<kevin.matthews@onstor.com>,<brian.baker@onstor.com>,<vikas.saini@onstor.com>,<dl-qa>,<maxim.kozlovsky@onstor.com>,<jonathan.goldick@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@onstor-exch02.onstor.net/INBOX	0	BB375AF679D4A34E9CA8DFA650E2B04E02EB2FFD@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Thu, 30 Aug 2007 19:23:09 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: "Ken Renshaw" <ken.renshaw@onstor.com>
Cc: "Kevin Matthews" <kevin.matthews@onstor.com>, "Brian Baker"
 <brian.baker@onstor.com>, "Vikas Saini" <vikas.saini@onstor.com>, dl-qa,
 Maxim Kozlovsky <maxim.kozlovsky@onstor.com>, Jonathan Goldick
 <jonathan.goldick@onstor.com>
Subject: Re: Cron <andys@ripper> cd /home/andy/git;
 /home/andy/bin/git_rsync.sh
Message-ID: <20070830192309.6465bc5a@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E02EB2FFD@onstor-exch02.onstor.net>
References: <BB375AF679D4A34E9CA8DFA650E2B04E02EB2FFD@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

Fascinating.  When mightydog is being a slow bitch (instead of a mighty
dog; get it?), ping times go straight and steady to 9.x ms, and when
it's being mighty again, they go right to sub 1 ms.  Like clockwork.

On Thu, 30 Aug 2007 18:50:55 -0700 "Ken Renshaw"
<ken.renshaw@onstor.com> wrote:

> Yes, we've seen spurious periods of. Low bandwidth access to MD in
> the past. No, we've never figured it out.
> 
> There was an ICMP problem that caused this sort of slowness, and
> we've also seen bad routes on the corporate routers cause it. 
> 
> Does the output of ping -s 10.0.0.222 look anomolous? Traceroute? Or
> is it purely the file access times?
> 
> Neither of those things has proven fruitful to date, but since what
> you're testing seems reproducible hopefully IT can chime in
> here...guys??
> 
> Thanks,
> 
> -Ken
> 
>  
> 
> -----Original Message-----
> From: Andy Sharp
> To: Ken Renshaw
> CC: Vikas Saini
> Sent: Thu Aug 30 18:14:31 2007
> Subject: Re: Cron <andys@ripper>
> cd /home/andy/git; /home/andy/bin/git_rsync.sh
> 
> The cron job just ran again about 30 minutes ago, this time it took 45
> seconds instead of 9.5 minutes.  As you can see, about a 10x
> difference in speed.  Sometin' 'zup.
> 
> On Thu, 30 Aug 2007 18:06:01 -0700 "Ken Renshaw"
> <ken.renshaw@onstor.com> wrote:
> 
> > Have you checked the sticky bits Andy?
> > 
> > 
> >  
> > 
> > -----Original Message-----
> > From: Vikas Saini
> > To: dl-qa <dl-qa>
> > Sent: Thu Aug 30 17:34:04 2007
> > Subject: RE: Cron <andys@ripper>
> > cd /home/andy/git; /home/andy/bin/git_rsync.sh
> > 
> > Ahaaaa !!!! sticky bit again... 
> > 
> > 
> > 
> > 
> > 
> > -----Original Message-----
> > From: Andy Sharp 
> > Sent: Thursday, August 30, 2007 5:07 PM
> > To: dl-qa; Kevin Matthews
> > Subject: Fw: Cron <andys@ripper>
> > cd /home/andy/git; /home/andy/bin/git_rsync.sh
> > 
> > Howdy QA & IT folks,
> > 
> > I'm seeing some issues with mightydog that I thought I would run by
> > y'all to see if anyone can shed some light.
> > 
> > I've been seeing some very poor performance, for a lot of yesterday,
> > and again I noticed it about 10 minutes ago.  Below is the record of
> > an rsync from my hard drive to my home directory which started at
> > 4:45. You can see it took about 9 1/2 minutes for a grand total of
> > about 40KB/sec. That's pretty slow even for a busy filer like
> > mightydog.
> > 
> > Yesterday afternoon, and continuing into the evening, I was
> > modifying and building a linux kernel repeatedly.  Editing a single
> > file, making a couple of changes, and then typing 'make', which
> > should take from 15-30 seconds depending if the filer is busy.
> > Instead I would say the average time was more than 5 minutes.
> > 
> > Anybody have a clue?  How can I go about finding out what's buggin'
> > mightydog?
> > 
> > Thanks,
> > 
> > a
> > 
> > Begin forwarded message:
> > 
> > Date: Thu, 30 Aug 2007 16:54:27 -0700
> > From: andys.sharp@onstor.com (Cron Daemon)
> > To: andy.sharp@onstor.com
> > Subject: Cron <andys@ripper>
> > cd /home/andy/git; /home/andy/bin/git_rsync.sh
> > 
> > 
> > Backing up directory /home/andy/git
> > PING mightydog.onstor.net (10.0.0.222) 56(84) bytes of data.
> > 64 bytes from mightydog.onstor.net (10.0.0.222): icmp_seq=1 ttl=255
> > time=1.88 ms
> > 
> > --- mightydog.onstor.net ping statistics ---
> > 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> > rtt min/avg/max/mdev = 1.880/1.880/1.880/0.000 ms
> > + nice -15 rsync -uvHax --delete . /homes/andys/.git
> > building file list ... done
> > 
> > sent 22122606 bytes  received 20 bytes  39051.41 bytes/sec
> > total size is 3915591381  speedup is 176.99
> > + set +x
> > Backup to mightydog finished at Thu Aug 30 16:54:27 PDT 2007 with
> > exit code 0
