AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20071004234936.5ceb5612@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<paul.hammer@onstor.com>,<brian.baker@onstor.com>,<sandrine.boulanger@onstor.com>,<brian.stark@onstor.com>,<raj.kumar@onstor.com>,<john.rogers@onstor.com>,<joshua.goldenhar@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@onstor-exch02.onstor.net/INBOX	0	BB375AF679D4A34E9CA8DFA650E2B04E05BFE50A@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Thu, 4 Oct 2007 23:50:04 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: "Paul Hammer" <paul.hammer@onstor.com>
Cc: "Brian Baker" <brian.baker@onstor.com>, "Sandrine Boulanger"
 <sandrine.boulanger@onstor.com>, "Brian Stark" <brian.stark@onstor.com>,
 "Raj Kumar" <raj.kumar@onstor.com>, "John Rogers" <john.rogers@onstor.com>,
 "Joshua Goldenhar" <joshua.goldenhar@onstor.com>
Subject: Re: networking betwixt camphell and pleztown
Message-ID: <20071004235004.5d8af9ff@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E05BFE50A@onstor-exch02.onstor.net>
References: <BB375AF679D4A34E9CA8DFA650E2B04E05DD73DC@onstor-exch02.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E068BA5@onstor-exch02.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05BFE50A@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

I think Paul is right, we are feeling our way and discovering quite a
lot as we go.  For instance, I didn't know we already had such high
quality routers.  That's great.

Here's my proposal:

1. Limit the DMIP to 3/4 of the link speed across the board.
2. Restrict the QA DMIP traffic to take place between 12am and 7am.
3. Mightydog volumes, if mirrored in Ptown, on an hourly mirroring
   schedule, should actually decrease the amount of traffic, because
   Pleasanton folksez won't need to use the link to access mightydog
   resident data.

I realize this may take more than a minute or two to put into place,
but let's try and make a plan as to what we can do by when.  Let me
know if there's anything I can do to help.  Of course if there is
anything really annoying that has to be done, well, that's what we
have Josh for.  That, and, the Riverbed thing.  Whatever that is.

Cheers,

a

 On Thu, 4 Oct 2007 23:29:09 -0700 "Paul Hammer"
<paul.hammer@onstor.com> wrote:

> Thanks Brian. The link saturatutaion was pure ignorance on our
> behalf, had no idea that DMIP would saturate the link to the point of
> hobbling productivity for the P'town team, this goes for both QA and
> Corporate, perhaps this is why Kevin never put the MD boxes in P'town
> 7 months ago as requested by Frank. Great point on the testing.
> Sandrine and John can you propose how we setup the DMIP feature for
> testing in Soak? I am okay with all of the suggestions that Brian
> listed. Thanks Brain, we did not know what impact this would have on
> the link for this test nor for MD. Now I understand a lot more about
> it. Thanks. John, Raj  and Sandrine lets work out what makes sense
> for SS and for MD with the link capacity in mind. Still curious
> though on the RiverBed angle. Thanks Brian. Go Rockies!! 2 Down and 1
> to Go! -Paul
> 
> ________________________________
> 
> From: Brian Baker
> Sent: Thu 10/4/2007 11:16 PM
> To: Paul Hammer; Andy Sharp
> Cc: Sandrine Boulanger; Brian Stark; Raj Kumar; John Rogers; Joshua
> Goldenhar Subject: RE: networking betwixt camphell and pleztown
> 
> 
> Paul,
> We already have layer 3/4 switches that support QoS . If DMIP
> generates this traffic then I will address it. I would use a method,
> per-port ingress-based enforcement bandwidth maximums. I'm assuming
> we won't have these spikes via DMIP for Mightydog as the baselines
> will be done here. No new network equipment needed no need to break
> out the checkbook. I think we need to revisit the testing that is
> done across the Pleasanton link. Why couldn't this testing be done
> with a DSL, cable line or t1 line that is hooked into the lab? This
> would give you the ability to test with different network equipment
> in front of your wan interfaces. John Rogers also mentioned that he
> was working on acquiring software to mimic wan speeds. I think this
> is the direction you need to take opposed to getting a additional
> line in Pleasanton. There shouldn't be a need for the geographic
> disparity, you just need the hops and latency to make it look like
> you aren't in the same room. DMIP will be considered general
> productivity traffic for this link and I will take it into account as
> I measure network traffic. Our problem in this instance was not
> understanding the impact of this test. I could have easily been
> someone downloading 1000's of torrents or 30GB of CIFS copies. If you
> give me some warning when utilizing corporate resources I can
> accommodate some traffic shaping. I only need some time to implement
> the solution and forewarning in the future. Brian 
> 
> ________________________________
> 
> From: Paul Hammer
> Sent: Thu 10/4/2007 7:06 PM
> To: Andy Sharp
> Cc: Brian Baker; Sandrine Boulanger; Brian Stark; Raj Kumar; John
> Rogers; Joshua Goldenhar Subject: RE: networking betwixt camphell and
> pleztown
> 
> 
> 
> Hey Andy,
> 
> Agree.
> 
> We need to plan accordingly for the MightyDog corporate DMIP
> session's that we plan to start soon.
> 
> Looks like we should have a rigid nightly sessions scheduled (so we
> have predicable perf hit). Else we need a bigger or different
> pipe/approach in place. A router may be the way to go, also
> interested in how a Riverbed deployment may be able to help us here.
> Added Josh for his perspective on this point.
> 
> Bottom line is we cannot negatively impact the Cougar schedule or our
> general productivity on this link with DMIP, so the solution has to
> take into account QoS as part of the solution.
> 
> Thanks,
> 
> -Paul
> 
> -----Original Message-----
> From: Andy Sharp
> Sent: Thursday, October 04, 2007 6:14 PM
> To: Paul Hammer
> Cc: Brian Baker; Sandrine Boulanger; Brian Stark; Raj Kumar
> Subject: Re: networking betwixt camphell and pleztown
> 
> The problem here, and the reason for this disconnect is that, IIUC, we
> are using a production asset for QA testing purposes.  I think this
> kind of testing is great -- but should we provision a separate link
> for it?  Or use a router that is capable of doing QoS itself to
> throttle back the DMIP traffic?
> 
> Having the line swamped between 8pm and 6am will still affect me
> greatly, sorry to say.
> 
> Cheers,
> 
> a
> 
> 
> On Thu, 4 Oct 2007 17:44:45 -0700 "Paul Hammer"
> <paul.hammer@onstor.com> wrote:
> 
> > Kind of funny, most folks want this thing to go way faster. I
> > understand why you would like slower.
> >
> > 
> >
> > 
> >
> > ________________________________
> >
> > From: Brian Baker
> > Sent: Thursday, October 04, 2007 3:59 PM
> > To: Sandrine Boulanger; Paul Hammer; Andy Sharp; Brian Stark
> > Cc: Raj Kumar
> > Subject: RE: networking betwixt camphell and pleztown
> >
> > 
> >
> > Sandrine,
> >
> > As you can see by the mrtg graph traffic has gone back down. As a
> > feature request to DMIP I would like to see bandwidth throttling on
> > the filer side.
> >
> > http://mrtg/mrtg/hp5308A/10.0.0.19_8.html
> >
> > 
> >
> > 
> >
> > From: Sandrine Boulanger
> > Sent: Thursday, October 04, 2007 3:50 PM
> > To: Paul Hammer; Brian Baker; Andy Sharp; Brian Stark
> > Cc: Raj Kumar
> > Subject: RE: networking betwixt camphell and pleztown
> >
> > 
> >
> > Is it better now?
> >
> > 
> >
> > I changed the schedules to run between 22 and 6am.
> >
> > I paused a mirror that was already at 66%, we can resume it tonight.
> >
> > I have an issue with one mirror for which I cannot modify the
> > schedule even if I do it from the GW that owns the vsvr. Mirror
> > show NAME finds the mirror but mirror schedule NAME does not...
> >
> > 
> >
> > Welcome to the ONStor NAS Gateway.
> >
> > 
> >
> > g2r5> vsvr show
> >
> > Virtual servers on nas gateway g2r5
> >
> > 
> >
> >  ID  State                             Name
> >
> > ====================================================
> >
> > 1    Enabled                           VS_MGMT_588
> >
> > 9    Enabled                           G2R5-VS1
> >
> > 10   Enabled                           G2R5-VS2
> >
> > 11   Enabled                           G2R5-VS3
> >
> > 12   Enabled                           G2R5-VS4
> >
> > 13   Enabled                           GNSSRVR
> >
> > 18   Enabled                           G2R5-VS5
> >
> > 19   Enabled                           G2R5-VS6
> >
> > g2r5> vsvr set G2R5-VS3
> >
> > g2r5 G2R5-VS3> mirror show
> >
> > 
> >
> >    Mirror Name    AdminState  Oper  State  Transferred      Source
> > Volume           Target Volume       VirtualServer
> >
> > ----------------  ----------  -----------  -----------
> > ----------------------  ----------------------  -------------
> >
> >   g2r5-v1v2-dmip     Enabled         Idle           0%
> > g2r5-vs1-vol2  g2r5-vs1-vol2@10.11.1.30       G2R5-VS1
> >
> >   g2r5-v2v1-dmip     Enabled         Idle           0%
> > g2r5-vs2-vol1  g2r5-vs2-vol1-dmip@10.11.1.30       G2R5-VS2
> >
> >   g2r5-v3v1-dmip     Enabled         Idle           0%
> > g2r5-vs3-vol1  g2r5-vs3-vol1-dmip@10.11.1.40       G2R5-VS3
> >
> >  g2r5vs5-vol1-m1     Enabled         Idle         100%
> > g2r5-vs5-vol1        g2r5-vs5-vol1-m1       G2R5-VS5
> >
> >  g2r5vs4-vol1-m1     Enabled         Idle           0%
> > g2r5-vs4-vol1         g2r5vs4-vol1-m1       G2R5-VS4
> >
> >  g2r5vs4-vol5-m1     Enabled         Idle         100%
> > g2r5-vs4-vol5         g2r5vs4-vol5-m1       G2R5-VS4
> >
> >   g4r8-v2v1-dmip     Enabled         Idle         100%
> > g4r8-vs2-vol1  g4r8-vs2-vol1-dmip@10.11.1.36       G4R8-VS2
> >
> >  g12r9-v1v1-dmip     Enabled       Paused          66%
> > g12r9-vs1-vol1  g12r9-vs1-vol1@10.11.1.36      G12R9-VS1
> >
> > g2r5 G2R5-VS3>
> >
> > g2r5 G2R5-VS3>
> >
> > g2r5 G2R5-VS3> mirror schedule g2r5-v3v1-dmip -h 0
> >
> >  /
> >
> > Error: Mirror[g2r5-v3v1-dmip] not found.
> >
> > % Command failure.
> >
> > g2r5 G2R5-VS3> elog show log 20
> >
> > Oct  4 15:35:16 g2r5 : 1:3:efs:INFO: 16516: FS: g2r5-vs5-vol1-m1
> > 0x24c000000dd - amDeltaInstallCommit - snp - snap resume: recal
> > refbyte snapcounts for 1
> >
> > Oct  4 15:35:16 g2r5 : 1:2:efs:INFO: 16517: FS: g2r5-vs5-vol1-m1
> > 0x24c000000dd - amDeltaInstallCommit - snp - snap resume: recal
> > refbyte snapcounts for 2
> >
> > Oct  4 15:35:16 g2r5 : 1:3:efs:INFO: 16518: FS: g2r5-vs5-vol1-m1
> > 0x24c000000dd - amDeltaInstallCommit - snp - snap resume: recal
> > refbyte snapcounts for 3
> >
> > Oct  4 15:35:19 g2r5 : 1:2:efs:INFO: 16519: FS: g2r5-vs5-vol1-m1
> > 0x24c000000dd - amDeltaInstallCommit - snp - snap revert mirror 3:
> > end
> >
> > Oct  4 15:35:19 g2r5 : 1:3:efs:INFO: 16520: FS: g2r5-vs5-vol1-m1
> > 0x24c000000dd - amDeltaInstallCommit - mirror - FS AM TARGET:
> > install commit reverted to snap 3 genNum 70 ( )
> >
> > Oct  4 15:35:19 g2r5 : 1:3:efs:INFO: 16521: FS: g2r5-vs5-vol1
> > 0x24c000000d9 - amDeltaComplete - mirror - FS AM: removing old
> > unused snapshot SANM_SS_g2r5vs5-vol1-m1_0000024c000000d9_0000019f 1
> > gen 70
> >
> > Oct  4 15:35:19 g2r5 : 1:2:efs:NOTICE: 16522: FS: g2r5-vs5-vol1
> > 0x24c000000d9 - amDeltaComplete - snp - snapnum 1 un pinned
> >
> > Oct  4 15:35:22 g2r5 : 1:3:efs:NOTICE: 16523: FS: g2r5-vs5-vol1
> > 0x24c000000d9 - amDeltaComplete - snp - snap remove complete for
> > SANM_SS_g2r5vs5-vol1-m1_0000024c000000d9_0000019f id 1
> >
> > Oct  4 15:35:22 g2r5 : 1:2:efs:INFO: 16524: FS: g2r5-vs5-vol1
> > 0x24c000000d9 - amDeltaComplete - mirror - FS AM: delta complete
> > session
> >
> > Oct  4 15:35:22 g2r5 : 0:0:sanm:NOTICE: Session terminated
> > successfully for mirror[g2r5vs5-vol1-m1]
> >
> > Oct  4 15:35:25 g2r5 : 0:0:sanm:NOTICE: SANM: Mirror Completed:
> > g2r5vs5-vol1-m1
> >
> > Oct  4 15:36:13 g2r5 : 0:0:sanm:NOTICE: sanm_procRemoteModifyRsp:
> > Error: Mirror[g2r5-v3v1-dmip] not found.
> >
> > Oct  4 15:36:13 g2r5 : 0:0:sanm:NOTICE: sanm_procRemoteModifyRsp:
> > remote modify failed, rc[-6654]
> >
> > Oct  4 15:36:46 g2r5 : 0:0:ssh:INFO: 'admin' logged in through
> > remote host(10.0.0.79)
> >
> > Oct  4 15:36:49 g2r5 : 0:0:nfxsh:NOTICE: cmd[0]: vsvr show :
> > status[0]
> >
> > Oct  4 15:36:54 g2r5 : 0:0:nfxsh:NOTICE: cmd[1]: vsvr set G2R5-VS3 :
> > status[0]
> >
> > Oct  4 15:36:59 g2r5 : 0:0:nfxsh:NOTICE: cmd[2]: mirror show  :
> > status[0]
> >
> > Oct  4 15:37:24 g2r5 : 0:0:sanm:NOTICE: sanm_procRemoteModifyRsp:
> > Error: Mirror[g2r5-v3v1-dmip] not found.
> >
> > Oct  4 15:37:24 g2r5 : 0:0:sanm:NOTICE: sanm_procRemoteModifyRsp:
> > remote modify failed, rc[-6654]
> >
> > Oct  4 15:37:24 g2r5 : 0:0:nfxsh:NOTICE: cmd[3]: mirror schedule
> > g2r5-v3v1-dmip -h 0 : status[11]
> >
> > g2r5 G2R5-VS3>
> >
> > g2r5 G2R5-VS3> mirror show g2r5-v3v1-dmip
> >
> > 
> >
> > Mirror Name             : g2r5-v3v1-dmip
> >
> > Admin State             : Enabled
> >
> > Operational State       : Idle
> >
> > Mirror Load             : MED
> >
> > 
> >
> > Source Vol Name         : g2r5-vs3-vol1
> >
> > Source Vol Id           : 0x24c00000080
> >
> > Mirror Vol Name         : g2r5-vs3-vol1-dmip@10.11.1.40
> >
> > Mirror Vol Id           : 0x4100000006b
> >
> > 
> >
> > Mirror Secondary Node Name : None
> >
> > 
> >
> > Mirror Schedule :
> >
> > -----------------
> >
> >    Minute       : *
> >
> >    Hour         : 0 2 4 6 8 10 12 14 16 18 20 22
> >
> >    Day Of Month : *
> >
> >    Month        : *
> >
> >    Day Of Week  : *
> >
> > 
> >
> > Counters:
> >
> > ---------
> >
> >    Mirror Start Count    : 0
> >
> >    Mirror Complete Count : 0
> >
> >    Mirror Abort Count    : 0
> >
> > 
> >
> > Last Mirror Session Information:
> >
> > --------------------------------
> >
> >    Mirror Started     :
> >
> >    Mirror Finished    :
> >
> >    Bytes Transferred  : 0 (MB)
> >
> >    Percent Transferred: 0%
> >
> >    Throughput         : 0 (bytes/s)
> >
> >    Snapshot Name      :
> > SANM_SS_g2r5-v3v1-dmip_0000024c00000080_0000109b
> >
> >    Session Status     :
> >
> > 
> >
> > g2r5 G2R5-VS3>
> >
> > 
> >
> > 
> >
> > -----Original Message-----
> > From: Paul Hammer
> > Sent: Thursday, October 04, 2007 3:13 PM
> > To: Brian Baker; Andy Sharp; Brian Stark
> > Cc: Raj Kumar; Sandrine Boulanger
> > Subject: RE: networking betwixt camphell and pleztown
> >
> > 
> >
> > Sandrine do you know?
> >
> > 
> >
> > -----Original Message-----
> >
> > From: Brian Baker
> >
> > Sent: Thursday, October 04, 2007 3:05 PM
> >
> > To: Andy Sharp; Brian Stark
> >
> > Cc: Raj Kumar; Paul Hammer; Sandrine Boulanger
> >
> > Subject: RE: networking betwixt camphell and pleztown
> >
> > 
> >
> > Raj,
> >
> > It appears that the dmip mirros are still going. Can you stop these
> > immediately? Let me know if you believe you have stopped your
> > mirror, if so I will find the culprit.
> >
> > 
> >
> > 
> >
> > -----Original Message-----
> >
> > From: Andy Sharp
> >
> > Sent: Thursday, October 04, 2007 3:02 PM
> >
> > To: Brian Stark
> >
> > Cc: Brian Baker
> >
> > Subject: Re: networking betwixt camphell and pleztown
> >
> > 
> >
> > I'm not sure where this got left off, but does this mean we know
> > what
> >
> > the traffic is?  Can we try pulling the plug on that IP address?
> >
> > 
> >
> > Thanks,
> >
> > 
> >
> > a
> >
> > 
> >
> > On Wed, 3 Oct 2007 18:04:22 -0700 "Brian Stark"
> >
> > <brian.stark@onstor.com> wrote:
> >
> > 
> >
> > > I'll go unplug the array that's spinning like crazy back there...
> >
> > > 
> >
> > >
> >
> > > > -----Original Message-----
> >
> > > > From: Andy Sharp
> >
> > > > Sent: Wednesday, October 03, 2007 5:48 PM
> >
> > > > To: Brian Baker
> >
> > > > Cc: Brian Stark
> >
> > > > Subject: Re: networking betwixt camphell and pleztown
> >
> > > >
> >
> > > > I'm all for blocking it and seeing {what happens,who
> >
> > > > squeals}.  Who's with me?
> >
> > > >
> >
> > > > I can't see the included link because I'm remote and I can't
> >
> > > > get an ssh proxy to work <hangs his head in shame>.
> >
> > > >
> >
> > > > a
> >
> > > >
> >
> > > >
> >
> > > > On Wed, 3 Oct 2007 17:18:33 -0700 "Brian Baker"
> >
> > > > <brian.baker@onstor.com> wrote:
> >
> > > >
> >
> > > > > Someone is hogging the connection up there. Take a look at
> > > > > MRTG.
> >
> > > > > http://mrtg/mrtg/hp5308A/10.0.0.19_8.html
> >
> > > > > This is the Pleasanton uplink.
> >
> > > > > We can ask who has been whoring the connection and ask them to
> >
> > > > > stop. Or I can look into it further and block their traffic.
> >
> > > > >
> >
> > > > > -----Original Message-----
> >
> > > > > From: Brian Baker
> >
> > > > > Sent: Wednesday, October 03, 2007 4:55 PM
> >
> > > > > To: Andy Sharp; Brian Stark
> >
> > > > > Subject: RE: networking betwixt camphell and pleztown
> >
> > > > >
> >
> > > > > Andy,
> >
> > > > > Thx for reporting this. Kevin was up there last week and he
> >
> > > > reported
> >
> > > > > the issue was resolved. I will take a look into this. But
> >
> > > > as you know
> >
> > > > > I'm very blah so I probably won't be able to look at it until
> >
> > > > > blah.
> >
> > > > >
> >
> > > > > -----Original Message-----
> >
> > > > > From: Andy Sharp
> >
> > > > > Sent: Wednesday, October 03, 2007 4:17 PM
> >
> > > > > To: Brian Stark
> >
> > > > > Cc: Brian Baker
> >
> > > > > Subject: networking betwixt camphell and pleztown
> >
> > > > >
> >
> > > > > I'm sure everyone knows by now, but I just thought I would put
> > > > > my
> >
> > > > > notice in just in case.
> >
> > > > >
> >
> > > > > Networking between Campbell and Pleasanton has slowed to a
> >
> > > > crawl, and
> >
> > > > > it's impacting my work, as well as others, I'm guessing,
> > > > > pretty
> >
> > > > > significantly.
> >
> > > > >
> >
> > > > > Transfering a kernel to 10.1.1.189 goes at about 100-160
> >
> > > > KiB/s, down
> >
> > > > > from 6-700 KiB yesterday.  It takes my cougar about 10
> >
> > > > minutes to boot
> >
> > > > > just to single user mode (NFS root is located in campbell),
> >
> > > > and that
> >
> > > > > means about 4-5 reboots an hour, when really I need about
> > > > > 12-15
> >
> > > > > reboots an hour.
> >
> > > > >
> >
> > > > > I know everyone has a lot to do and blah-blah, but I didn't
> > > > > want
> >
> > > > > to keep it a secret either.
> >
> > > > >
> >
> > > > > Cheers,
> >
> > > > >
> >
> > > > > a
> >
> > > >
> >
> 
> 
