AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:<20071004235329.61c7209e@ripper.onstor.net>
CFG:
PT:0
S:andy.sharp@onstor.com
RQ:
SSV:onstor-exch02.onstor.net
NSV:
SSH:
R:<brian.baker@onstor.com>,<paul.hammer@onstor.com>,<sandrine.boulanger@onstor.com>,<brian.stark@onstor.com>,<raj.kumar@onstor.com>,<john.rogers@onstor.com>,<joshua.goldenhar@onstor.com>
MAID:1
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/andys@onstor.net@onstor-exch02.onstor.net/INBOX	0	BB375AF679D4A34E9CA8DFA650E2B04E068BA6@onstor-exch02.onstor.net
X-Sylpheed-End-Special-Headers: 1
Date: Thu, 4 Oct 2007 23:59:01 -0700
From: Andrew Sharp <andy.sharp@onstor.com>
To: "Brian Baker" <brian.baker@onstor.com>
Cc: "Paul Hammer" <paul.hammer@onstor.com>, "Sandrine Boulanger"
 <sandrine.boulanger@onstor.com>, "Brian Stark" <brian.stark@onstor.com>,
 "Raj Kumar" <raj.kumar@onstor.com>, "John Rogers" <john.rogers@onstor.com>,
 "Joshua Goldenhar" <joshua.goldenhar@onstor.com>
Subject: Re: networking betwixt camphell and pleztown
Message-ID: <20071004235901.4e80dcb0@ripper.onstor.net>
In-Reply-To: <BB375AF679D4A34E9CA8DFA650E2B04E068BA6@onstor-exch02.onstor.net>
References: <BB375AF679D4A34E9CA8DFA650E2B04E05DD73DC@onstor-exch02.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E068BA5@onstor-exch02.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E05BFE50A@onstor-exch02.onstor.net>
	<BB375AF679D4A34E9CA8DFA650E2B04E068BA6@onstor-exch02.onstor.net>
Organization: Onstor
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

If we wanted to go the compression angle, we could simply set up an
ssh tunnel across the link, with compression, and we could probably do
it with a couple machines sitting derelict in the junk pile.  Total
cost: $0.01 (priceless) ~:^)

On Thu, 4 Oct 2007 23:44:08 -0700 "Brian Baker"
<brian.baker@onstor.com> wrote:

> It would be interesting to see how a WAN link compression device
> would handle the traffic. In the early days I used a Peribit
> appliance and it worked well with common traffic. NFS CIFS, FTP, SMTP
> even SQL. It did not speak off the wall protocols and I wonder if it
> could comprende DMIP enough to compress it. The 25k it would cost to
> get them could easily be used to replace ONStor's entire core switch
> infrastructure. Corporate never gets the toys :( Rockies! SWEEP! Brian
> 
> ________________________________
> 
> From: Paul Hammer
> Sent: Thu 10/4/2007 11:29 PM
> To: Brian Baker; Andy Sharp
> Cc: Sandrine Boulanger; Brian Stark; Raj Kumar; John Rogers; Joshua
> Goldenhar Subject: RE: networking betwixt camphell and pleztown
> 
> 
> Thanks Brian. The link saturatutaion was pure ignorance on our
> behalf, had no idea that DMIP would saturate the link to the point of
> hobbling productivity for the P'town team, this goes for both QA and
> Corporate, perhaps this is why Kevin never put the MD boxes in P'town
> 7 months ago as requested by Frank. Great point on the testing.
> Sandrine and John can you propose how we setup the DMIP feature for
> testing in Soak? I am okay with all of the suggestions that Brian
> listed. Thanks Brain, we did not know what impact this would have on
> the link for this test nor for MD. Now I understand a lot more about
> it. Thanks. John, Raj  and Sandrine lets work out what makes sense
> for SS and for MD with the link capacity in mind. Still curious
> though on the RiverBed angle. Thanks Brian. Go Rockies!! 2 Down and 1
> to Go! -Paul
> 
> ________________________________
> 
> From: Brian Baker
> Sent: Thu 10/4/2007 11:16 PM
> To: Paul Hammer; Andy Sharp
> Cc: Sandrine Boulanger; Brian Stark; Raj Kumar; John Rogers; Joshua
> Goldenhar Subject: RE: networking betwixt camphell and pleztown
> 
> 
> Paul,
> We already have layer 3/4 switches that support QoS . If DMIP
> generates this traffic then I will address it. I would use a method,
> per-port ingress-based enforcement bandwidth maximums. I'm assuming
> we won't have these spikes via DMIP for Mightydog as the baselines
> will be done here. No new network equipment needed no need to break
> out the checkbook. I think we need to revisit the testing that is
> done across the Pleasanton link. Why couldn't this testing be done
> with a DSL, cable line or t1 line that is hooked into the lab? This
> would give you the ability to test with different network equipment
> in front of your wan interfaces. John Rogers also mentioned that he
> was working on acquiring software to mimic wan speeds. I think this
> is the direction you need to take opposed to getting a additional
> line in Pleasanton. There shouldn't be a need for the geographic
> disparity, you just need the hops and latency to make it look like
> you aren't in the same room. DMIP will be considered general
> productivity traffic for this link and I will take it into account as
> I measure network traffic. Our problem in this instance was not
> understanding the impact of this test. I could have easily been
> someone downloading 1000's of torrents or 30GB of CIFS copies. If you
> give me some warning when utilizing corporate resources I can
> accommodate some traffic shaping. I only need some time to implement
> the solution and forewarning in the future. Brian 
> 
> ________________________________
> 
> From: Paul Hammer
> Sent: Thu 10/4/2007 7:06 PM
> To: Andy Sharp
> Cc: Brian Baker; Sandrine Boulanger; Brian Stark; Raj Kumar; John
> Rogers; Joshua Goldenhar Subject: RE: networking betwixt camphell and
> pleztown
> 
> 
> 
> Hey Andy,
> 
> Agree.
> 
> We need to plan accordingly for the MightyDog corporate DMIP
> session's that we plan to start soon.
> 
> Looks like we should have a rigid nightly sessions scheduled (so we
> have predicable perf hit). Else we need a bigger or different
> pipe/approach in place. A router may be the way to go, also
> interested in how a Riverbed deployment may be able to help us here.
> Added Josh for his perspective on this point.
> 
> Bottom line is we cannot negatively impact the Cougar schedule or our
> general productivity on this link with DMIP, so the solution has to
> take into account QoS as part of the solution.
> 
> Thanks,
> 
> -Paul
> 
> -----Original Message-----
> From: Andy Sharp
> Sent: Thursday, October 04, 2007 6:14 PM
> To: Paul Hammer
> Cc: Brian Baker; Sandrine Boulanger; Brian Stark; Raj Kumar
> Subject: Re: networking betwixt camphell and pleztown
> 
> The problem here, and the reason for this disconnect is that, IIUC, we
> are using a production asset for QA testing purposes.  I think this
> kind of testing is great -- but should we provision a separate link
> for it?  Or use a router that is capable of doing QoS itself to
> throttle back the DMIP traffic?
> 
> Having the line swamped between 8pm and 6am will still affect me
> greatly, sorry to say.
> 
> Cheers,
> 
> a
> 
> 
> On Thu, 4 Oct 2007 17:44:45 -0700 "Paul Hammer"
> <paul.hammer@onstor.com> wrote:
> 
> > Kind of funny, most folks want this thing to go way faster. I
> > understand why you would like slower.
> >
> > 
> >
> > 
> >
> > ________________________________
> >
> > From: Brian Baker
> > Sent: Thursday, October 04, 2007 3:59 PM
> > To: Sandrine Boulanger; Paul Hammer; Andy Sharp; Brian Stark
> > Cc: Raj Kumar
> > Subject: RE: networking betwixt camphell and pleztown
> >
> > 
> >
> > Sandrine,
> >
> > As you can see by the mrtg graph traffic has gone back down. As a
> > feature request to DMIP I would like to see bandwidth throttling on
> > the filer side.
> >
> > http://mrtg/mrtg/hp5308A/10.0.0.19_8.html
> >
> > 
> >
> > 
> >
> > From: Sandrine Boulanger
> > Sent: Thursday, October 04, 2007 3:50 PM
> > To: Paul Hammer; Brian Baker; Andy Sharp; Brian Stark
> > Cc: Raj Kumar
> > Subject: RE: networking betwixt camphell and pleztown
> >
> > 
> >
> > Is it better now?
> >
> > 
> >
> > I changed the schedules to run between 22 and 6am.
> >
> > I paused a mirror that was already at 66%, we can resume it tonight.
> >
> > I have an issue with one mirror for which I cannot modify the
> > schedule even if I do it from the GW that owns the vsvr. Mirror
> > show NAME finds the mirror but mirror schedule NAME does not...
> >
> > 
> >
> > Welcome to the ONStor NAS Gateway.
> >
> > 
> >
> > g2r5> vsvr show
> >
> > Virtual servers on nas gateway g2r5
> >
> > 
> >
> >  ID  State                             Name
> >
> > ====================================================
> >
> > 1    Enabled                           VS_MGMT_588
> >
> > 9    Enabled                           G2R5-VS1
> >
> > 10   Enabled                           G2R5-VS2
> >
> > 11   Enabled                           G2R5-VS3
> >
> > 12   Enabled                           G2R5-VS4
> >
> > 13   Enabled                           GNSSRVR
> >
> > 18   Enabled                           G2R5-VS5
> >
> > 19   Enabled                           G2R5-VS6
> >
> > g2r5> vsvr set G2R5-VS3
> >
> > g2r5 G2R5-VS3> mirror show
> >
> > 
> >
> >    Mirror Name    AdminState  Oper  State  Transferred      Source
> > Volume           Target Volume       VirtualServer
> >
> > ----------------  ----------  -----------  -----------
> > ----------------------  ----------------------  -------------
> >
> >   g2r5-v1v2-dmip     Enabled         Idle           0%
> > g2r5-vs1-vol2  g2r5-vs1-vol2@10.11.1.30       G2R5-VS1
> >
> >   g2r5-v2v1-dmip     Enabled         Idle           0%
> > g2r5-vs2-vol1  g2r5-vs2-vol1-dmip@10.11.1.30       G2R5-VS2
> >
> >   g2r5-v3v1-dmip     Enabled         Idle           0%
> > g2r5-vs3-vol1  g2r5-vs3-vol1-dmip@10.11.1.40       G2R5-VS3
> >
> >  g2r5vs5-vol1-m1     Enabled         Idle         100%
> > g2r5-vs5-vol1        g2r5-vs5-vol1-m1       G2R5-VS5
> >
> >  g2r5vs4-vol1-m1     Enabled         Idle           0%
> > g2r5-vs4-vol1         g2r5vs4-vol1-m1       G2R5-VS4
> >
> >  g2r5vs4-vol5-m1     Enabled         Idle         100%
> > g2r5-vs4-vol5         g2r5vs4-vol5-m1       G2R5-VS4
> >
> >   g4r8-v2v1-dmip     Enabled         Idle         100%
> > g4r8-vs2-vol1  g4r8-vs2-vol1-dmip@10.11.1.36       G4R8-VS2
> >
> >  g12r9-v1v1-dmip     Enabled       Paused          66%
> > g12r9-vs1-vol1  g12r9-vs1-vol1@10.11.1.36      G12R9-VS1
> >
> > g2r5 G2R5-VS3>
> >
> > g2r5 G2R5-VS3>
> >
> > g2r5 G2R5-VS3> mirror schedule g2r5-v3v1-dmip -h 0
> >
> >  /
> >
> > Error: Mirror[g2r5-v3v1-dmip] not found.
> >
> > % Command failure.
> >
> > g2r5 G2R5-VS3> elog show log 20
> >
> > Oct  4 15:35:16 g2r5 : 1:3:efs:INFO: 16516: FS: g2r5-vs5-vol1-m1
> > 0x24c000000dd - amDeltaInstallCommit - snp - snap resume: recal
> > refbyte snapcounts for 1
> >
> > Oct  4 15:35:16 g2r5 : 1:2:efs:INFO: 16517: FS: g2r5-vs5-vol1-m1
> > 0x24c000000dd - amDeltaInstallCommit - snp - snap resume: recal
> > refbyte snapcounts for 2
> >
> > Oct  4 15:35:16 g2r5 : 1:3:efs:INFO: 16518: FS: g2r5-vs5-vol1-m1
> > 0x24c000000dd - amDeltaInstallCommit - snp - snap resume: recal
> > refbyte snapcounts for 3
> >
> > Oct  4 15:35:19 g2r5 : 1:2:efs:INFO: 16519: FS: g2r5-vs5-vol1-m1
> > 0x24c000000dd - amDeltaInstallCommit - snp - snap revert mirror 3:
> > end
> >
> > Oct  4 15:35:19 g2r5 : 1:3:efs:INFO: 16520: FS: g2r5-vs5-vol1-m1
> > 0x24c000000dd - amDeltaInstallCommit - mirror - FS AM TARGET:
> > install commit reverted to snap 3 genNum 70 ( )
> >
> > Oct  4 15:35:19 g2r5 : 1:3:efs:INFO: 16521: FS: g2r5-vs5-vol1
> > 0x24c000000d9 - amDeltaComplete - mirror - FS AM: removing old
> > unused snapshot SANM_SS_g2r5vs5-vol1-m1_0000024c000000d9_0000019f 1
> > gen 70
> >
> > Oct  4 15:35:19 g2r5 : 1:2:efs:NOTICE: 16522: FS: g2r5-vs5-vol1
> > 0x24c000000d9 - amDeltaComplete - snp - snapnum 1 un pinned
> >
> > Oct  4 15:35:22 g2r5 : 1:3:efs:NOTICE: 16523: FS: g2r5-vs5-vol1
> > 0x24c000000d9 - amDeltaComplete - snp - snap remove complete for
> > SANM_SS_g2r5vs5-vol1-m1_0000024c000000d9_0000019f id 1
> >
> > Oct  4 15:35:22 g2r5 : 1:2:efs:INFO: 16524: FS: g2r5-vs5-vol1
> > 0x24c000000d9 - amDeltaComplete - mirror - FS AM: delta complete
> > session
> >
> > Oct  4 15:35:22 g2r5 : 0:0:sanm:NOTICE: Session terminated
> > successfully for mirror[g2r5vs5-vol1-m1]
> >
> > Oct  4 15:35:25 g2r5 : 0:0:sanm:NOTICE: SANM: Mirror Completed:
> > g2r5vs5-vol1-m1
> >
> > Oct  4 15:36:13 g2r5 : 0:0:sanm:NOTICE: sanm_procRemoteModifyRsp:
> > Error: Mirror[g2r5-v3v1-dmip] not found.
> >
> > Oct  4 15:36:13 g2r5 : 0:0:sanm:NOTICE: sanm_procRemoteModifyRsp:
> > remote modify failed, rc[-6654]
> >
> > Oct  4 15:36:46 g2r5 : 0:0:ssh:INFO: 'admin' logged in through
> > remote host(10.0.0.79)
> >
> > Oct  4 15:36:49 g2r5 : 0:0:nfxsh:NOTICE: cmd[0]: vsvr show :
> > status[0]
> >
> > Oct  4 15:36:54 g2r5 : 0:0:nfxsh:NOTICE: cmd[1]: vsvr set G2R5-VS3 :
> > status[0]
> >
> > Oct  4 15:36:59 g2r5 : 0:0:nfxsh:NOTICE: cmd[2]: mirror show  :
> > status[0]
> >
> > Oct  4 15:37:24 g2r5 : 0:0:sanm:NOTICE: sanm_procRemoteModifyRsp:
> > Error: Mirror[g2r5-v3v1-dmip] not found.
> >
> > Oct  4 15:37:24 g2r5 : 0:0:sanm:NOTICE: sanm_procRemoteModifyRsp:
> > remote modify failed, rc[-6654]
> >
> > Oct  4 15:37:24 g2r5 : 0:0:nfxsh:NOTICE: cmd[3]: mirror schedule
> > g2r5-v3v1-dmip -h 0 : status[11]
> >
> > g2r5 G2R5-VS3>
> >
> > g2r5 G2R5-VS3> mirror show g2r5-v3v1-dmip
> >
> > 
> >
> > Mirror Name             : g2r5-v3v1-dmip
> >
> > Admin State             : Enabled
> >
> > Operational State       : Idle
> >
> > Mirror Load             : MED
> >
> > 
> >
> > Source Vol Name         : g2r5-vs3-vol1
> >
> > Source Vol Id           : 0x24c00000080
> >
> > Mirror Vol Name         : g2r5-vs3-vol1-dmip@10.11.1.40
> >
> > Mirror Vol Id           : 0x4100000006b
> >
> > 
> >
> > Mirror Secondary Node Name : None
> >
> > 
> >
> > Mirror Schedule :
> >
> > -----------------
> >
> >    Minute       : *
> >
> >    Hour         : 0 2 4 6 8 10 12 14 16 18 20 22
> >
> >    Day Of Month : *
> >
> >    Month        : *
> >
> >    Day Of Week  : *
> >
> > 
> >
> > Counters:
> >
> > ---------
> >
> >    Mirror Start Count    : 0
> >
> >    Mirror Complete Count : 0
> >
> >    Mirror Abort Count    : 0
> >
> > 
> >
> > Last Mirror Session Information:
> >
> > --------------------------------
> >
> >    Mirror Started     :
> >
> >    Mirror Finished    :
> >
> >    Bytes Transferred  : 0 (MB)
> >
> >    Percent Transferred: 0%
> >
> >    Throughput         : 0 (bytes/s)
> >
> >    Snapshot Name      :
> > SANM_SS_g2r5-v3v1-dmip_0000024c00000080_0000109b
> >
> >    Session Status     :
> >
> > 
> >
> > g2r5 G2R5-VS3>
> >
> > 
> >
> > 
> >
> > -----Original Message-----
> > From: Paul Hammer
> > Sent: Thursday, October 04, 2007 3:13 PM
> > To: Brian Baker; Andy Sharp; Brian Stark
> > Cc: Raj Kumar; Sandrine Boulanger
> > Subject: RE: networking betwixt camphell and pleztown
> >
> > 
> >
> > Sandrine do you know?
> >
> > 
> >
> > -----Original Message-----
> >
> > From: Brian Baker
> >
> > Sent: Thursday, October 04, 2007 3:05 PM
> >
> > To: Andy Sharp; Brian Stark
> >
> > Cc: Raj Kumar; Paul Hammer; Sandrine Boulanger
> >
> > Subject: RE: networking betwixt camphell and pleztown
> >
> > 
> >
> > Raj,
> >
> > It appears that the dmip mirros are still going. Can you stop these
> > immediately? Let me know if you believe you have stopped your
> > mirror, if so I will find the culprit.
> >
> > 
> >
> > 
> >
> > -----Original Message-----
> >
> > From: Andy Sharp
> >
> > Sent: Thursday, October 04, 2007 3:02 PM
> >
> > To: Brian Stark
> >
> > Cc: Brian Baker
> >
> > Subject: Re: networking betwixt camphell and pleztown
> >
> > 
> >
> > I'm not sure where this got left off, but does this mean we know
> > what
> >
> > the traffic is?  Can we try pulling the plug on that IP address?
> >
> > 
> >
> > Thanks,
> >
> > 
> >
> > a
> >
> > 
> >
> > On Wed, 3 Oct 2007 18:04:22 -0700 "Brian Stark"
> >
> > <brian.stark@onstor.com> wrote:
> >
> > 
> >
> > > I'll go unplug the array that's spinning like crazy back there...
> >
> > > 
> >
> > >
> >
> > > > -----Original Message-----
> >
> > > > From: Andy Sharp
> >
> > > > Sent: Wednesday, October 03, 2007 5:48 PM
> >
> > > > To: Brian Baker
> >
> > > > Cc: Brian Stark
> >
> > > > Subject: Re: networking betwixt camphell and pleztown
> >
> > > >
> >
> > > > I'm all for blocking it and seeing {what happens,who
> >
> > > > squeals}.  Who's with me?
> >
> > > >
> >
> > > > I can't see the included link because I'm remote and I can't
> >
> > > > get an ssh proxy to work <hangs his head in shame>.
> >
> > > >
> >
> > > > a
> >
> > > >
> >
> > > >
> >
> > > > On Wed, 3 Oct 2007 17:18:33 -0700 "Brian Baker"
> >
> > > > <brian.baker@onstor.com> wrote:
> >
> > > >
> >
> > > > > Someone is hogging the connection up there. Take a look at
> > > > > MRTG.
> >
> > > > > http://mrtg/mrtg/hp5308A/10.0.0.19_8.html
> >
> > > > > This is the Pleasanton uplink.
> >
> > > > > We can ask who has been whoring the connection and ask them to
> >
> > > > > stop. Or I can look into it further and block their traffic.
> >
> > > > >
> >
> > > > > -----Original Message-----
> >
> > > > > From: Brian Baker
> >
> > > > > Sent: Wednesday, October 03, 2007 4:55 PM
> >
> > > > > To: Andy Sharp; Brian Stark
> >
> > > > > Subject: RE: networking betwixt camphell and pleztown
> >
> > > > >
> >
> > > > > Andy,
> >
> > > > > Thx for reporting this. Kevin was up there last week and he
> >
> > > > reported
> >
> > > > > the issue was resolved. I will take a look into this. But
> >
> > > > as you know
> >
> > > > > I'm very blah so I probably won't be able to look at it until
> >
> > > > > blah.
> >
> > > > >
> >
> > > > > -----Original Message-----
> >
> > > > > From: Andy Sharp
> >
> > > > > Sent: Wednesday, October 03, 2007 4:17 PM
> >
> > > > > To: Brian Stark
> >
> > > > > Cc: Brian Baker
> >
> > > > > Subject: networking betwixt camphell and pleztown
> >
> > > > >
> >
> > > > > I'm sure everyone knows by now, but I just thought I would put
> > > > > my
> >
> > > > > notice in just in case.
> >
> > > > >
> >
> > > > > Networking between Campbell and Pleasanton has slowed to a
> >
> > > > crawl, and
> >
> > > > > it's impacting my work, as well as others, I'm guessing,
> > > > > pretty
> >
> > > > > significantly.
> >
> > > > >
> >
> > > > > Transfering a kernel to 10.1.1.189 goes at about 100-160
> >
> > > > KiB/s, down
> >
> > > > > from 6-700 KiB yesterday.  It takes my cougar about 10
> >
> > > > minutes to boot
> >
> > > > > just to single user mode (NFS root is located in campbell),
> >
> > > > and that
> >
> > > > > means about 4-5 reboots an hour, when really I need about
> > > > > 12-15
> >
> > > > > reboots an hour.
> >
> > > > >
> >
> > > > > I know everyone has a lot to do and blah-blah, but I didn't
> > > > > want
> >
> > > > > to keep it a secret either.
> >
> > > > >
> >
> > > > > Cheers,
> >
> > > > >
> >
> > > > > a
> >
> > > >
> >
> 
> 
