AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:
CFG:
PT:0
S:andy.sharp@lsi.com
RQ:
SSV:mhbs.lsil.com
NSV:
SSH:
R:<Rendell.Fong@lsi.com>
MAID:2
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/LSI/INBOX	0	1254766688.14287.180.camel@rendellf
X-Sylpheed-End-Special-Headers: 1
Date: Mon, 5 Oct 2009 11:55:57 -0700
From: Andrew Sharp <andy.sharp@lsi.com>
To: Rendell Fong <Rendell.Fong@lsi.com>
Subject: Re: vsd change description
Message-ID: <20091005115557.7b22a164@ripper.onstor.net>
In-Reply-To: <1254766688.14287.180.camel@rendellf>
References: <1254759281.14287.169.camel@rendellf>
	<20091005093247.7d7bade0@ripper.onstor.net>
	<1254766688.14287.180.camel@rendellf>
Organization: LSI
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

Some useless comments inline.  See what you can dig up regarding how
DNS will be handled, then give me a "final" version and I will throw it
on the wiki.  Or you can do it yourself, your choice.

On Mon, 5 Oct 2009 12:18:08 -0600 Rendell Fong <Rendell.Fong@lsi.com>
wrote:

> On Mon, 2009-10-05 at 10:32 -0600, Andrew Sharp wrote:
> > On Mon, 5 Oct 2009 10:14:41 -0600 Rendell Fong
> > <Rendell.Fong@lsi.com> wrote:
> > 
> > > This is a first cut at describing the tentative VSD changes.
> > > 
> > > 
> > > 
> > > Description:
> > > 
> > > In Tuxstor, the user space daemons and TXRX/FP kernel code will
> > > be co-resident in a unified SMP target platform.  Virtual server
> > > interfaces, IP addresses, and routes can be configured from user
> > > space. So rather than issuing request messages to IPM like in the
> > > existing implementation, VSD can perform this function itself to
> > > reduce complexity.  IPM will be eliminated but IPMD shall
> > > continue to be utilized for processing stats requests originating
> > > from local daemons (and remote ones when in cluster mode).
> > > 
> > > Virtual server interfaces shall be created by setting up virtual
> > > interfaces in Linux with the ifconfig utility.  The virtual
> > > interface
> > 
> > The ip utility, actually.
> 
> You won't be able to get stats per virtual server if you do it
> strictly with the ip utility.  Using virtual interfaces will allow IP
> packet/byte stats to be collected as expected.

Hm, why would the utility make a difference?  But I ask questions I
don't care about.  Whatever is the right choice.

> > > name shall be generated based on the base network device name,
> > > virtual server Id, and virtual server interface index (e.g.
> > > eth0:8-1 where eth0=NetDevice, 8=VsId, and 1=VsIfIndex).  By
> > > utilizing a virtual interface, stats for incoming/outgoing IP
> > > packet and data bytes will be collected on an individual virtual
> > > interface basis.  The stats for a virtual server will be derived
> > > from a summation of its virtual interface stats (or realm
> > > stats).  We still need to determine how to get ICMP, UDP, and TCP
> > > stats on a virtual server basis.
> > > 
> > > When a virtual server is configured to use vlan tagging, an
> > > additional virtual network interface device shall be created in
> > > Linux with the vconfig utility.  The name of the network
> > > interface device is automatically created by the utility based on
> > > the base network device name and vlan Id (e.g. eth0:8-1.10 where
> > > 10=vlanId). 
> > > 
> > > When a virtual server is configured to use an aggregated L-Port, a
> > > network interface device shall be created in Linux with the
> > > ifenslave utility to attach it to the bonding device.  The
> > > bonding device will
> > 
> > can't the ip utility do all this?  not that I care.
> 
> I don't know.  These are things to figure out once its implemented. 

Agreed.

> > 
> > > act like a normal Ethernet network device to the kernel, but will
> > > send out the packets via the slave devices using a simple
> > > round-robin scheduler. This supports simple load-balancing.
> > > Setup for the failover L-Port is to be determined.  
> > > 
> > > IP routing for each virtual server shall be administered by RPDB
> > > in Linux.  A separate routing table will be setup for each
> > > virtual server using the ip and iptables utilities.  Once the
> > > routes are added, the ip route cache shall be flushed to commit
> > > the changes.  The existing private virtual server IP address for
> > > each of its interfaces shall be
> > 
> > The wha?  Private virtual server IP address?  Wha be dis?  Why be
> > dis? Das history.
> > 
> 
> It might be easier to get things working this way initially.  Forcing
> use the the virtual server external IP addresses directly may make it
> more difficult to correlate to the virtual server since the IP
> addresses may not be unique except per network interface.  As a
> result, extra parameter(s) may need to be passed around.

Nah.  We're gonna give the private IP think das boot.

> > > supported using the iptable nat function to translate this IP
> > > address to/from its external IP address for outgoing/incoming
> > > packets.
> > > 
> > > 
> > > Issues:
> > > 
> > > 1. Need TCP/UDP/ICMP stats per vsvr.
> > 
> > low priority details ~:^)
> > 
> > > 2. Virtual network device names will used.  Existing code that
> > > references network interface info will probably be affected.
> > > 3. Is the length of the network device name a problem?
> > 
> > There's got to be a better naming scheme that isn't so...cyborgish.
> > 
> 
> I'll leave it to you since it sounds like you have a better idea.

No, I'm leaving it to you to wrack your brain to improve upon your
original idea ~:^)

> > > 4. Certain stat counters shall probably change based on whatever
> > > exists.
> > 
> > But of course.
> > 
> > 5.  Library code that does the 192.167.major.bites thing will have
> > to be converted to the new world order.  Same goes for any daemon
> > code that messes under the covers directly with those addresses.
> > Minor, but should be mentioned.
> > 
> > 
> > > Example:
> > > 
> > > IP=11.1.1.1  NETMSK=255.0.0.0  GW=11.0.0.1
> > > NETIP=11.0.0.0  NETMB=8
> > > VSID=8  IDX=1  [VLAN=10]
> > > DEV=eth0
> > > EDEV=DEV:VSID-IDX
> > > VDEV=EDEV.VLAN
> > > EVDEV=EDEV [| VDEV]
> > > 
> > > Virtual Server Enable Operation:
> > > 
> > > ifconfig EDEV IP netmask NETMSK up
> > > [vconfig add EDEV 10]
> > > ip route add to NETIP/NETMB src IP dev EVDEV table VSID realm VSID
> > > [ip route add default via GW dev EVDEV table VSID realm VSID]
> > > ip rule add from NETIP/NETMB to IP iif EVDEV table VSID realms
> > > VSID/VSID ip addr add 192.167.VSID.IDX/32 scope host dev EVDEV
> > > ip rule add from 192.167.VSDID.IDX/32 table VSID realms VSID/VSID
> > > iptables -t nat -A POSTROUTING -s 192.167.VSID.IDX/32 -o EVDEV -j
> > > SNAT --to-source IP
> > > ip route flush cache
> > > 
> > > 
> > > Virtual Server Disable Operation:
> > > 
> > > iptables -t nat -D POSTROUTING -s 192.167.VSID.IDX/32 -o EVDEV -j
> > > SNAT --to-source IP
> > 
> > nack.  192.167.bite.me is history.
> > 
> > > ip rule delete from 192.167.VSID.IDX/32 table VSID realms
> > > VSID/VSID ip addr delete 192.167.VSID.IDX/32 scope host dev EVDEV
> > > ip rule delete from NETIP/NETMB to IP iif EVDEV table VSID realms
> > > VSID/VSID
> > > [ip route delete default via NETIP dev EVDEV table VSID realm
> > > VSID] ip route delete to NETIP/NETMB src IP dev EVDEV table VSID
> > > realm VSID [vconfig rem VDEV]
> > > ifconfig EDEV down
> > > ip route flush cache
> > > 
> > > 
> 
