AF:
NF:0
PS:10
SRH:1
SFN:
DSR:
MID:
CFG:
PT:0
S:andy.sharp@lsi.com
RQ:
SSV:mhbs.lsil.com
NSV:
SSH:
R:<Rendell.Fong@lsi.com>
MAID:2
X-Sylpheed-Privacy-System:
X-Sylpheed-Sign:0
SCF:#mh/Mailbox/sent
RMID:#imap/LSI/INBOX	0	1254852979.14287.230.camel@rendellf
X-Sylpheed-End-Special-Headers: 1
Date: Tue, 6 Oct 2009 13:08:43 -0700
From: Andrew Sharp <andy.sharp@lsi.com>
To: Rendell Fong <Rendell.Fong@lsi.com>
Subject: Re: vsd change description
Message-ID: <20091006130843.030931d6@ripper.onstor.net>
In-Reply-To: <1254852979.14287.230.camel@rendellf>
References: <1254759281.14287.169.camel@rendellf>
	<20091005093247.7d7bade0@ripper.onstor.net>
	<1254766688.14287.180.camel@rendellf>
	<20091005115557.7b22a164@ripper.onstor.net>
	<1254852979.14287.230.camel@rendellf>
Organization: LSI
X-Mailer: Sylpheed-Claws 2.6.0 (GTK+ 2.8.20; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

The answer is simple: it has to specify the interface.  's got to be a
way.

On Tue, 6 Oct 2009 12:16:18 -0600 Rendell Fong <Rendell.Fong@lsi.com>
wrote:

> I've got some questions:
> 
> If we don't use private IPs, then how can user apps determine the vsvr
> context when the vsvr IPs are not unique?  Socket info doesn't include
> any info regarding the interface from which incoming packets were
> received.
> 
> When user apps are sending packets relative to a vsvr context, how do
> they specify the routing table wrt the socket call to ensure that
> proper routing occurs?  Is there some socket option or bind option
> that must be set?
> 
> 
> On Mon, 2009-10-05 at 12:55 -0600, Andrew Sharp wrote:
> > Some useless comments inline.  See what you can dig up regarding how
> > DNS will be handled, then give me a "final" version and I will
> > throw it on the wiki.  Or you can do it yourself, your choice.
> > 
> > On Mon, 5 Oct 2009 12:18:08 -0600 Rendell Fong
> > <Rendell.Fong@lsi.com> wrote:
> > 
> > > On Mon, 2009-10-05 at 10:32 -0600, Andrew Sharp wrote:
> > > > On Mon, 5 Oct 2009 10:14:41 -0600 Rendell Fong
> > > > <Rendell.Fong@lsi.com> wrote:
> > > > 
> > > > > This is a first cut at describing the tentative VSD changes.
> > > > > 
> > > > > 
> > > > > 
> > > > > Description:
> > > > > 
> > > > > In Tuxstor, the user space daemons and TXRX/FP kernel code
> > > > > will be co-resident in a unified SMP target platform.
> > > > > Virtual server interfaces, IP addresses, and routes can be
> > > > > configured from user space. So rather than issuing request
> > > > > messages to IPM like in the existing implementation, VSD can
> > > > > perform this function itself to reduce complexity.  IPM will
> > > > > be eliminated but IPMD shall continue to be utilized for
> > > > > processing stats requests originating from local daemons (and
> > > > > remote ones when in cluster mode).
> > > > > 
> > > > > Virtual server interfaces shall be created by setting up
> > > > > virtual interfaces in Linux with the ifconfig utility.  The
> > > > > virtual interface
> > > > 
> > > > The ip utility, actually.
> > > 
> > > You won't be able to get stats per virtual server if you do it
> > > strictly with the ip utility.  Using virtual interfaces will
> > > allow IP packet/byte stats to be collected as expected.
> > 
> > Hm, why would the utility make a difference?  But I ask questions I
> > don't care about.  Whatever is the right choice.
> > 
> > > > > name shall be generated based on the base network device name,
> > > > > virtual server Id, and virtual server interface index (e.g.
> > > > > eth0:8-1 where eth0=NetDevice, 8=VsId, and 1=VsIfIndex).  By
> > > > > utilizing a virtual interface, stats for incoming/outgoing IP
> > > > > packet and data bytes will be collected on an individual
> > > > > virtual interface basis.  The stats for a virtual server will
> > > > > be derived from a summation of its virtual interface stats
> > > > > (or realm stats).  We still need to determine how to get
> > > > > ICMP, UDP, and TCP stats on a virtual server basis.
> > > > > 
> > > > > When a virtual server is configured to use vlan tagging, an
> > > > > additional virtual network interface device shall be created
> > > > > in Linux with the vconfig utility.  The name of the network
> > > > > interface device is automatically created by the utility
> > > > > based on the base network device name and vlan Id (e.g.
> > > > > eth0:8-1.10 where 10=vlanId). 
> > > > > 
> > > > > When a virtual server is configured to use an aggregated
> > > > > L-Port, a network interface device shall be created in Linux
> > > > > with the ifenslave utility to attach it to the bonding
> > > > > device.  The bonding device will
> > > > 
> > > > can't the ip utility do all this?  not that I care.
> > > 
> > > I don't know.  These are things to figure out once its
> > > implemented. 
> > 
> > Agreed.
> > 
> > > > 
> > > > > act like a normal Ethernet network device to the kernel, but
> > > > > will send out the packets via the slave devices using a simple
> > > > > round-robin scheduler. This supports simple load-balancing.
> > > > > Setup for the failover L-Port is to be determined.  
> > > > > 
> > > > > IP routing for each virtual server shall be administered by
> > > > > RPDB in Linux.  A separate routing table will be setup for
> > > > > each virtual server using the ip and iptables utilities.
> > > > > Once the routes are added, the ip route cache shall be
> > > > > flushed to commit the changes.  The existing private virtual
> > > > > server IP address for each of its interfaces shall be
> > > > 
> > > > The wha?  Private virtual server IP address?  Wha be dis?  Why
> > > > be dis? Das history.
> > > > 
> > > 
> > > It might be easier to get things working this way initially.
> > > Forcing use the the virtual server external IP addresses directly
> > > may make it more difficult to correlate to the virtual server
> > > since the IP addresses may not be unique except per network
> > > interface.  As a result, extra parameter(s) may need to be passed
> > > around.
> > 
> > Nah.  We're gonna give the private IP think das boot.
> > 
> > > > > supported using the iptable nat function to translate this IP
> > > > > address to/from its external IP address for outgoing/incoming
> > > > > packets.
> > > > > 
> > > > > 
> > > > > Issues:
> > > > > 
> > > > > 1. Need TCP/UDP/ICMP stats per vsvr.
> > > > 
> > > > low priority details ~:^)
> > > > 
> > > > > 2. Virtual network device names will used.  Existing code that
> > > > > references network interface info will probably be affected.
> > > > > 3. Is the length of the network device name a problem?
> > > > 
> > > > There's got to be a better naming scheme that isn't
> > > > so...cyborgish.
> > > > 
> > > 
> > > I'll leave it to you since it sounds like you have a better idea.
> > 
> > No, I'm leaving it to you to wrack your brain to improve upon your
> > original idea ~:^)
> > 
> > > > > 4. Certain stat counters shall probably change based on
> > > > > whatever exists.
> > > > 
> > > > But of course.
> > > > 
> > > > 5.  Library code that does the 192.167.major.bites thing will
> > > > have to be converted to the new world order.  Same goes for any
> > > > daemon code that messes under the covers directly with those
> > > > addresses. Minor, but should be mentioned.
> > > > 
> > > > 
> > > > > Example:
> > > > > 
> > > > > IP=11.1.1.1  NETMSK=255.0.0.0  GW=11.0.0.1
> > > > > NETIP=11.0.0.0  NETMB=8
> > > > > VSID=8  IDX=1  [VLAN=10]
> > > > > DEV=eth0
> > > > > EDEV=DEV:VSID-IDX
> > > > > VDEV=EDEV.VLAN
> > > > > EVDEV=EDEV [| VDEV]
> > > > > 
> > > > > Virtual Server Enable Operation:
> > > > > 
> > > > > ifconfig EDEV IP netmask NETMSK up
> > > > > [vconfig add EDEV 10]
> > > > > ip route add to NETIP/NETMB src IP dev EVDEV table VSID realm
> > > > > VSID [ip route add default via GW dev EVDEV table VSID realm
> > > > > VSID] ip rule add from NETIP/NETMB to IP iif EVDEV table VSID
> > > > > realms VSID/VSID ip addr add 192.167.VSID.IDX/32 scope host
> > > > > dev EVDEV ip rule add from 192.167.VSDID.IDX/32 table VSID
> > > > > realms VSID/VSID iptables -t nat -A POSTROUTING -s
> > > > > 192.167.VSID.IDX/32 -o EVDEV -j SNAT --to-source IP
> > > > > ip route flush cache
> > > > > 
> > > > > 
> > > > > Virtual Server Disable Operation:
> > > > > 
> > > > > iptables -t nat -D POSTROUTING -s 192.167.VSID.IDX/32 -o
> > > > > EVDEV -j SNAT --to-source IP
> > > > 
> > > > nack.  192.167.bite.me is history.
> > > > 
> > > > > ip rule delete from 192.167.VSID.IDX/32 table VSID realms
> > > > > VSID/VSID ip addr delete 192.167.VSID.IDX/32 scope host dev
> > > > > EVDEV ip rule delete from NETIP/NETMB to IP iif EVDEV table
> > > > > VSID realms VSID/VSID
> > > > > [ip route delete default via NETIP dev EVDEV table VSID realm
> > > > > VSID] ip route delete to NETIP/NETMB src IP dev EVDEV table
> > > > > VSID realm VSID [vconfig rem VDEV]
> > > > > ifconfig EDEV down
> > > > > ip route flush cache
> > > > > 
> > > > > 
> > > 
> 
