Description: In Tuxstor, the user space daemons and TXRX/FP kernel code will be co-resident in a unified SMP target platform. Virtual server interfaces, IP addresses, and routes can be configured from user space. So VSD no longer needs to issue IPM message requests to the TXRX code (now in the kernel) and have it perform these basic operations. My plan is to implement them within the IPMD library since the API already exists. The code changes should have minimal impact on the existing API. By putting the code in this library, it can be shared by VSD, IPMD, and perhaps test utilities. IPMD Library APIs to be used: ----------------------------- ipmd_CreateVifCluster ipmd_DeleteVifCluster ipmd_PrefVifCluster ipmd_AddLinkVifCluster ipmd_RmLinkVifCluster ipmd_AddIpCluster ipmd_DelIpCluster ipmd_AddRtCluster ipmd_DelRtCluster ipmd_AddArpCluster ipmd_DelArpCluster ipmd_GetIfStatsCluster ipmd_GetIpStatsCluster ipmd_GetIcmpStatsCluster ipmd_GetUdpStatsCluster ipmd_GetTcpStatsCluster ipmd_ClearTxrxIntfStat ipmd_ClearTxrxICMPStat ipmd_ClearTxrxIpStat ipmd_ClearTxrxUdpStat ipmd_ClearTxrxTcpStat APIs to be removed: ----------------------------- ipmd_EnableIntfCluster ipmd_DisableIntfCluster ipmd_EnableIntfAutoCluster ipmd_DisableIntfAutoCluster ipmd_SetIntfMtuCluster ipmd_RestartCluster ipmd_ShutDownCluster ipmd_GetTcpConnCluster ipmd_GetUdpConnCluster (perhaps others that aren't in use) When a virtual server is configured to use an aggregated L-Port, a virtual network interface device shall be created in Linux with the ifenslave utility to attach the specified network interfaces to the bonding device. The bonding device will act like a normal Ethernet network device to the kernel, but will send out the packets via the slave devices using a simple round-robin scheduler. This supports simple load-balancing. Setup for the failover L-Port is to be determined. When a virtual server is configured to use vlan tagging, an virtual network interface device shall be created in Linux with the vconfig utility. The name of the network interface device is automatically created by the utility based on the base network device name (or bonding device) and vlan Id (e.g. eth0:10 where 10=vlanId). IP routing for each virtual server shall be administered by RPDB in Linux. A separate routing table will be setup for each virtual server using the ip command utility. The ip route cache shall be flushed to commit any changes once made. Rules shall be configured with respect to each virtual server context for processing network traffic. No changes are expected for dealing with network traffic via mgmt ports routed via the main routing table. Since the mgmt bus interface is going away, private virtual server IPs will no longer exist. Currently, they are used throughout the system by SSC (user) apps to convey a network association to a specific virtual server. The third and forth bytes of the IP mapped to the virtual server Id and interface index, respectively. Instead, apps will set/get virtual server Id (and/or interface index) context using setsockopt/getsockopt prior to sending or after receiving a packet. This functionality may be buried in a yet to be defined new api. IPMD: IPMD shall continue to be utilized for processing IPM message requests originating from remote daemon when in cluster mode. But rather than relaying message requests to the TXRX code, it will handle them itself and utilize the IPMD library API. VSD: The majority of changes are concentrated in IPM interface routines (residing in vs-ipm.c). The routines will be changed to reference IPMD library API instead of sending message requests to IPM in TXRX code. Virtual Server API / Data Structure: For the most part, changes to the existing virtual server api and data structure (vs_t) will be minimal if any. A few routines may be changed to pass virtual server Id instead of a vs_t address pointer to reduce reference count dependencies (as pointed out by Bill). Other changes may be related to where some tpl connection info will now be stored. Virtual Server Stats: The incoming/outgoing IP packet and data byte stats will be collected for each virtual server based on packet classification by realm. Each of the routes associated with a virtual server shall specify a realm for accounting purposes. These stats shall be accessible via the rtacct utility. ICMP, UDP, TCP, and other stats by protocol shall be collected by configuring a set of iptables rules for each virtual server interface. The protocol specific stats shall be accessible via the proc filesystem (/proc/net/ipt_account). The summation of these protocol specific stats may be used to derive IP stats rather than collection of IP stats by realm as previously described. These stats shall be retrieved upon request by ipmd using the existing msg api. Network Device Stats: IP stats for each physical network device (ethN) shall be collected by parsing ifconfig utility output or via network device ioctl. Stats for vlan enabled and bonded network interface devices may be included as well. These stats shall be retrieved upon request by ipmd using the existing msg api. --------------------------------------------------------------------------- Underlying Network Mgmt System Calls: (Example) IP=11.1.1.1 NETMSK=255.0.0.0 GW=11.0.0.1 NETIP=11.0.0.0 NETMB=8 BDCS=11.255.255.255 VSID=8 IDX=1 [VLAN=10] DEV=eth0,eth1,eth2,eth3 BDEV=(aggregation of devices eth0..eth3) EDEV=DEV [| BDEV] VDEV=EDEV.VLAN EVDEV=EDEV [| VDEV] Aggregated Port (Bonding) Setup/Shutdown Operation: [ifenslave BDEV DEV1 DEV2] [ifenslave -d BDEV DEV1 DEV2] Virtual Server Enable Operation: [vconfig add EDEV 10] ip addr add IP/NETMB broadcast BDCS dev EVDEV ip route del to NETIP/NETMB src IP dev EVDEV (remove auto generated route?) ip route add to NETIP/NETMB src IP dev EVDEV table VSID realm VSID ip rule add from NETIP/NETMB to IP iif EVDEV table VSID realms VSID/VSID iptables -A INPUT -m account -aname vs.VSID -aaddr IP iptables -A OUTPUT -m account -aname vs.VSID -aaddr IP ip route flush cache Virtual Server Disable Operation: iptables -D INPUT -m account -aname vs.VSID -aaddr IP iptables -D OUTPUT -m account -aname vs.VSID -aaddr IP ip rule del from NETIP/NETMB to IP iif EVDEV table VSID realms VSID/VSID ip route del to NETIP/NETMB src IP dev EVDEV table VSID realm VSID ip addr del IP/NETMB dev EVDEV [vconfig rem VDEV] ip route flush cache Interface Enable/Disable Operations: ip link set DEV up|down Route Setup/Shutdown Operation: ip route add|del default|IP/NETMB via GW table VSID realm VSID ip route flush cache