Description: ------------ For Tuxstor, configuration of the virtual server interfaces, IP addresses, and routes can be performed from user space. This facet of the virtual server code changes will vary depending on whether the non-merged or merged approach is chosen for running User Apps and TXRX/FP code. Non-Merged Approach: -------------------- In the non-merged approach, the user space apps (SSC daemons) and TXRX/FP kernel code are assumed to be running on separate cores (the existing Cougar hardware). Changes to VSD should be minimal since it must issue basically the same request messages to IPM in the TXRX code to perform the configuration operations. The TXRX IPM code needs to be created as a user space app that can receive the request messages and execute the network mgmt system calls from user space. IPMD (the SSC daemon) shall continue to service stat requests from remote nodes in the cluster. The actual stat counters for IP, ICMP, UDP, TCP will need to be updated based on what Linux has defined regardless of the approach taken. Merged Approach: ---------------- In the merged approach, the user space apps (daemons) and TXRX/FP kernel code will be co-resident in a unified target platform. VSD no longer needs to issue IPM message requests to the TXRX code (now in the kernel) and have it perform the configuration operations. It can perform the operations itself by executing system calls. The code may be implemented within the IPMD library since the API already exists. By putting the code in this library, it can be shared by VSD, IPMD, and perhaps test utilities. IPMD Library APIs to be used: ----------------------------- ipmd_CreateVifCluster ipmd_DeleteVifCluster ipmd_PrefVifCluster ipmd_AddLinkVifCluster ipmd_RmLinkVifCluster ipmd_SetIntfMtuCluster ipmd_AddIpCluster ipmd_DelIpCluster ipmd_AddRtCluster ipmd_DelRtCluster ipmd_GetIfStatsCluster ipmd_GetIpStatsCluster ipmd_GetIcmpStatsCluster ipmd_GetUdpStatsCluster ipmd_GetTcpStatsCluster ipmd_ClearTxrxIntfStat ipmd_ClearTxrxICMPStat ipmd_ClearTxrxIpStat ipmd_ClearTxrxUdpStat ipmd_ClearTxrxTcpStat APIs to be removed: ----------------------------- ipmd_EnableIntfCluster ipmd_DisableIntfCluster ipmd_EnableIntfAutoCluster ipmd_DisableIntfAutoCluster ipmd_RestartCluster ipmd_ShutDownCluster ipmd_GetTcpConnCluster ipmd_GetUdpConnCluster ipmd_AddArpCluster ipmd_DelArpCluster (perhaps others that aren't in use) When a virtual server is configured to use an aggregated L-Port, a virtual network interface device shall be created in Linux with the ifenslave utility to attach the specified network interfaces to the bonding device. The bonding device will act like a normal Ethernet network device to the kernel, but will send out the packets via the slave devices using a simple round-robin scheduler. This supports simple load-balancing. Setup for the failover L-Port is to be determined. When a virtual server is configured to use vlan tagging, an virtual network interface device shall be created in Linux with the vconfig utility. The name of the network interface device is automatically created by the utility based on the base network device name (or bonding device) and vlan Id (e.g. eth0:10 where 10=vlanId). IP routing for each virtual server shall be administered by RPDB in Linux. A separate routing table will be setup for each virtual server using the ip command utility. The ip route cache shall be flushed to commit any changes once made. Rules shall be configured with respect to each virtual server context for processing network traffic. No changes are expected for dealing with network traffic via mgmt ports routed via the main routing table. Since the mgmt bus interface is going away, private virtual server IPs will no longer exist. Currently, they are used throughout the system by SSC (user) apps to convey a network association to a specific virtual server. The third and forth bytes of the IP mapped to the virtual server Id and interface index, respectively. Instead, apps will set/get virtual server Id (and/or interface index) context using setsockopt/getsockopt prior to sending or after receiving a packet. This functionality may be buried in a yet to be defined new api. IPMD: IPMD shall continue to be utilized for processing IPM message requests originating from remote daemon when in cluster mode. But rather than relaying message requests to the TXRX code, it will handle them itself and utilize the IPMD library API. VSD: The majority of changes are concentrated in IPM interface routines (residing in vs-ipm.c). The routines will be changed to reference IPMD library API instead of sending message requests to IPM in TXRX code. Virtual Server API / Data Structure: For the most part, changes to the existing virtual server api and data structure (vs_t) will be minimal if any. A few routines may be changed to pass virtual server Id instead of a vs_t address pointer to reduce reference count dependencies (as pointed out by Bill). Other changes may be related to where some tpl connection info will now be stored. Virtual Server Stats: The incoming/outgoing IP packet and data byte stats will be collected for each virtual server based on packet classification by realm. Each of the routes associated with a virtual server shall specify a realm for accounting purposes. These stats shall be accessible via the rtacct utility. ICMP, UDP, TCP, and other stats by protocol shall be collected by configuring a set of iptables rules for each virtual server interface. The protocol specific stats shall be accessible via the proc filesystem (/proc/net/ipt_account). The summation of these protocol specific stats may be used to derive IP stats rather than collection of IP stats by realm as previously described. These stats shall be retrieved upon request by ipmd using the existing msg api. Network Device Stats: IP stats for each physical network device (ethN) shall be collected by parsing ifconfig utility output or via network device ioctl. Stats for vlan enabled and bonded network interface devices may be included as well. These stats shall be retrieved upon request by ipmd using the existing msg api. --------------------------------------------------------------------------- Underlying Network Mgmt System Calls: (Example) IP=11.1.1.1 NETMSK=255.0.0.0 GW=11.0.0.1 NETIP=11.0.0.0 NETMB=8 BDCS=11.255.255.255 VSID=8 IDX=1 [VLAN=10] DEV=eth0,eth1,eth2,eth3 BDEV=(aggregation of devices eth0..eth3) EDEV=DEV [| BDEV] VDEV=EDEV.VLAN EVDEV=EDEV [| VDEV] Aggregated Port (Bonding) Setup/Shutdown Operation: [ifenslave BDEV DEV1 DEV2] [ifenslave -d BDEV DEV1 DEV2] Virtual Server Enable Operation: [vconfig add EDEV 10] ip addr add IP/NETMB broadcast BDCS dev EVDEV ip route del to NETIP/NETMB src IP dev EVDEV (remove auto generated route?) ip route add to NETIP/NETMB src IP dev EVDEV table VSID realm VSID ip rule add from NETIP/NETMB to IP iif EVDEV table VSID realms VSID/VSID iptables -A INPUT -m account -aname vs.VSID -aaddr IP iptables -A OUTPUT -m account -aname vs.VSID -aaddr IP ip route flush cache Virtual Server Disable Operation: iptables -D INPUT -m account -aname vs.VSID -aaddr IP iptables -D OUTPUT -m account -aname vs.VSID -aaddr IP ip rule del from NETIP/NETMB to IP iif EVDEV table VSID realms VSID/VSID ip route del to NETIP/NETMB src IP dev EVDEV table VSID realm VSID ip addr del IP/NETMB dev EVDEV [vconfig rem VDEV] ip route flush cache Interface Enable/Disable Operations: ip link set DEV up|down Route Setup/Shutdown Operation: ip route add|del default|IP/NETMB via GW table VSID realm VSID ip route flush cache