You are currently on IBM Systems Media’s archival website. Click here to view our new website.

Linux on Power > Tips and Techniques > Systems Management

VIOS 101: Network

In Part 1, I wrote an introduction to virtualization and then discussed CPU virtualization with PowerVM. In Part 2, we addressed memory technologies. In this article, we’ll be looking at network options and then in Part 4, we’ll review I/O.

Network Options

As with nonvirtualized LPARs, dedicated network adapters are always an option, even for LPARs using micropartitions (shared processor pool LPARs). The key point to remember is that LPM (live partition mobility) requires that all adapters be virtualized at the time of the move, so no dedicated adapters can be in use at that time. With respect to virtualized network options, you need to understand some specific terminologies, specifically Link Aggregation (standard and 8023AD/LACP), Virtual Ethernet and Shared Ethernet Adapters.

Link Aggregation

EtherChannel and IEEE 802.3ad Link Aggregation are network port aggregation technologies that allow several Ethernet adapters to be aggregated together to form a single pseudo Ethernet device.

For example, the physical adapters ent0 and ent1 can be aggregated to a pseudo adapter ent3; interface en3 would then be configured with an IP address. The system treats the aggregated adapters as one adapter. All adapters in the EtherChannel or Link Aggregation are given the same hardware (MAC) address, so they are treated by remote systems as if they were one adapter.

The main benefit of EtherChannel and IEEE 802.3ad Link Aggregation is that they have the network bandwidth of all of their adapters in a single network presence and, if an adapter fails, the packets are automatically sent on the next available adapter without disruption to existing user connections. The adapter is automatically returned to service on the EtherChannel or Link Aggregation when it recovers. Thus, link aggregation provides both additional bandwidth and redundancy.

For redundancy, I typically have an aggregate on VIO1 that is connected to one network switch and an aggregate on VIO2 that is connected to a separate network switch. That way, there is redundancy within the VIO and across switches. All the physical ports in the aggregation group must reside on the same switch except in the case of a switch stack, where they can reside on different switches on the stack. Thus, having each VIO on a different switch helps provide redundancy. There are now network switches that can handle aggregates across switches but, for our purposes, we will assume those are not being used. Lastly, the network switch will need to be set to either standard or 802.3ad (LACP) aggregation. I typically use LACP but this should be discussed with the network team to see what they prefer/support.

NIB – Network Interface Backup

NIB is a type of EtherChannel that is used for high-availability only. NIB allows an aggregated adapter to have a backup. If all adapters that compose the aggregation fail, then communication is switched to the backup adapter until any adapter in the main channel recovers

In the NIB mode of operation, you have an adapter in the main channel and a backup adapter. While NIB by itself doesn’t provide better bandwidth than the physical adapter, it can be used to work around switch failures. Usually port aggregation requires all adapters to be connected to the same switch, which makes the switch the single point of failure. By using NIB, and by connecting the primary and backup adapters to different switches, communication won’t be lost by the failure of a single switch.

To help detect loss of network reachability (in addition to detecting failures in the adapter and its connection to the switch), NIB allows specifying an address to be pinged. If the given address cannot be reached after a given number of attempts (both specified when NIB is defined), then the current active adapter is considered down, resulting in the backup adapter taking over communication.

Virtual Ethernet

Virtual Ethernet has been around since AIX 5.3 and requires the use of an HMC or IVM (Integrated Virtualization Manager). IVM only supports the use of a single VIO server, which reduces redundancy. Virtual Ethernet allows LPARs to communicate with each other without having to assign physical hardware to the LPARs. The LPARs communicate via the Hypervisor over Virtual Ethernet channels. It also allows for VLANs and other security mechanisms.

Virtual Ethernet adapters are connected to an IEEE 802.1q (VLAN)-style Virtual Ethernet switch, which allows LPARs to share a common logical network. The system transmits packets by copying the packet directly from the memory of the sender LPAR to the receive buffers of the receiver LPAR without any intermediate buffering of the packet.

Each Virtual Ethernet adapter can be used to access up to 20 networks—the port VLAN ID and up to 19 additional VLAN IDs. The HMC generates a locally administered Ethernet MAC address for the virtual Ethernet adapters so that these addresses don’t conflict with physical Ethernet adapter MAC addresses.

Virtual Ethernets are a great way to set up a private, secure, fast network between LPARs on the same server. Virtual Ethernet requires some CPU and memory to transfer network traffic. Performance and overhead (CPU and memory) are affected by entitlement as well as by the MTU size. If the data is never going external to the server then it’s worth looking at either 9000 or 65394 as potential MTU sizes for the Virtual Ethernet as this can significantly reduce CPU overhead while increasing network bandwidth across the Virtual Ethernet. Details on this are provided in Alexander Paul’s presentation at

Jaqui Lynch is an independent consultant, focusing on enterprise architecture, performance and delivery on Power Systems with AIX and Linux.



2019 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.


Planned Active Use of Elastic CoD Into Power Systems Solutions


Introducing IBM FlashSystem 840


HMC Architecture Options

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store