Configure Network Teaming: LACP on Linux - Red Hat
LACP Bonding
This page is to document instructions on how to configure LACP bonding for different OS variants.
We are using LACP bonding in CRB (UDCC) so that networking can do maintenance on either side of the switch infrastructure without impacting our services.
- Bonding can have different modes. e.g. in Linux mode=1 means active/standby, but what we want in CRB is mode=4 which is an LACP bond.
- N.B. the values in the BONDING_OPTS variable are per Charles Spurgeon's recommendations in BCP - Server Network Interface Configuration for Link Aggregation (LACP) in UDC-C
NOTES for LINUX SYSTEMS:
- The IPV6 kernel module must be loaded for bonding to work. Since this is the default behavior for RHEL5 and RHEL6, typically no action is required. See IPv6 Notes below for details.
- This procedure will not work unless the bonding driver is version 3.4.0-1 or later. If the running kernel is older than 2.16.18-238 this might be the case. Check bonding driver version by:
strings /lib/modules/`uname -r`/kernel/drivers/net/bonding/bonding.ko | grep description= |
- Firewall rules may need to be updated. Changing references to eth* interfaces in /etc/sysconfig/iptables to bond* is typically sufficient.
Configurations
This process requires making changes to several network files located in /etc/sysconfig/network-scripts/. Additionally, the bonding module will need to be loaded by adding a file in /etc/modprobe.d or changing the file /etc/modprobe.conf. Changes will not take affect until reboot or network restart.
- Make backup copies of the files you are going to change.
- Edit the current ifcfg-ethN files to make them bond slaves.
- Important note: The 'HWADDR=' setting needs to be present, especially when using add-on Ethernet cards. Bonds across the built-in and add-on NICs can fail if Linux sequences the interfaces differently than you expect. Putting the hardware address in the ifcfg files forces a known binding between NIC port and eth device name. This makes it a lot easier to relay to UDC which ports you are talking about.
# vi /etc/sysconfig/network-scripts/ifcfg-eth0
ifcfg-eth0 |
ifcfg-eth0 |
---|---|
DEVICE=eth0 |
DEVICE=eth0 |
# vi /etc/sysconfig/network-scripts/ifcfg-eth1
ifcfg-eth1 |
ifcfg-eth1 |
---|---|
DEVICE=eth1 |
DEVICE=eth1 |
- Create a new file for the bonding.
# vi /etc/sysconfig/network-scripts/ifcfg-bond0
ifcfg-bond0 |
---|
DEVICE=bond0 |
- Load the bonding module by either editing the /etc/modprobe.conf file or adding a new file called bonding.conf in /etc/modprobe.d/. The edit is the same for both.
# vi /etc/modprobe.conf OR # vi /etc/modprobe.d/bonding
… insert the following (Note, Bonding needs the IPv6 module loaded on RHEL 5.4 and later. Add the option to load, but disable it.) …
modprobe.conf |
bonding.conf |
---|---|
alias bond0 bonding |
alias bond0 bonding |
- If the new bonding requires routing, a routing file named route-bondN will also need to be added in the /etc/sysconfig-network/ directory.
# vi /etc/sysconfig/network-scripts/route-bond0
route-bond0 |
---|
GATEWAY0=XXX.XXX.XXX |
- Note - the name of the file designates the interface it is routing. The number inside the file for each route must start at 0 (GATEWAY0) and go up.
- Old secondary routes were kept in the file /etc/sysconfig/static-routes, which can now be removed.
Testing
- A simple service network restart (/sbin/service network restart) will typically put these changes into effect.
- A system reboot is highly recommend to ensure the system is fully functional.
- The tests below should be run before considering the system functional.
- Recommended procedure is to boot into single-user mode and run tests from the console.
# /sbin/shutdown -r now
If connected to the console:
- Wait for the "Press any key to continue" messages.
- Press SPACE until the boot menu comes up.
- Boot current kernel in single-user mode.
- Wait until the shell prompt comes up.
- # ifup bond0
- # ifconfig -a (see below)
- # cat /proc/net/bonding/bond0 (see below)
- # tcpdump -i bond0 (verifies the bond is seeing traffic)
- Repeat for all other bonds
- Reboot to multi-user.
Sample output from /sbin/ifconfig -a
bond0 Link encap:Ethernet HWaddr 00 : 21 :AB: 00 :CD: 12 inet addr:XXX.XXX.XXX.XXX Bcast:XXX.XXX.XXX.XXX Mask: 255.255 .XXX.XXX inet6 addr: fe80:: 221 :28ff:fe00:5d72/ 64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU: 1500 Metric: 1 RX packets: 252266 errors: 0 dropped: 0 overruns: 0 frame: 0 TX packets: 252695 errors: 0 dropped: 0 overruns: 0 carrier: 0 collisions: 0 txqueuelen: 0 RX bytes: 100303436 ( 95.6 MiB) TX bytes: 39060292 ( 37.2 MiB) eth0 Link encap:Ethernet HWaddr 00 : 21 :AB: 00 :CD: 12 UP BROADCAST RUNNING SLAVE MULTICAST MTU: 1500 Metric: 1 RX packets: 119539 errors: 0 dropped: 0 overruns: 0 frame: 0 TX packets: 132236 errors: 0 dropped: 0 overruns: 0 carrier: 0 collisions: 0 txqueuelen: 1000 RX bytes: 47440581 ( 45.2 MiB) TX bytes: 21755091 ( 20.7 MiB) Interrupt: 106 eth1 Link encap:Ethernet HWaddr 00 : 21 :AB: 00 :CD: 13 UP BROADCAST RUNNING SLAVE MULTICAST MTU: 1500 Metric: 1 RX packets: 132727 errors: 0 dropped: 0 overruns: 0 frame: 0 TX packets: 120459 errors: 0 dropped: 0 overruns: 0 carrier: 0 collisions: 0 txqueuelen: 1000 RX bytes: 52862855 ( 50.4 MiB) TX bytes: 17305201 ( 16.5 MiB) Interrupt: 114 Base address: 0x2000 Note: On RHEL5 the HWaddr of the slaves of a bond will all be that of the first slave. The only way to see the real MAC address of bond slaves is via /proc/net/bonding/bond*. |
Sample output from /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3. 6.0 (September 26 , 2009 ) Bonding Mode: IEEE 802 .3ad Dynamic link aggregation Transmit Hash Policy: layer3+ 4 ( 1 ) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802 .3ad info LACP rate: slow Aggregator selection policy (ad_select): stable Active Aggregator Info: Aggregator ID: 1 Number of ports: 2 Actor Key: 17 Partner Key: 33479 Partner Mac Address: 00 : 23 : 04 :ee:be: 07 Slave Interface: eth0 MII Status: up Link Failure Count: 0 Permanent HW addr: 00 : 21 :AB: 00 :CD: 12 Aggregator ID: 1 Slave queue ID: 0 Slave Interface: eth1 MII Status: up Link Failure Count: 0 Permanent HW addr: 00 : 21 :AB: 00 :CD: 13 Aggregator ID: 1 Slave queue ID: 0 Notes: The HW addr here is the true 'factory' MAC address. |
Some final notes:
- Be sure to turn of system monitoring as loss of network connectivity will trigger alarms.
- Make backup copies of any file you are going to change.
- Be sure to check the hardware MAC addresses.
- On certain machines (SunFire x4100s), Redhat Linux will swap interfaces compared to the order of network connections on the hardware proper:
- Interface 1 is swapped with interface 3
- Interface 2 is swapped with interface 4
- By putting the hardware addresses in the ifcfg-eth config files in the proper order, it forces a bind between logical and physical devices and makes it a lot easier to relay to UDC which ports you are talking about.
IPv6 Notes
Something we normally do on managed servers is completely disable IPV6 for various reasons: not currently used so no need to waste those resources, security, DNS lookups, etc, etc. However, after several tries, we noticed that bonding would not work. After doing some research, found out that the bonding drivers have a dependency in ipv6 in RHEL 5.4 and later that it requires ipv6 kernel modules. There was a bug filed with Red hat and you can read up on it here: https://bugzilla.redhat.com/show_bug.cgi?id=531873, however Red hat considers it a normal requirement now.
While the module is required to be loaded, it does not need to be enabled. To meet the bonding requirements and still disable the IPv6 protocol, add the following line to the modprobe.conf file. (note this change will require a reboot to take effect.)
Edit /etc/modprobe.conf add "options ipv6 disable=1" and reboot.
Permalink: /utss/KAhome.do?number=KB0012569