This site requires JavaScript to be enabled
Welcome|
Recent searches
IE BUMPER

Configure Network Teaming: LACP on Linux - Red Hat

Number of views : 398
Article Number : KB0018562
Published on : 2024-01-05
Last modified : 2024-01-05 18:08:18
Knowledge Base : IT Public Self Help

LACP Bonding

This page is to document instructions on how to configure LACP bonding for different OS variants.

We are using LACP bonding in CRB (UDCC) so that networking can do maintenance on either side of the switch infrastructure without impacting our services.

  • Bonding can have different modes. e.g. in Linux mode=1 means active/standby, but what we want in CRB is mode=4 which is an LACP bond.

NOTES for LINUX SYSTEMS:

  • The IPV6 kernel module must be loaded for bonding to work.  Since this is the default behavior for RHEL5 and RHEL6, typically no action is required. See IPv6 Notes below for details.
  • This procedure will not work unless the bonding driver is version 3.4.0-1 or later.  If the running kernel is older than 2.16.18-238 this might be the case. Check bonding driver version by:
strings /lib/modules/`uname -r`/kernel/drivers/net/bonding/bonding.ko | grep description=
  • Firewall rules may need to be updated. Changing references to eth* interfaces in /etc/sysconfig/iptables to bond* is typically sufficient.

Configurations

This process requires making changes to several network files located in /etc/sysconfig/network-scripts/. Additionally, the bonding module will need to be loaded by adding a file in /etc/modprobe.d or changing the file /etc/modprobe.conf. Changes will not take affect until reboot or network restart.

  • Make backup copies of the files you are going to change.
  • Edit the current ifcfg-ethN files to make them bond slaves.
  • Important note: The 'HWADDR=' setting needs to be present, especially when using add-on Ethernet cards. Bonds across the built-in and add-on NICs can fail if Linux sequences the interfaces differently than you expect.  Putting the hardware address in the ifcfg files forces a known binding between NIC port and eth device name.  This makes it a lot easier to relay to UDC which ports you are talking about.

# vi /etc/sysconfig/network-scripts/ifcfg-eth0

ifcfg-eth0 

ifcfg-eth0 

DEVICE=eth0 
BOOTPROTO=none 
HWADDR=00:21:AB:00:CD:12 
IPADDR=XXX.XXX.XXX.XXX 
NETMASK=255.255.XXX.XXX 
ONBOOT=yes 
USERCTL=no 
                                                    (original)

DEVICE=eth0 
BOOTPROTO=none 
HWADDR=00:21:AB:00:CD:12 
ONBOOT=yes 
USERCTL=no 
MASTER=bond0 
SLAVE=yes 
                                                         (new)

# vi /etc/sysconfig/network-scripts/ifcfg-eth1

ifcfg-eth1 

ifcfg-eth1 

DEVICE=eth1 
BOOTPROTO=none 
HWADDR=00:21:AB:00:CD:13 
IPADDR=XXX.XXX.XXX.XXX 
NETMASK=255.255.XXX.XXX 
ONBOOT=yes 
USERCTL=no 
                                                    (original)

DEVICE=eth1 
BOOTPROTO=none 
HWADDR=00:21:AB:00:CD:13 
ONBOOT=yes 
USERCTL=no 
MASTER=bond0 
SLAVE=yes 
                                                         (new)

  • Create a new file for the bonding.

# vi /etc/sysconfig/network-scripts/ifcfg-bond0

ifcfg-bond0

DEVICE=bond0 
BOOTPROTO=none 
IPADDR=XXX.XXX.XXX.XXX 
NETMASK=255.255.XXX.XXX 
ONBOOT=yes 
USERCTL=no 
BONDING_OPTS="miimon=100 mode=4 lacp_rate=0 xmit_hash_policy=layer3+4" 
                                                                                                                            (new)

  • Load the bonding module by either editing the /etc/modprobe.conf file or adding a new file called bonding.conf in /etc/modprobe.d/. The edit is the same for both.

# vi /etc/modprobe.conf  OR # vi /etc/modprobe.d/bonding
   … insert the following (Note, Bonding needs the IPv6 module loaded on RHEL 5.4 and later. Add the option to load, but disable it.) …

modprobe.conf 

bonding.conf 

alias bond0 bonding 
alias bond1 bonding (if needed) 
alias bond2 bonding (if needed) 
options ipv6 disable=1
alias eth0 forcedeth 
alias eth1 forcedeth 
... 
                                                                

alias bond0 bonding 
alias bond1 bonding (if needed) 
alias bond2 bonding (if needed)
options ipv6 disable=1                               

  • If the new bonding requires routing, a routing file named route-bondN will also need to be added in the /etc/sysconfig-network/ directory.

# vi /etc/sysconfig/network-scripts/route-bond0

route-bond0

GATEWAY0=XXX.XXX.XXX 
NETMASK0=255.255.XXX.XXX 
ADDRESS0=XXX.XXX.XXX.XXX 
GATEWAY1=XXX.XXX.XXX.XXX 
NETMASK1=255.255.XXX.XXX 
ADDRESS1=XXX.XXX.XXX.XXX 
                                                                 

  • Note - the name of the file designates the interface it is routing. The number inside the file for each route must start at 0 (GATEWAY0) and go up.
  • Old secondary routes were kept in the file /etc/sysconfig/static-routes, which can now be removed.

Testing

  • A simple service network restart (/sbin/service network restart) will typically put these changes into effect.
  • A system reboot is highly recommend to ensure the system is fully functional.
  • The tests below should be run before considering the system functional.
  • Recommended procedure is to boot into single-user mode and run tests from the console.

# /sbin/shutdown -r now

If connected to the console:

  1. Wait for the "Press any key to continue" messages.
  2. Press SPACE until the boot menu comes up.
  3. Boot current kernel in single-user mode.
  4. Wait until the shell prompt comes up.
  5. # ifup bond0
  6. # ifconfig -a (see below)
  7. # cat /proc/net/bonding/bond0 (see below)
  8. # tcpdump -i bond0 (verifies the bond is seeing traffic)
  9. Repeat for all other bonds
  10. Reboot to multi-user.

Sample output from /sbin/ifconfig -a

bond0     Link encap:Ethernet  HWaddr 00:21:AB:00:CD:12
          inet addr:XXX.XXX.XXX.XXX  Bcast:XXX.XXX.XXX.XXX  Mask:255.255.XXX.XXX
          inet6 addr: fe80::221:28ff:fe00:5d72/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:252266 errors:0 dropped:0 overruns:0 frame:0
          TX packets:252695 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:100303436 (95.6 MiB)  TX bytes:39060292 (37.2 MiB)
 
eth0      Link encap:Ethernet  HWaddr 00:21:AB:00:CD:12
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:119539 errors:0 dropped:0 overruns:0 frame:0
          TX packets:132236 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:47440581 (45.2 MiB)  TX bytes:21755091 (20.7 MiB)
          Interrupt:106
 
eth1      Link encap:Ethernet  HWaddr 00:21:AB:00:CD:13
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:132727 errors:0 dropped:0 overruns:0 frame:0
          TX packets:120459 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:52862855 (50.4 MiB)  TX bytes:17305201 (16.5 MiB)
          Interrupt:114 Base address:0x2000
 
Note: On RHEL5 the HWaddr of the slaves of a bond will all be that of the first slave.  The only way to see the real MAC address of bond slaves is via /proc/net/bonding/bond*.

Sample output from /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.6.0 (September 262009)
 
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
 
802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 17
        Partner Key: 33479
        Partner Mac Address: 00:23:04:ee:be:07
 
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:21:AB:00:CD:12
Aggregator ID: 1
Slave queue ID: 0
 
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:21:AB:00:CD:13
Aggregator ID: 1
Slave queue ID: 0
 
Notes: The HW addr here is the true 'factory' MAC address.

Some final notes:

  • Be sure to turn of system monitoring as loss of network connectivity will trigger alarms.
  • Make backup copies of any file you are going to change.
  • Be sure to check the hardware MAC addresses.
  • On certain machines (SunFire x4100s), Redhat Linux will swap interfaces compared to the order of network connections on the hardware proper:
    • Interface 1 is swapped with interface 3
    • Interface 2 is swapped with interface 4
  • By putting the hardware addresses in the ifcfg-eth config files in the proper order, it forces a bind between logical and physical devices and makes it a lot easier to relay to UDC which ports you are talking about.

IPv6 Notes

Something we normally do on managed servers is completely disable IPV6 for various reasons: not currently used so no need to waste those resources, security, DNS lookups, etc, etc. However, after several tries, we noticed that bonding would not work. After doing some research, found out that the bonding drivers have a dependency in ipv6 in RHEL 5.4 and later that it requires ipv6 kernel modules. There was a bug filed with Red hat and you can read up on it here:  https://bugzilla.redhat.com/show_bug.cgi?id=531873, however Red hat considers it a normal requirement now.

While the module is required to be loaded, it does not need to be enabled. To meet the bonding requirements and still disable the IPv6 protocol, add the following line to the modprobe.conf file. (note this change will require a reboot to take effect.)

Edit /etc/modprobe.conf add "options ipv6 disable=1" and reboot.

 

 

 

Permalink: /utss/KAhome.do?number=KB0012569

Thank You! Your feedback has been submitted.

Feedback