How to: Linux Ethernet Bonding

How to create:

[root@VMXDB01 u01]# cat /etc/modprobe.conf
alias eth0 bnx2
alias eth1 bnx2
alias eth2 bnx2
alias eth3 bnx2
alias eth4 bnx2
alias eth5 bnx2
alias eth6 bnx2
alias eth7 bnx2
alias scsi_hostadapter megaraid_sas
alias scsi_hostadapter1 ata_piix
alias scsi_hostadapter2 qla2xxx
alias bond0 bonding
alias bond1 bonding
options bonding miimon=100 mode=1
alias net-pf-10 off
alias ipv6 off
install usb-storage :

[root@VMXDB01 u01]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
USERCTL=no
ONBOOT=yes
NETMASK=255.255.255.224
BROADCAST=192.168.0.159
GATEWAY=192.168.0.129
IPADDR=192.168.0.144
BOOTPROTO=none
TYPE=Ethernet
IPV6INIT=no
PEERDNS=yes

[oracle@VMXDB01 ~]$ cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
TYPE=Ethernet
IPV6INIT=no
PEERDNS=yes

[oracle@VMXDB01 ~]$ cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
TYPE=Ethernet
IPV6INIT=no
PEERDNS=yes

How to monitor:

[root@VMXDB01 u01]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:10:18:4F:BA:E0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:2877200 errors:0 dropped:0 overruns:0 frame:0
          TX packets:560853 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4091658143 (3.8 GiB)  TX bytes:62462481 (59.5 MiB)
          Interrupt:169 Memory:ce000000-ce012100

[root@VMXDB01 u01]# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:10:18:4F:BA:E0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:1043 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:66822 (65.2 KiB)  TX bytes:0 (0.0 b)
          Interrupt:218 Memory:da000000-da012100

[root@VMXDB01 u01]# ifconfig bond0
bond0     Link encap:Ethernet  HWaddr 00:10:18:4F:BA:E0
          inet addr:192.168.0.144  Bcast:192.168.0.159  Mask:255.255.255.224
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:2878296 errors:0 dropped:0 overruns:0 frame:0
          TX packets:560895 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4091729879 (3.8 GiB)  TX bytes:62468081 (59.5 MiB)

[root@VMXDB01 u01]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:10:18:4f:ba:e0

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:10:18:4f:b9:94

Interesting informationfrom dmesg and /var/log/messages

eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem ce000000, IRQ 169, node addr 0010184fbae0
... ...
eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem da000000, IRQ 218, node addr 0010184fb994
... ...
bonding: MII link monitoring set to 100 ms
bonding: bond0: Adding slave eth0.
bnx2: eth0: using MSIX
bonding: bond0: enslaving eth0 as a backup interface with a down link.
bonding: bond0: Adding slave eth1.
bnx2: eth1: using MSIX
bonding: bond0: enslaving eth1 as a backup interface with a down link.
bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex
bonding: bond0: link status definitely up for interface eth0.
bonding: bond0: making interface eth0 the new active one.
bonding: bond0: first active interface up!
bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex
bonding: bond0: link status definitely up for interface eth1.

How many "mode" do we have:

/usr/src/kernels/2.6.18-128.2.1.el5-x86_64/include/linux/if_bonding.h

#define BOND_MODE_ROUNDROBIN    0
#define BOND_MODE_ACTIVEBACKUP  1
#define BOND_MODE_XOR           2
#define BOND_MODE_BROADCAST     3
#define BOND_MODE_8023AD        4
#define BOND_MODE_TLB           5
#define BOND_MODE_ALB           6 /* TLB + RLB (receive load balancing) */

Important: Switch configuration required

For this section, "switch" refers to whatever system the bonded devices are directly connected to (i.e., where the other end of the cable plugs into). This may be an actual dedicated switch device, or it may be another regular system (e.g., another computer running Linux),

The active-backup, balance-tlb and balance-alb modes do not require any specific configuration of the switch.

The 802.3ad mode requires that the switch have the appropriate ports configured as an 802.3ad aggregation. The precise method used to configure this varies from switch to switch, but, for example, a Cisco 3550 series switch requires that the appropriate ports first be grouped together in a single etherchannel instance, then that etherchannel is set to mode "lacp" to enable 802.3ad (instead of standard EtherChannel).

The balance-rr, balance-xor and broadcast modes generally require that the switch have the appropriate ports grouped together. The nomenclature for such a group differs between switches, it may be called an "etherchannel" (as in the Cisco example, above), a "trunk group" or some other similar variation. For these modes, each switch will also have its own configuration options for the switch's transmit policy to the bond. Typical choices include XOR of either the MAC or IP addresses. The transmit policy of the two peers does not need to match. For these three modes, the bonding mode really selects a transmit policy for an EtherChannel group; all three will interoperate with another EtherChannel group.


Reference:
Net: Bonding
Metalink Notes: 456055.1, 436555.1, 434375.1