Create OpenFlow network with multiple PCs/NetFPGAs

Setup OpenFlow network controlled by NOX0.6 with a set of PCs in the Lab in half day.

1. Description of an Example Setup

In this page, we'll create an OpenFlow network with three OpenFlow switches controlled by NOX 0.6 controller. Among three OpenFlow switches, two are PC based software OpenFlow switches and one is NetFPGA based hardware OpenFlow switch. If you don't have NetFPGA board, then just use a PC instead.

Configuration Summary

OpenFlow Switches

Switch	IP address to talk to the controller	Controller Ethernet Port	OpenFlow Ethernet Ports
Switch1 (PC1)	192.168.0.1/255.255.255.0	eth0	eth1, eth2
Switch2 (PC2 with NetFPGA board)	192.168.0.2/255.255.255.0	eth0	nf2c0, nf2c1, nf2c2, nf2c3
Switch3 (PC3)	192.168.0.3/255.255.255.0	eth0	eth1, eth2

OpenFlow Controller (NOX 0.6)

IP address	TCP port number
192.168.0.100/255.255.255.0	6633

2. Hardware Requirements

PCs for OpenFlow Switches

These are PC1, PC2, PC3 shown in the figure. We'll run OpenFlow software reference design on those PCs. Any PC running Linux 2.6 kernel would work, but there are some dependency on the linux distribution. Please check the release notes in OpenFlow reference design. Depending on the network topology you would like to create, you need to install NICs (network interface cards) on those PCs. Note that NetFPGA only work with 1Gb/s ethernet (not 100Mbps or 10Mbps), so the interface connecting to the NetFPGA interface has to be gigabit ethernet NIC. In the configuration shown in the figure, you need three ethernet ports on each PC.

PC for OpenFlow Switch with NetFPGA card

Go User Guide page in the NetFPGA website (http://netfpga.org/foswiki/bin/view/NetFPGA/OneGig/Guide) to see the requirement of the PC and how to get NetFPGA card. Pre-built system is also available.

PC for OpenFlow Controller

This is for PC4 shown in the figure. Currently we have three different types of OpenFlow controllers, NOX, SNAC and Reference Controller. In this page, we use SNAC 0.6 for the controller. Any Linux PC would work, but we assume it runs Debian Stable (Lenny) in this page.

3. Setup OpenFlow Switches on PCs/NetFPGAs

3.1. Setup OpenFlow Switch on PC

Step1. Download Software

 $ git clone git://openflowswitch.org/openflow.git
 $ cd openflow
 $ git checkout -b openflow_089_rev4 origin/release/0.8.9 

Step2. Compile

 $ ./boot.sh
 $ ./configure --with-l26=/lib/modules/`uname -r`
 $ make

Step3. Run

Insert OpenFlow kernel module, ofdatapath.ko.

 $ su -
 # /sbin/insmod ./openflow/datapath/linux-2.6/ofdatapath.ko

Create OpenFlow switch datapath.

 # ./utilities/dpctl adddp nl:0
 # /sbin/ifconfig of0 hw ether 00:4E:46:32:43:04

Note that the ether HW address (00:4E:46:32:43:04) given to of0 interface (the LOCAL interface of OpenFlow switch) will be used as the datapath ID. The datapath ID has to be unique among the switches controlled by a single OpenFlow controller. If you don't specify this (i.e., don't run the second line of the commands), OpenFlow software will automatically assign random datapath ID, which differs every time you create OpenFlow switch datapath. If you want to assign consistent datapath ID, it is a good idea to use the MAC address of one of the ethernet ports as follows (because it is guaranteed to be unique):

 # ifconfig of0 hw ether `ifconfig eth0 | grep HWaddr | awk '{print $5}'`

Then, add interface to the datapath. Interfaces names (eth1, eth2) depend on your system.

 # ./utilities/dpctl addif nl:0 eth1
 # ./utilities/dpctl addif nl:0 eth2

Then let OpenFlow protocol module talk to the controller. Here we assume the controller is running on 192.168.0.100 port 6633.

 # ./secchan/ofprotocol nl:0 tcp:192.168.0.100:6633

3.2. Setup OpenFlow Switch on NetFPGA

Step0. Basic Setup of NetFPGA

Please refer http://netfpga.org/foswiki/bin/view/NetFPGA/OneGig/Guide to setup NetFPGA.

Step1. Download Software

 $ git clone git://openflowswitch.org/openflow.git
 $ cd openflow
 $ git checkout -b openflow_089_rev4 origin/release/0.8.9 
 

Step2. Compile

 $ ./boot.sh
 $ ./configure --with-l26=--with-l26=/lib/modules/`uname -r`/build --enable-hw-tables=nf2  
 $ make

Step3. Run

Program NetFPGA card as OpenFlow switch.

 # /root/NF2/lib/scripts/cpci_reprogram/cpci_reprogram.pl 
 # nf2_download ./datapath/hwtable_nf2/openflow_switch.bit

Insert OpenFlow kernel modules, ofdatapath.ko and ofdatapath_netfpga.ko (Note: the order of insert matters).

 
$ su - 
# /sbin/insmod ./openflow/datapath/linux-2.6/ofdatapath.ko 
# /sbin/insmod ./openflow/datapath/linux-2.6/ofdatapath_netfpga.ko 

Create OpenFlow switch datapath.

 
 # ./utilities/dpctl adddp nl:0
 # /sbin/ifconfig of0 hw ether 00:4E:46:32:43:06

Then, add all the NetFPGA interfaces (nf2c0, nf2c1, nf2c2, nf2c3). Always add all the NetFPGA interfaces, otherwise it won't work.

 
 # ./utilities/dpctl addif nl:0 nf2c0
 # ./utilities/dpctl addif nl:0 nf2c1
 # ./utilities/dpctl addif nl:0 nf2c2
 # ./utilities/dpctl addif nl:0 nf2c3

Then let OpenFlow protocol module talk to the controller. Here we assume the controller is running on 192.168.0.100 port 6633.

 # ./secchan/ofprotocol nl:0 tcp:192.168.0.100:6633

We recommend adding the above commands in a of-start.sh shell script. Similarly, create an of-stop.sh script with the necessary teardown commands with:

$HOME/openflow/utilities/dpctl delif nl:0 nf2c0
$HOME/openflow/utilities/dpctl delif nl:0 nf2c1
$HOME/openflow/utilities/dpctl delif nl:0 nf2c2
$HOME/openflow/utilities/dpctl delif nl:0 nf2c3
$HOME/openflow/utilities/dpctl deldp nl:0
/sbin/rmmod $HOME/openflow/datapath/linux-2.6/ofdatapath.ko 
/sbin/rmmod $HOME/openflow/datapath/linux-2.6/ofdatapath_netfpga.ko

4. Controller Setup

We'll setup NOX 0.6 here. For the complete information, visit NOX website.

Step0. Install Pre-requisite Packages

Prerequiste packages depend on the linux distribution on the controller PC. In the case of Debian Stable (Lenny), you have to install the following packages:

 
$ apt-get install autoconf automake g++ libtool python python-twisted \ 
       swig libboost1.35-dev libxerces-c2-dev libssl-dev make        \
       libsqlite3-dev python-simplejson        \
        python-sphinx

Step1. Download Software

 $ git clone git://noxrepo.org/nox
           $ cd nox

By default, you'll get OpenFlow v0.8.9 compatible NOX (as of 5/20/2010). If you want to OpenFlow v1.0 compatible NOX, then you need to checkout the corresponding branch. Here is the instruction:

 #### this is only for OpenFlow v1.0 user
   $ git branch -a
      * master
      remotes/origin/HEAD -> origin/master
      remotes/origin/destiny
      remotes/origin/master
      remotes/origin/openflow-0.9
      remotes/origin/openflow-1.0
   $ git checkout -b nox_v06_ofv1.0 remotes/origin/openflow-1.0
   $ git branch -a
        master
      * nox_v06_ofv1.0
         remotes/origin/HEAD -> origin/master
         remotes/origin/destiny
         remotes/origin/master
         remotes/origin/openflow-0.9
         remotes/origin/openflow-1.0

Step2. Compile

 
$ ./boot.sh
$ mkdir build
$ cd build/
$ ../configure
$ make

Step3. Run

Run NOX with routing module only. Routing module will do the shortest path routing.

 
$ cd src
$ ./nox_core -i ptcp:6633 routing

5. Basic Test

On the controller PC, first check the switches are connected to the controller. You can check this by

$ netstat -an | grep 6663

Assign the IP addresses to Client1 and Client2 (on the same subnet) and run ping to check they can communicate. On each switch, you can run the following command to see the flow table is actually installed.

$ cd <your OpenFlow comipled dir>
$ ./utilities/dpctl dump-flows nl:0