|
|
ViewsIndigoReleaseNotesFrom OpenFlow WikiIndigo Release Notes Indigo OpenFlow Switching Software Package
Stanford Broadcom Binary Code Release
OpenFlow Version 1.0
Stanford University
June 2010
History indigo-1.0-beta-2: Released 7/22/10
Fixes for VLAN tag processing. Dataplane management.
indigo-1.0-beta: Released 6/28/10
Version information is now reported.
indigo-1.0-alpha-5: Released 6/7/10
Hardware addresses are now assigned to each OpenFlow Switch port
indigo-1.0-alpha-4: Released 5/21/10
Patch version of indigo-1.0-alpha-3. Allow DPID to be set
from boot parameters. Fixed crash on port modification.
Support no_flood port configuration setting. Watchdog is
now "opt-in" rather than "opt-out".
indigo-1.0-alpha-3: Released 4/19/10.
Patch version of indigo-1.0-alpha-2 addressing flow insertion
performance issue. Watchdog feature add.
indigo-1.0-alpha-2: Released 2/22/10.
Patch version of indigo-1.0-alpha addressing critical memory
leak.
indigo-1.0-alpha: Released 2/17/10.
Initial userspace Stanford Reference Release for LB4G.
OverviewThis software release includes binary files that can be used to boot Broadcom based Quanta LB4G Ethernet switch and the Pronto 3240. Future support for the the Broadcom 956634 reference design is planned with preliminary support for the 956820 reference design to follow. Throughout this document references to the LB4G apply equally to the Pronto 3240. The distribution supports the Quanta hardware as an OpenFlow native switch with hardware flow support. It is based on OpenFlow version 1.0. Materials This release includes:
u-boot.bin: The U-Boot boot loader for the LB4G.
uImage-lb4g: A binary image of the Linux kernel for the LB4G
uInitrd2m-lb4g: A binary image of the ram disk root file for
the LB4G
RELEASE_NOTES: This text file.
SFS_usage.txt: A text file explaining the use of SFS
license: A directory containing additional license info
License InformationThe Linux kernel code for the LB4G is based on version 2.6.15 and is released under GPLv2. The Linux kernel code for the Broadcom reference designs is based on version 2.6.25 and is released under GPLv2. U-Boot is based on version 1.1.6 and is released under the GPLv2. It is available at http://www.denx.de/wiki/U-Boot. Patches from Quanta have been applied and are available from Stanford. Busybox is version 1.4.2 and is released under the GPLv2. It is available at http://busybox.net/downloads/. Dropbear is version 0.52; see the accompanying file in licenses/dropbear for additional information. The OpenFlow code is covered by the OpenFlow license detailed at http://www.openflowswitch.org/wp/legal/. It is available at http://www.openflowswitch.org/. The linux-bcm-core.ko file is released as a binary-only file under the OpenFlow license (see above). The of-bcm-cli code is released as a binary-only file under the OpenFlow license (see above). This CLI itself is copyright Broadcom.
Getting Started NOTE TO USERS of the original LB4G release: If you placed a
system_init or system_config file in your sfs/ directory (see
below) you will need to update it before booting to an indigo
image.
The file IMAGE_SETUP explains the steps you will need to get the
switch set up and loaded with the new image. This includes:
Connecting to the switch over a serial port which will be
necessary in order to configure the switch.
Updating the bootrom to U-Boot which is necessary to load the
Linux kernel for the LB4G platform. The CFE boot loader on
the Broadcom Reference designs should not need to be changed.
Loading the Linux kernel and ram disk images into flash.
You can reference this document on line at
http://www.openflowswitch.org/wk/index.php/IndigoImageSetup
General Notes and Limitations This software distribution is intended to make the target
hardware into a pure OpenFlow switch. This means that
non-OpenFlow configuration is limited.
VLANs are an example of configuration that is often of concern.
In order for OpenFlow to control VLANs:
* One VLAN is reserved as the system's untagged VLAN
* Incoming untagged packets are mapped to this VLAN; if no
other VLAN operations are carried out on the packet, it will
exit the switch untagged.
* All other VLANs are created and all ports are added as
"tagged" ports. This is because the VLAN tag could be
wildcarded and would then apply to all VLAN ids.
In order to strip a tag, a packet is mapped to the untagged VLAN
which will result in the VLAN tag being stripped on egress. Note
that an explicit "modify VLAN id to the untagged VLAN" will
usually have the same effect. Additionally, see item 21 below
regarding Trac issue 104.
Due to the way VLAN actions are currently implemented, the number
of flows which act on VLAN tags may be more limited than the
number of other types of flows. Currently there is a limit of 256
such flows.
For systems without an external management port, the dataplane
management port is reserved for this use and should not have
OpenFlow rules associated with it. It is not yet known if this
has implications in interoperability with FlowVisor.
The MTU of switch ports defaults to the Ethernet standard of 1518
(1522 for VLAN tagged packets). This is not currently
configurable.
Bringing Up the Switch: LB4G Once the system is able to boot Linux, you will see the ~ $ prompt
after booting. At this point, the core system support modules
have been loaded, followed by the Broadcom hardware support
modules.
Normal initialization proceeds with the execution of /etc/rc.sh.
If you are booting from flash, this file cannot be changed. To
provide flexibility in the initial parameters and sequence, rc.sh
will normally create the directory /sfs/ and copy flash data into
this file system. It will then check if the file /sfs/system_init
exists and, if so, execute that file.
If not, a default version /etc/system_init.default will be
executed. This file brings up network interfaces, starts ssh,
and inserts additional user/kernel communication modules. It will
then try to execute the file /sfs/system_config. If that does not
exist, the file /sfs/system_config.default is executed.
The system_config file is normally where the OF-BCM shell is
started and the ofprotocol daemon to communicate to the controller
is started. See the default version of the file for more details.
In the default setup, the following boot parameters may be
configured. Recall that the management interfaces on the LB4G are
labeled eth1 and eth2, but appear in the Linux shell as eth0 and
eth1.
In system_init:
ETH0_IP The IP address to use for eth0
ETH1_IP If defined, bring up eth1 with this address
GATEWAY If defined, use this as the gateway IP
no_sysconf If set, exit after inserting core system modules
This avoids starting OF datapath and OF protocol
no_ssh If set, do not start SSH daemon
start_telnet If set, start telnetd
In system_config:
CONTROLLER If defined, contact controller at this IP address
CTRL_PORT If defined, use this port number for the controller
Notice that by default SSH is started and telnet is not. The
username for SSH is root and the password is OpenFlow.
The boot loader appears to allow lines up to approximately 250
characters. Also, it's a bit picky about quotes in its comands.
Thus it is suggested that you define variables successively with
commands like:
=> setenv extra_boot_args $extra_boot_args new_variable=new_value
Variables up to 160 characters have been tested and boot arguments
over 260 characters have been tested. Once set, make sure the
extra arguments are visible to the boot command:
# Add $extra_boot_args to the boot command
=> setenv bootcmd 'setenv bootargs root=/dev/ram rw \
console=$consoledev,$baudrate $extra_boot_args; \
bootm ffd00000 ff000000'
# Save the changes to flash
=> saveenv
Normally the output from the CLI is sent to /cli.log
If you wish further customization of the switch's initialization,
you can generate your own system_init and/or system_config file in
an SFS file system. This file system is stored in the /sfs/
directory and saved to flash with the sfs_create command.
Note that if you had a system_config file in your sfs/ directory
with a previous release of this software, you will need to update
it to work with Indigo. See the system_config.default file in the
root file system.
The execution of system_config may be disabled (if the default
system_init is run) by setting the environment variable no_sysconf
in the Linux boot arguments. If this variable is not defined, and
if an SFS file system exists and the file system_config is found,
then that file is executed. Otherwise, the file
/etc/system_config.default is executed.
Once you have configured the /sfs directory as you like for system
initialization, use the command /sbin/sfs_create to save it to
flash. Direct use of the sfsctl utility is not recommended.
Interacting With the OpenFlow/Broadcom CLI Note that there are two CLIs involved here: The Linux command
line with the prompt ~ $ and the OpenFlow/Broadcom CLI with the
prompt OF-BCM.0>. Below, CLI refers the the OpenFlow/Broadcom
CLI.
If the above initialization sequence is used, the CLI will not be
available for interactive use. If the system is configured
manually (for example, by setting no_sysconf=1 in
extra_boot_args) then the CLI may be accessed to provide limited
control and monitoring of the switch. Here is a sample sequence:
* System brought to ~ $ prompt with no_sysconf=1 set.
* Connected via serial port or by ssh (user: root, default
password is OpenFlow). You'll need two windows (serial or
ssh).
* Edit /etc/rc.soc to comment out the call to ofd
+ You can create a new rc.soc in your sfs system; the
default system_config will overwrite the existing one on
boot.
* In one window, execute /sbin/of-bcm-cli.
+ See below for using the CLI
+ Run the OpenFlow datapath in the background:
OF-BCM.0> setenv portlist 1,2,3,4,5,6,7
OF-BCM.0> bg "ofd --interfaces=$portlist ptcp:"
+ Use ? for help in the of-bcm shell.
* In another window a the Linux prompt, start the ofprotocol
executable to connect the datapath and controller:
~ $ ofprotocol tcp:127.0.0.1 tcp:172.16.12.12:6634
* You can use /sbin/dpctl to examine the status of ofdatapath
as normal
Highlights of the CLI:
* Use ? for help
* There are two physical devices controlled independently.
Use 0: and 1: to switch between them. Creating a VLAN on
one, for example, will not create the VLAN on the second.
The "current" device is indicated in the prompt.
* Ports are referred to as ge0, ge1, ... on each switch
device. The OpenFlow port numbers map ge0 -> 1, ge1 ->2,
etc., for the first device and ge0 -> 25, ge1 -> 26, etc.,
for the second device.
* The command 'show' will give a variety of MIB counters for
ports.
* The command 'ofdatapath' takes the same arguments as the
user-datapath executable of the same name in the OpenFlow
reference software release.
* The shell command language is similar to csh and provides
various control structures and environment features
* Only a limited number of switch control features are
available to control ports and VLANs
Notes on Specific FeaturesDataPlane ManagementFor platforms without network connections external to the switch fabric, special support has been added under the name dp-mgmt. This is signaled to the OpenFlow driver by passing the argument --dp-mgmt to ofdatapath. The (OpenFlow) port number may be specified with the additional argument --dp-mgmt-port=<port>. Although this special port is given an OpenFlow number and this OpenFlow number may be exposed to the controller, no flows should be made that affect the dp-mgmt port. This is currently enforced in the driver where flows that refer to the dp-mgmt port are modified so as to remove it. Version Update NotesChanges from indigo-1.0-alpha to indigo-1.0-alpha-2Addition of dropbear SSH server support. Set default root password to OpenFlow. Addressed hardware issue where management ethernet ports would be inoperative after a soft reset. Limited some debug output to CLI. Decouple packet receive thread using queue into buffers for main event loop. Resolved memory leak on broadcast packets transmitted from the switch's CPU. Changes from indigo-1.0-alpha-2 to indigo-1.0-alpha-3Added support for watchdog timer. Disable with kernel parameter no_watchdog=1 or set no_watchdog=1 in config.bcm. If enabled and software freezes, the system will reset in about 1 minute. A separate thread collecting counters has been deployed to address the issue that flow timeout and expirations were taking a long time. A major bottle neck for flow insertions was addressed by changing how flow priorities are handled in the table. This addresses a major performance issue for exact match flows. High throughput of wildcarded flows will still result in some performance issues. At the CLI, the command 'ofd' without parameters now prints out various counters and other state information. Changes from indigo-1.0-alpha-3 to indigo-1.0-alpha-4The watchdog timer is now "opt-in" rather than "opt-out". That is, specify "watchdog=1" in the arguments to the kernel to enable the software watchdog. (Previously, you specified "no_watchdog=1" to disable it.) The datapath ID may be specified in the kernel arguments as "dpid=0123456789abcdef". Note that the DPID MUST BE exactly 12 hex digits. Previously, the no_flood bit in the port configuration was ignored. This has been updated so that the no_flood bit can be set for the port (which also caused a segfault). However, changing the setting while running will not update existing flows that use the flood action. Changes from indigo-1.0-alpha-4 to indigo-1.0-alpha-5Previously the hardware addresses associated with hardware controlled (dataplane) OpenFlow ports were always reported as all 0. This caused obvious problems for controllers using a link layer discovery protocol that used those addresses. The current approach is to use the OpenFlow OUI (00:26:E1), a random number for the next 16 bits and the port number for the bottom 8 bits. Changes from indigo-1.0-alpha-5 to indigo-1.0-betaThe version, indigo-1.0-beta, is reported in various places including: during the system initialization script rc.sh, via the 'version' command at the Linux prompt (which prints the contents of /etc/indigo-version) and in the of-bcm-cli via the command 'show version'. Changes from indigo-1.0-beta to indigo-1.0-beta-2Support for dataplane management added. This release includes changes to support VLAN tag processing. In previous releases, VLAN modification actions were rejected by the hardware driver and so were not properly applied. The implemented solution to this configures all VLANs on the switch since it is not known what VLAN tags may appear at ingress and flow modifications with the VLAN ID wildcard may be specified. See Known Issues 22 and 23 below.
Known Issues 1. This release has received only limited validation and many
features are untested. Expect bug-fix releases soon.
1A. RESOLVED in release indigo-1.0-alpha-2. (Trac #64)
There is a known memory leak in the image. It appears to
use memory on a moderately loaded switch (about 160 active flows
at a time with a lot of broadcast arp traffic) at the rate of
about 2-3 % per minute. On an isolated system with lower flow
table turnover (~ten flows/minute) the system has stayed up for
5 days or more.
2. (Trac #79) The datapath includes both a HW table and a SW
table. The HW table may "shadow" the SW table if overlapping flow
descriptions are placed in the two tables (specifically, if lower
priority descriptions are put in the HW table). No checks are
currently done by the OpenFlow driver to prevent this, although it
should not be an issue unless either the hardware table becomes
full or actions that are not supported by hardware are used.
3. Similar to (2) above, if the HW flow table space is exhausted,
all following flows will be placed in the SW flow table which may
result in the "priority inversion" described above.
4. (Trac #69) Per-flow counters: For flows, only packet counters
or byte counters are supported, not both simultaneously. In the
configuration of this release, only packet counters are supported.
5. The switch has only been tested in an untagged environment,
although it should pass through tags transparently to the
controller and should support matching and rewriting of VLAN
tags. You can create a VLAN for the purposes of being untagged by
using the CLI discussed above.
6. Not all OpenFlow actions are supported. The following are not
currently supported:
OFPAT_SET_DL_SRC (Set L2 MAC source address)
OFPAT_SET_DL_DST (Set L2 MAC destination address)
OFPAT_SET_NW_SRC (Set L3 IP source address)
OFPAT_SET_NW_DST (Set L3 IP destination address)
OFPAT_SET_TP_SRC (Set L4 source port)
OFPAT_SET_TP_DST (Set L4 destination port)
OFPAT_STRIP_VLAN (Strip VLAN tag): This may be supported
by setting to the system's untagged VLAN, but that is
untested.
7. Actions sequencing is not supported. For example, the action
list with 4 actions: "(1) set VID to X, (2) output to port A, (3)
set VID to Y, (4) output to port B" has the intention of sending
the packet out on different ports with different VLANs. This is
not supported. However, sending to multiple ports by specifying
each port in a different output action is supported.
8. Actions with a wildcard source port may not specify "flood"
nor multicast with the expectation of source port blocking.
9. RESOLVED: There is an issue with the management ports
(labelled eth1 and eth2 on the front pane)l. These ports may
become inoperable after a reset, particularly from the boot
prompt. A power cycle appears to fix the problem. This issue is
under investigation by Quanta.
10. There are observed issues where dpctl is used locally when
the switch is undergoing heavy use. The switch driver's transmit
thread is starved and fails. This eventually results in DMA
packet free failures. Use of dpctl is currently discouraged when
the switch is connected to a controller and under heavy load.
11. COS control is not currently supported in the CLI. This will
be added with more complete queuing support. Queues may be created
using dpctl and the enqueue action is supported, though not
completely verified.
12. (Trac #68) On certain exit conditions from the CLI, the
terminal echo will be disabled, though command processing
continues.
13. (Trac #74) The "reason" value for packet_in commands may not
accurately reflect the reason the packet was forwarded to the
controller.
14. (Trac #71) ADDRESSED in alpha-3 release, though not fully
resolved. Flow set up time is still very slow (and has not
been fully evaluated). In alpha-3, we have measured a reliable
100 flows/second (including packet forwarding through the controller
and flow setups) with hping.
15. (Trac #91) ADDRESSED in alpha-4 release. The NO_FLOOD setting
of the port configuration was ignored. So the flood action sent
the packet out to all ports independent of this setting.
16. (Trac #94) If the NO_FLOOD setting for a port is changed, the
change applies only to new flows. Existing flows with the FLOOD
action will not be updated.
17. (Trac #95) Repeated re-connections with the controller have
been seen with Nox 0.6. This can result in network instability.
18. (Trac #98) RESOLVED in alpha-5. The dataplane ports MAC
addresses were always set to all 0.
19. (Trac #96). The ordering of actions in an action list may
not be respected. Normally the switch is supposed to reject a
flow mod if it cannot carry out the actions in the order specified
in the flow mod's action list. This does not happen.
20. (Trac #102). RESOLVED in beta-1. Switch capabilities were
being incorrectly reported. Only the capabilities of the hardware
will be reported for actions that can be applied to a flow.
21. (Trac #105). RESOLVED in beta-1. VLAN tag modification
actions resulted in error when installed.
22. (Trac #104). Differentiating on VLAN properties only may
result in flow collapse. If two flows differ only in VLAN tagging
characteristics and the VLAN tags are manipulated (the VLAN tag is
stripped or the VID is changed or an untagged packet gets a new
VLAN tag) then treatment of the two flows for some operations may
be combined.
The work around is to ensure that flow modifications have some
additional qualification to differentiate flows.
No known solution to this problem currently exists.
23. There is currently no support for virtualizing the switch by
way of VLAN tags. All VLANs are reserved for use by OpenFlow.
Other NotesThe MTU of switch ports defaults to the Ethernet standard of 1518 (1522 for VLAN tagged packets). This is not currently configurable. |
Quick NavigationOpenFlow White Paper Download the OpenFlow Whitepaper (PDF).
OpenFlow Demo Video Watch the Demo that received the best demo award at SIGCOMM 2008. About OpenFlow OpenFlow is supported bythe Stanford Clean Slate Program. Wiki ToolsPersonal toolsProjects |