PSPace User Guide

This document describes the usage of PSPacer (Precise Software Pacer), which achieves precise network bandwidth control and smoothing of bursty traffic without any special hardware. PSPacer is implemented based on the iproute2 which is the traffic control framework for the Linux platform. This framework provides three basic elements: queueing discipline (qdisc), class, and filter [lartc]. PSPacer consists of two parts: a loadable kernel module and a user-level library. The former (i.e. sch_psp) is a classful qdisc, and the latter (i.e. q_psp.so) communicates with this module for control and monitoring it via the tc (8) command.

PSPacer uses the IEEE 802.3x PAUSE frame as a gap between packets. Therefore, you can not use the PAUSE frame to stop transmission from the switch/router to which the system is connected. It is recommended to disable IEEE802.3x flow control function of the network switch (to which a PC with PSPacer is connected) in order to avoid unexpected behavior. For your information, see our PFLDnet2005 paper [PFLDnet05], and the Web page http://www.gridmpi.org/pspacer.jsp.

Since the kernel and the iproute2 package are tightly coupled, you must use a proper combination. We have tested this package using the FedoraCore 3 on IA32 machines with the Intel PRO/1000, and the FedoraCore 5 on x86_64 machines with Broadcom BCM5704. The complete list of tested environments is in the Appendix B.

The following source codes are required for the compilation:

The kernel re-compilation is not required, however the compilation of the sch_psp module requires the kernel source code. If you use the RedHat Linux based distribution, install the kernel-devel (or kernel-smp-devel) RPM. For your information, if you have built the custom kernel, make sure that the CONFIG_NET_SCHED and CONFIG_NET_CLS options have been selected.

Make sure the current working version of iproute2 is available in your Linux box. To check the version, issue the following command:

Get the iproute2 archive file from the iproute2 web page [iproute2], and extract it to /opt/iproute2 directory.

(EXPERIMENTAL) On version 2.1 or later, it provides to support libnl - netlink library for applications to set parameters of PSPacer. To enable libnl with PSPacer support, get the libnl library from libnl web page [libnl].

To make and install this software, issue the following commands:

'./configure --help' shows all options.

“make install” will install the following files. Please check these files.

You can do configure without extracting the iproute2 package. Issue the configure script with the option --with-iproute2-pkg as follows:

In above case, the iproute2 package is extracted on the current directory.

To enable libnl library with PSPacer support, issue the following commands:

Alternatively you can install from a RPM package. To build the RPM package from the source tarball, issue the following command:

And then, install the RPM package.

(EXPERIMENTAL) For Debian GNU/Linux users, you can also install from a Debian package. To build the Debian package from the source tarball, issue as follows:

And then, install the Debian package.

As an example, here we assume the following network configuration which connects three local area networks (LANs): the network A (192.168.1.0/24), the network B (192.168.2.0/24), and the network C (192.168.3.0/24).

On each LAN, PCs are connected by Gigabit Ethernet, and then the bottleneck link’s bandwidth of the link 1 and the link 2 are 500 Mbps and 200 Mbps, respectively. Both RTTs are 200 ms. In addition, we change the txqueuelen value to 10000 as follows (see Interface queue size):

To classify traffic toward each others' network, we will create a traffic control tree which has three leafs shown below:

Each class has its classid and each qdisc has its handle. A classid or a handle consists of two parts, a major number and a minor number: [major]:[minor]. It is customary to name the root qdisc "1:", which is equal to "1:0". The minor number of a qdisc is always 0.

In this example, traffics of the link 1 and 2 are controlled by qdiscs 10: and 20:, respectively, and other (unclassified) traffics are controlled by qdisc 30:. Here qdisc 10: and 20: correspond to the leaf class 1:1 and 1:2 respectively, and qdisc 30: corresponds to the leaf class 1:3, which is the default class. Default class is a class to which packets are routed if there is no filter specifications. And then each class except the default class has the target rate for pacing. In this example, the target rate of the link 1 and 2 should be regulated to 500 Mbps and 200 Mbps, respectively.

You can set/get parameters to the sch_psp qdisc via the tc (8) command. The general information of the tc command can be found in the man page and the Web page [lartc]. The man page of tc-psp is also included in this package.

Each interface has one root qdisc. To use PSPacer, the root qdisc should be the sch_psp. First, to add the sch_psp as the root qdisc, and set the default class as 1:3, issue the following command:

Next, to add leaf classes to the root qdisc, issue the following commands:

Next, we add sub qdiscs to the leaf classes. Here we use PFIFO qdisc. This is the default qdisc which has 3-band FIFO queue. Unless you want to do something special with another qdisc, use PFIFO.

To add PFIFO (sub) qdiscs within each leaf class, issue the following

Filters are used to classify traffic into the proper sub-qdiscs. Finally, to attach two filters to root qdisc, issue the following commands:

Here we use the u32 filter in order to classify traffic by destination IP addresses. Use u32 unless you want to do something special using another filter. By issuing these commands, packets whose destination are 192.168.2.0/24 are controlled by qdisc 1:1 while packets whose destination are 192.168.3.0/24 are controlled by qdisc 1:2.

Now, the configuration has been completed. To print the current parameter settings and statistics, issue the show option of the tc command. The -s option means statistics, and -d option means detail.

Let’s examine what we have created. First, you can see the root “psp” qdisc and three “pfifo” sub-qdiscs:

Here “max rate” shows the maximum transmission bandwidth of the network interface.

Next, you can see three “psp” classes:

Finally, you can see two “u32” filters:

You can confirm the default class configuration by using ping. Here ping generates unclassified traffic.

As you can see, all traffic to 192.168.1.2 is transmitted through qdisc 30:, which is the qdisc under the default (unclassified) class.

Next, we do some bulk data transfer by using iperf:

As you see, all traffic is via qdisc 10:, and then the iperf throughput is about 480 Mbps. This rate is the half of the physical bandwidth, thus the amount of gap packets is about the same amount of real packets.

Some parameters can be modified in place. To change parameters, issue the following command:

The following is an example of a class removal:

The following is an example of a qdisc removal:

If you remove the root qdisc, all classes and qdiscs are also removed.

To install and uninstall the sch_psp module, issue the following commands, as super user:

The system should have capability to transmit packets at the maximum rate of the connected network link to realize precise pacing.

If your PC is cabled by Gigabit Ethernet, your network interface should be connected to the PC by PCI-Express, PCI-X, 66MHz/64bit PCI, or CSA. Otherwise, the I/O bus becomes a bottleneck, and maximum transmission rate can not be achieved. The processor itself may have enough performance if it is Pentium 4 or above. If your PC is cabled by Fast Ethernet, most of today’s PCs may have enough performance.

This is an independent matter from PSPacer. Here are some tips.

Modify kernel parameters using sysctl(8) command. To change TCP settings, add the entries below to the file /etc/sysctl.conf, and then run "sysctl -p".

The optimal socket buffer size is twice the size of the bandwidth \* delay product (BDP) of the link. However, the default max TCP buffer size is too small on long fat networks. For example, to fill a 125MB/s (i.e. 1Gbps) pipe with 100ms one-way latency requires 125 \* 0.2 = 25MB of data. To change the socket buffer size, do the following:

Another thing you can try is to increase the size of the interface queue (i.e. the transmit queue of the network interface). To do this, issue the following using ifconfig (8) command:

netdev_max_backlog specifies the maximum length of the input queues for the processors. The default value is 300 (packets). Linux has to wait up to scheduling time to flush buffers (due to bottom half mechanism). Therefore, this value can limit the network bandwidth when receiving packets as follows:

If you want to get higher throughput, you need to increase netdev_max_backlog:

When you run benchmarks and other experiments, it is often useful not to save the route metrics (cwnd, rtt, etc) from each connection. You can turn off caching the metrics:

The same effect can be achieved by executing this before each TCP session:

PSPacer uses raw bandwidth (i.e. data link layer bandwidth) which includes IP, TCP or UDP headers as the target rate. On the other hand, iperf shows payload bandwidth. Therefore, iperf shows lower bandwidth than the target rate.

In addition, especially when TCP/IP is used, the bandwidth may be regulated by congestion control or buffer size, etc. See "How can I achieve Gigabit speeds on Long Fat Networks using TCP/IP?".

Here we call a Linux PC on which a PSPacer installed a PSP box. The example below uses a PSP box to smooth traffic from LAN (1Gbps) to WAN (100Mbps, 20ms). Network Configuration

The network configuration is as follows:

If PC1 and PC2 are Linux boxes, configurations are as below. Of course you can use other network connected PCs or equipments.

To enable IP forwarding, issue the following command:

In the PSP box, issue commands as follows:

This error message implies that TCP Segmentation Offloading (TSO) is enabled. Because PSPacer is not supporting TSO, you have to disable TSO before you install it.

If you can see the following message by looking at the output of 'dmesg' command, TSO is enabled.

If so, you have to disable TSO by using 'ethtool' (8) command and retry 'tc qdisc add' as follows:

Table.1 and Table.2 show tested environments which we have tested.

Here we show some experimental results in a WAN emulation environment. In these experiments, a hardware network testbed, GtrcNET-1, emulates 200ms RTT latency. We measured the the bandwidth generated by iperf with a milli-second resolution.