PSPACER RELEASE NOTES

2010-03-31

CREDIT

This software was developed by the National Institute of Advanced Industrial Science and Technology (AIST), Japan. A part of the development was supported by a grant from the Ministry of Education, Sports, Culture, Science and Technology (MEXT) of Japan through the NAREGI (National Research Grid Initiative) Project. PSPACER IS LICENSED UNDER THE GNU GPL.

WHAT IS PSPACER?

PSPacer (Precise Software Pacer) is a qdisc module which realizes precise transmission bandwidth control. It makes bursty traffic which is often generated by TCP smooth without any special hardware.

Bursty traffic can degrade the communication performance, because it causes buffer overflow at intermediate network nodes and results in packet losses. In a bursty traffic, packets are sent back to back. By adding a short pause in between the packets, traffic bursts can be avoided.

PSPacer controls the interval between outgoing packets very precisely. The key idea of PSPacer is to determine transmission timing of packets by the number of bytes transferred. If packets are transferred back to back, the timing a packet is sent can be determined by the number of bytes sent before the packet. PSPacer fills the gaps between time aligned "real packets" (the packets which are sent by user program) by "gap packets". The real packets and gap packets are sent back to back, and thus the timing of transmission of each real packet can be precisely controlled by adjusting the gap packet size. As the gap packets, the IEEE 802.3x PAUSE frames are used. PAUSE frames are discarded at a switch input port, and only real packets go through the switch keeping the original intervals.

In the past, some software-based pacing schemes have been proposed. These schemes use timer interrupt based packet transmission timing control. Therefore, to achieve precise pacing, they require the operating system to maintain a high resolution timer, which could incur a large overhead.

For detailed description and the usage of PSPacer, please refer to our project page (http://www.gridmpi.org/pspacer.jsp), and the paper "Design and Evaluation of Precise Software Pacing Mechanisms for Fast Long-Distance Networks," in PFLDnet2005.

Note	PSPacer version 2.x does not support the Linux kernel 2.4! If you want to run PSPacer on the Linux kernel 2.4, you should use PSPacer version 1.x.

LICENSE         - The GPL license.
Makefile        - Makefile.
README          - This file.
ChangeLog       - The ChangeLog.
dectd/          - dectd: IP flow-based traffic control automation daemon.
doc/            - Miscellaneous documents (user guide, paper).
kernel/         - The kernel module.
man/            - The man pages.
scripts/        - Some sample setup/test scripts.
tc/             - The shared library for tc (8).

LIMITATIONS

(1) PSPacer controls the bandwidth according to the ratio of the target bandwidth in the maximum transmission bandwidth of the system. Therefore, the system (computer, network interface, operating system, buffer settings, etc.) should have a capability to transmit packets at the maximum transmission rate (i.e. 1 Gbps for 1000BASE, 100 Mbps for 100BASE) to realize a precise pacing.

For example, if a 33MHz/32bit PCI bus is used to connect a Gigabit Ethernet network interface, the bottleneck is the PCI bus, and the system can not transmit packets at the rate of gigabit/sec. In this case, you can explicitly specify the maximum transmission bandwidth using qdisc’s "rate" parameter as follows:

# /sbin/tc qdisc add ... psp ... rate 700mbit
                                 ^^^^^^^^^^^^

However, in this case, the output traffic may be imprecise since the behavior of PCI bus is not stable enough.

Therefore, if you want to control Gigabit Ethernet traffic, we recommend to use PCI-X, 66MHz/64bit PCI or CSA connected network interface. If the total of target bandwidth of the output streams is less than 100Mbps, you can set the network interface to use 100BASE mode so as to obtain precise pacing. For the same reason, avoid using a shared switch (dumb hub) for the edge switch to which the PC with PSPacer is connected.

(2) PSPacer uses the IEEE 802.3x PAUSE frame as the gap between packets. Therefore, you can not use the PAUSE frame to stop transmission from the switch/router to the PC. Since PSPacer generates PAUSE frames with zero pause time, there should not be any side effects other than you can not stop transmission from the switch. However, it is recommended to disable IEEE 802.3x flow control function of the switch (to which a PC with PSPacer is connected) in order to avoid unexpected behavior.

(3) PSPacer can only work as a root qdisc since it produces (gap) packets itself and thereby violates the rule that a qdisc can only hand out packets that were enqueued to it. Using it as a leaf qdisc causes that qlen counters become inconsistent between itself and the upper qdiscs.

COMPILATION AND INSTALLATION

Installation from the source tallball

The following source codes are required for the compilation:

Linux kernel 2.6.x
iproute2

Note	Since the iproute2 package is tightly coupled with the kernel, you must use a proper combination. see TESTED ENVIRONMENT for more information.

Unpack the iproute2 tarball:

        $ scripts/dl_iproute2.py
        Success: iproute2-<version>.tar.gz
        $ tar zxvf iproute2-<version>.tar.gz -C /opt
        $ ln -s /opt/iproute2-<version> /opt/iproute2
or
        $ tc -V  # check the iproute2 version.
        tc utility, iproute2-<version>
        $ cd /opt
        $ wget http://devresources.linux-foundation.org/dev/iproute2/download/
          iproute2-<version>.tar.gz
        $ tar zxvf iproute2-<version>.tar.gz
        $ ln -s iproute2-<version> iproute2

Unpack the PSPacer tarball:
```
$ tar zxvf pspacer-<version>.tar.gz
```

Do make and install:

$ cd pspacer-<version>
$ ./configure
$ make
$ su
# make install

(OPTIONAL) To install dectd (see dectd/README), issue the following commands:
```
$ cd dectd
$ ./configure
$ make
$ su
# make install
```

To build for kernel different than the installed kernel, issue make command with KERNELRELEASE variable:

$ make KERNELRELEASE=2.6.27
$ su
# make KERNELRELEASE=2.6.27 install

Installation from the binary package

You can also build a RedHat RPM or Debian deb package from the tarball. See the PSPacer User Guide (doc/usage.en.txt).

SIMPLE USAGE

You check whether PSPacer works properly by using iperf as follows:

Setup

(add the PSPacer qdisc as the root qdisc)
# /sbin/tc qdisc add dev eth0 root handle 1: psp default 1
(add the PSPacer class whose target rate is 500Mbps)
# /sbin/tc class add dev eth0 parent 1: classid 1:1 psp rate 500mbit
(add the PFIFO qdisc as the sub qdisc)
# /sbin/tc qdisc add dev eth0 parent 1:1 handle 10: pfifo

Run iperf

$ iperf -c <server host> -i 10 -t 60
------------------------------------------------------------
Client connecting to XXXX, TCP port 5122
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.1 port 46457 connected with 192.168.1.2 port 5122
[  3]  0.0-10.0 sec    567 MBytes    476 Mbits/sec
[  3] 10.0-20.0 sec    567 MBytes    476 Mbits/sec
        :

Cleanup

(remove the PFIFO sub qdisc)
# /sbin/tc qdisc del dev eth0 parent 1:1 handle 10:
(remove the PSPacer class)
# /sbin/tc class del dev eth0 parent 1: classid 1:1
(remove the PSPacer qdisc)
# /sbin/tc qdisc del dev eth0 root handle 1:
(remove the PSPacer module)
# /sbin/rmmod sch_psp

For more information, see the PSPacer Usage Guide (doc/user.en.txt).

TESTED ENVIRONMENT

The software is tested on the following environments:

Linux distribution:
- CentOS 5 (kernel 2.6.18 + iproute2 ss061002)
- Ubuntu 9.04 (kernel 2.6.27 + iproute2 ss080417)
Network interface card:
- Myricom Myri-10G (PCI-Express)
- Intel PRO/1000 (PCI-X, CSA)
- Intel PRO/100
- Broadcom BCM5704 (PCI-X)