Home | GridMPI | GridTCP | Publications | Download

Curret Release: GridMPI-2.1.3
English | Japanese

GridTCP FAQ

PSPacer

What is the requirement for the PC and the network interface hardware to run PSPacer?

The system should have capability to transmit packets at the maximum rate of the connected network link to realize precise pacing.

If your PC is cabled by Gigabit Ethernet, your network interface should be connected to the PC by PCI-Express, PCI-X, 66MHz/64bit PCI, or CSA. Otherwise, the I/O bus becomes a bottleneck, and maximum transmission rate can not be achieved. The processor itself may have enough performance if it is Pentium 4 or above.

If your PC is cabled by Fast Ethernet, most of today's PCs may have enough performance.

How can I achieve Gigabit speeds on Long Fat Networks using TCP/IP?

This is an independent matter from PSPacer. Here are some tips.

Modify kernel parameters using sysctl(8) command. To change TCP settings, add the entries below to the file /etc/sysctl.conf, and then run "sysctl -p".

Socket buffer size

The optimal socket buffer size is twice the size of the bandwidth * delay product (BDP) of the link. However, the default max TCP buffer size is too small on long fat networks. For example, to fill a 125MB/s (i.e. 1Gbps) pipe with 100ms one-way latency requires 125 * 0.2 = 25MB of data. To change the socket buffer size, do the following: 

  # increase max TCP buffer size to 32MB (32 * 1024 * 1024)
  net.core.rmem_max = 33554432
  net.core.wmem_max = 33554432
Notice: Linux doubles the requested buffer size. So the above setting means that the max TCP buffer size is set to 64MB.

Interface queue size

Another thing you can try is to increase the size of the interface queue (i.e. the transmit queue of the network interface). To do this, issue the following using ifconfig(8) command: 

  # ifconfig eth0 txqueuelen 10000

Input queue size

netdev_max_backlog specifies the maximum length of the input queues for the processors. The default value is 300 (packets). Linux has to wait up to scheduling time to flush buffers (due to bottom half mechanism). Therefore, this value can limit the network bandwidth when receiving packets as follows: 

  300       *      1,000     =      300,000
  packets          HZ               packets/s

  300,000   *      1,000     =      300 M
  packets    averate (bytes/packet) throughput (Bytes/s)
If you want to get higher throughput, you need to increase netdev_max_backlog: 
  net.core.netdev_max_backlog = 2500
Notice: In the kernel 2.4.x, HZ is 100.

Turn off caching metrics

When you run benchmarks and other experiments, it is often useful not to save the route metrics (cwnd, rtt, etc) from each connection. You can turn off caching the metrics: 

  net.ipv4.tcp_no_metrics_save = 1
The same effect can be achieved by executing this before each TCP session: 
  # echo 1 > /proc/sys/net/ipv4/route/flush

Why iperf shows bandwidth lower than the target rate I set using tc command?

PSPacer uses raw bandwidth (i.e. data link layer bandwidth) which includes IP, TCP or UDP headers as the target rate. On the other hand, iperf shows payload bandwidth. Therefore, iperf shows lower bandwidth than the target rate.

In addition, especially when TCP/IP is used, the bandwidth may be regulated by congestion control or buffer size, etc. See "How can I achieve Gigabit speeds on Long Fat Networks using TCP/IP?".

Can I use PSPacer as an intermediate network box which regulates bandwidth of through traffic?

Here we call a Linux PC on which a PSPacer installed a PSP box. The example below uses a PSP box to smooth traffic from LAN (1Gbps) to WAN (100Mbps, 20ms).

Network Configuration

The network configuration is as follows: 

                    +-------+
                    |  PC1  | 192.168.1.2
                    +-------+
                        |         192.168.1.0/24
         ----------------------------------
                 |<-- 1Gbps (LAN)
                 |
                 | eth0 192.168.1.1
           +-----------+
           |           |
           |  PSP Box  |
           |           |
           +-----------+
                 | eth0 192.168.2.1
                 |
                 |<-- 100Mbps, 20ms (WAN)
         ----------------------------------
                        |          192.168.2.0/24
                    +-------+
                    |  PC2  | 192.168.2.2
                    +-------+

Setup PC1 and PC2

If PC1 and PC2 are Linux boxes, configurations are as below. Of course you can use other network connected PCs or equipments. 

  PC1:
  # ifconfig eth0 192.168.1.2
  # route add -net 192.168.2.0 netmask 255.255.255.0 \
    gw 192.168.1.1

  PC2:
  # ifconfig eth0 192.168.2.2
  # route add -net 192.168.1.0 netmask 255.255.255.0 \
    gw 192.168.2.1

Setup PSP Box

To enable IP forwarding, issue the following command: 

  # echo 1 > /proc/sys/net/ipv4/ip_forward

Setup PSPacer

In the PSP box, issue commands as follows: 

  # tc qdisc add dev eth0 root handle 1: psp default 2
  # tc class add dev eth0 parent 1: classid 1:1 psp rate 100mbit
  # tc class add dev eth0 parent 1: classid 1:2 psp mode 0
  # tc qdisc add dev eth0 parent 1:1 handle 10: pfifo
  # tc qdisc add dev eth0 parent 1:2 handle 20: pfifo
  # tc filter add dev eth0 protocol ip parent 1: pref 1 u32 \
    match ip dst 192.168.1.0/24 classid 1:1

[Contribution] For your information, psp-shaper script has been released by Mr. Denis Kaganovich, which may help for the above configuration.

When I install PSPacer, tc command fails with an error message "Unknown qdisc "psp", hence option "default" is unparsable"

tc dynamically loads a shared library q_psp.so at the startup time. If this library cannot be found, this error occurs. Make sure q_psp.so is installed to /usr/lib/tc.

NOTE: The Linux system run on 64 bit architecture search the shared library from /usr/lib64 instead of /usr/lib. In some distributions (e.g., Ubuntu 8.10), /usr/lib64 is just a symbolic link to /usr/lib. Please move q_psp.so to /usr/lib64/tc by hand, if you need.

NOTE: The CentOS version 6 and later install tc shared libraries into /usr/share/tc instead of /usr/lib/tc or /usr/lib64/tc. This problem will be fixed in the netxt release. Please check the "--with-tclib-dir" from messages of the configure script: 

  $ ./configure
  :
  Configuration
    --enable-debug                  no
    --with-iproute2-dir             /opt/iproute2
    --with-tclib-dir                /usr/share/tc
  :

Also, if the version of tc and that of q_psp.so are mismatched, this error may occur. Make sure the current working version of iproute2 is available in your Linux box. To check the version, issue the following command: 

  $ /sbin/tc -V
  tc utility, iproute2-ss040831
To build the PSPacer, use the proper version of iproute2.

When I install PSPacer, tc command fails with an error message "RTNETLINK answers: Invalid argument"

This error message implies that TCP Segmentation Offloading (TSO) is enabled. Because PSPacer is not supporting TSO, you have to disable TSO before you install it.

If you can see the following message by looking at the output of dmesg command, TSO is enabled. 

  psp: sch_psp does not support TSO.  You must disable it:
  "ethtool -K eth0 tso off"
If so, you have to disable TSO by using ethtool(8) command and retry tc qdisc add as follows: 
  # /sbin/ethtool -K eth0 tso off
  # /sbin/tc qdisc add dev eth0 ...

I can not compile PSPacer on the Linux kernel 2.4

The compilation of PSPacer 1.2 failed on the Linux kernel 2.4, and error messages are shown as follows: 
make[1]: Entering directory `/home/takano/works/pspacer-1.2/pspd'
cc -g -Wall -I /lib/modules/`uname -r`/build/include -I ../kernel -c -o conf.o conf.c
In file included from /lib/modules/2.4.22-gridmpi/build/include/linux/rtnetlink.h:4,
                 from conf.c:29:
/lib/modules/2.4.22-gridmpi/build/include/linux/netlink.h:25: parse error before "__u32"
/lib/modules/2.4.22-gridmpi/build/include/linux/netlink.h:25: warning: no semicolon at end of struct or union
	... snip ...
make[1]: *** [conf.o] Error 1
make[1]: Leaving directory `/home/takano/works/pspacer-1.2/pspd'
make: *** [all] Error 2
You need to fix pspd/conf.c as follows: 
--- conf.orig.c 2006-04-26 11:40:03.000000000 +0900
+++ conf.c      2006-04-26 11:41:41.000000000 +0900
@@ -26,6 +26,7 @@
 #include <string.h>
 #include <arpa/inet.h>
 #include <netinet/in.h>
+#include <linux/types.h>
 #include <linux/rtnetlink.h>
 #include <sys/socket.h>
PSPacer 2.x does not support the Linux kernel 2.4.


Return to the GridTCP Project Page.

($Date: 2012-03-19 02:11:31 $)