Thursday, February 26, 2015

Simple experiment to illustrate bufferbloat using two Linux machines

 Tools required: iptraf, ethtool, nttcp

1. Connect two machines with a direct ethernet cable. Label one machine A and the other B.
2. Force the ethernet speed to 10Mbps. This only needs to be done on one machine, the other will comply. Run this command on machine A:
        sudo ethtool -s eth0 speed 10 duplex full
3. Start nttcp on machine B
        nttcp -i
4. Start a ping from machine A to machine B and leave it running as you follow the rest of the steps. Notice the RTT value.
5. Start the nttcp session from machine A
        nttcp -t -D -n2048000 <machine A ip address>
6. Notice the RTT value increase. This is because buffer is building up at the tx queue for the interface on machine A. This is not the NIC ring buffer, it's the tx queue inside the kernel.
7. On machine A, open IPTraf and monitor the TCP flow speed of the connection.
8. Eliminate the tx queue and watch the RTT go down. Change it back and watch the RTT increase again. The flow speed doesn't change. The change in delay is just bufferbloat.
        sudo ifconfig eth0 txqueuelen 0
        sudo ifconfig eth0 txqueuelen 1000

--> Default txqueuelen value is 1000. It makes sense to eliminate txqueue or make it smaller on a host and network where contention is not likely and link speed is stable. However for fluctuating capacity links (such as wireless) a reasonably sized buffer is important to absorb the changes and achieve high utilization.

Check the following pages for more info. The steps above are extracted from the second post.


  1. A couple thoughts:

    1) You can use netperf-wrapper to make repeatable measurements of latency. There's an Ubuntu build on

    2) Once you've tested your current configuration, you can reset the txqueuelen values and install the fq_codel qdisc. This will dynamically control queue lengths so that each packet's "sojourn time" - the time from when it is queued until it is dequeued - does not get too long (typically set to 5 msec).

  2. I tried fq_codel before but I haven't seen netperf-wrapper. Thanks for the hint.