This page includes experimental results of the following paper:

H. Cai, D. Eun, S. Ha, I. Rhee and L. Xu, Stochastic Ordering for Internet Congestion Control and its Applications, To appear in INFOCOM 2007.

1. Testbed Configuration

  • TCP Sender1 send long-lived flows using iperf to TCP Receiver1
  • TCP Sender2 send long-lived flows using iperf to TCP Receiver2
  • Background Traffic Generators 1-4 send and receive long-lived and short-lived traffic in forward as well as backward direction
  • Bottleneck Bandwidth is set by the Dunmmynet Router 1 (FreeBSD 5.2.1)

  • Delay (RTT) is generated by Dummynet Router 2 (FreeBSD 5.2.1)
    • [1] Paul Barford and Mark Crovella, "Generating Representative Web Workloads for Network and Server Performance Evaluation", ACM SIGMETRICS 1998.
    [2] Joel Sommers, Paul Barford, and Hyungsuk Kim, "Harpoon: A Flow-Level Traffic Generator for Router and Network Tests", extended abstract, ACM SIGMETRICS 2004.
    [3] F. Hernandez-Campos, F.D. Smith, and K. Jeffay, "Generating Realistic TCP Workloads", in Proceeding of CMG 2004, December 2004.
    [4] Jay Aikat, Jasleen Kaur, F. Donelson Smith, and Kevin Jeffay, "Variability in TCP Roundtrip Times", ACM IMC 2003.



    We use a dumbbell topology of dummynet routers where each end points consists of a set of Dell Linux servers dedicated to high-speed TCP variant flows and background traffic. Background traffic is generated by using a modification of a web-traffic generator, called Surge[1] and Iperf. We modified the traffic generator to generate a wider range of flow sizes in order to increase variability in cross traffic because medium size flows tend to fully execute the slow start and increase the variability in available bandwidth. The RTT of each background flow is set based on an exponential distribution found in [4]. The maximum bandwidth of the bottleneck router is set to 400 Mbps. The same amount of background traffic is pushed into forward and backward directions of the dumbbell. We use the drop-tail router at the bottleneck.

    2. Background traffic

    Type 1: moderate mid-sized flows (CoV of available BW = 0.05)
    Type 2: extremely varying mid-sized flows (CoV of available BW = 0.15))



    We use three types of background traffic with different degrees of rate variations and congestion. The first two traffic types consume about 70Mbps when they run without any other flows, representing congestion-free network conditions. These two traffic types differ only in the amount of variations in available bandwidth (the first one is less varying). By varying the distribution of flow sizes, we vary the CoV of transmission rates which vary the available bandwidth usable by other competing flows. The figure in above shows the cumulative rate distribution of the two cross traffic types - one extremely varying and the other moderately so (the CoVs of available bandwidth are 0.15 and 0.05 respectively). The third traffic type emulates a congested network environment which is created by adding a few tens of long-lived TCP(sack) flows on top of the first traffic type. In most experiments we run, we use the first traffic type. We later use the more varying one to see the effect of increased traffic variability. With background traffic, we run multiple flows of a protocol being examined from two end points and measure the performance parameters at the bottleneck router.

    3. Is CoV a good metric?

    Rate fluctuations influence fluctuations in router queue sizes and frequent queue overflows. These overflows may cause loss synchronizations across many co-existing flows and severe under-utilization of link capacity. When the under-utilization due to loss synchronization occurs very often, we say that the network is ``unstable''. Therefore, the window adjustment policies of protocols have a great impact on network stability.

    A realistic experiment setting always contains multiple flows of a protocol being observed and a significant amount of cross traffic whose rates are varying over time. In this environment, we can consider two possible metrics for rate variance. One is to consider aggregate flow statistics -- the CoV of aggregate transmission rate samples of all the flows of that protocol (CoV-PTA) and the other is to consider per-flow statistics -- the CoV of per-flow transmission rate samples (CoV-PTP). Both are measured in the bottleneck router at a fixed interval of one second.

    We first measure the degree of network instability that these metrics represent. This correlation is important because if rate variance does not significantly affect network stability (i.e., if they are good metrics of network stability), then although we measure the relative ordering in the rate variance of protocols, our theoretical would not have much practical value.

    Our experiments are run in two different network environments: In the first one, high-speed protocol flows are dominant. We only run the cross traffic with CoV of available bandwidth 0.05 and different number of high-speed flows. In this experiment, cross traffic consumes about 10 to 15% of the total bandwidth. The second environment contains a large amount of long lived TCP-SACK flows on top of the same cross traffic as the first one. By only TCP traffic alone, the network has about 70-90% utilization. In this environment, we run only two to four flows of high-speed protocols. This environment represents a congested environment where high-speed flows are not dominating.

    Under moderate congestion



    Under heavy congestion



    The figures in above plot the correlation coefficients between rate fluctuations and network stability in the above two different environments. We measure network stability by three metrics: link utilization, CoV of queue variations, and CoV of link utilization variations. The correlation coefficients are obtained from data collected from over 1500 experimental runs, irrespective of protocols being tested. In these tests, we vary RTTs, buffer sizes, and the number of high speed flows. In both environments, we find that CoV-PTA has a stronger correlation to network stability than CoV-PTP. When the network traffic is dominated by high-speed flows, we find more correlations between CoV-PTA and network stability and also between CoV-PTP and network stability. When the network is congested, protocol rate fluctuations do not have many cases of low link utilization. This is the reason why CoV-PTP and CoV-PTA show lower correlations. But even in this case, we find that some high-speed protocols cause many loss synchronizations and thus low utilization. Although weaker than in the uncontested case, the correlation between CoV-PTA and network stability under congestion is a lot higher than CoV-PTP.

    We explain this phenomenon as follows. In a network environment with a significant amount of cross traffic, protocol flows are always adapting themselves to the time-varying available bandwidth. So there exist always some amount of inherent variations of per-flow transmission rates due to this adaptation. However, not all these adaptive variations of protocol rates cause loss synchronization and fluctuation in the link utilization. This behavior is highly dependent on the aggressiveness (or stability) of the protocol. As protocols are more aggressive, these variations are translated into loss in the utilization more often. Therefore, per-flow statistics do not have a way to filter out these benign fluctuations and thus may not faithfully represent the degree of inherent protocol stability. On the other hand, aggregate flow statistics have this filtering effect and tend to better capture the rate fluctuations affecting the global network stability. The high correlation between CoV-PTA and CoV of link utilization is the strong evidence.

    We claim that CoV-PTA is also a more faithful representation of protocol stability or protocol rate variance that is captured by our theoretical analysis. Our theory assumes that a protocol flow does not affect the loss process. In reality, there is always some amount of self-induced losses by a protocol flow because high-speed flows tend to have large window sizes. Per-flow statistics, therefore, factor in this effect. On the other hand, aggregate flow rate statistics tend to ameliorate it, and represents the protocol window variance independent of its flows the theoretical results more faithfully than per-flow statistics. We use CoV-PTA throughout this paper as the main metric of window size fluctuations of protocols under ideal conditions.

    4. Cumulative distribution of total bits seen in the bottleneck link

      Bottleneck Bandwidth is fixed to 400Mbps
      High-speed flows: 4 flows (forward direction), RTTs are set to 320ms
      Long-Lived backgrond flows: forward direction (iperf 0flows), backward direction (iperf 2flows)[4](RTT is exponentially distributed)
      Mid-sized background flows: Background traffic 1 (forward direction, backward direction) [1][4] (RTT is exponentially distributed)
      Queue size: Fixed to 1Mbytes (6.5% BDP)
      Duration: 600sec, Sampled during 10sec-600sec(The highspeed flows start 0 and 10sec, respectively)

  • Link Utilization

    • RTT CUBICBICHTCPHSTCPSTCPFASTSACKWESTWOOD
    320ms 0.871549[3] [4] [5] [6] [7] [8] [9] [11] [12] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] 0.884664[1] [2] [3] [4] [5] [6] [7] [8] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [25] 0.797348[2] [3] [4] [5] [6] [] [8] [9] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [23] [24] [25] 0.794895[1] [2] [3] [4] [5] [6] [7] [10] [12] [13] [14] [15] [16] [17] [19] [20] [21] [22] [23] [24] 0.857578[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] 0.676606[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] 0.436469[2] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [23] [24] [25] 0.417863[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25]


    To illustrate the degree of rate fluctuations, the figure in above shows the cumulative distribution of the total number of bits observed in the bottleneck link at each second for each protocol during an experimental run with 320ms RTT. The total bits include the bits from all sources including the background traffic. HTCP has the widest distribution indicating high fluctuation in utilization over the course of the experiment and BIC shows the steepest distribution implying very stable performance.

    5. Impact of RTTs

      Bottleneck Bandwidth is fixed to 400Mbps
      High-speed flows: 4 flows (forward direction), RTTs are varied at each experiment 40ms, 80ms, 160ms, and 320ms
      Long-Lived backgrond flows: forward direction (iperf 0flows), backward direction (iperf 2flows)[4](RTT is exponentially distributed)
      Mid-sized background flows: Background traffic 1 (forward direction, backward direction) [1][4] (RTT is exponentially distributed)
      Queue size: Fixed to 1Mbytes (6.5% BDP)
      Duration: 600sec, Sampled during 10sec-600sec(The highspeed flows start 0 and 10sec, respectively)




  • Link Utilization
    • Delay CUBICBICHTCPHSTCPSTCPFASTSACKWESTWOOD
    40ms 0.947994[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] 0.955576[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.900703[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.945350[3] [4] [6] [7] [8] [9] [10] [11] [12] 0.948443[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.887704[1] [3] [4] [9] [11] 0.894284[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.946308[1] [3] [4] [5] [6] [7] [8] [9] [10] [11]
    80ms 0.930742[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.952801[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.867730[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.924289[3] [4] [5] [6] [7] [8] [9] [10] [12] 0.936101[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.916822[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.822665[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.936654[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]
    160ms 0.901630[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [11] 0.930647[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.852555[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.889809[1] [3] [4] [6] [7] [9] [10] [11] [12] 0.916791[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.729630[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.674717[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] 0.810367[1] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]
    320ms 0.871549[3] [4] [5] [6] [7] [8] [9] [11] [12] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] 0.884664[1] [2] [3] [4] [5] [6] [7] [8] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [25] 0.797348[2] [3] [4] [5] [6] [] [8] [9] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [23] [24] [25] 0.794895[1] [2] [3] [4] [5] [6] [7] [10] [12] [13] [14] [15] [16] [17] [19] [20] [21] [22] [23] [24] 0.857578[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] 0.676606[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] 0.436469[2] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [23] [24] [25] 0.417863[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25]


    6. Impact of buffer sizes

      Bottleneck Bandwidth is fixed to 400Mbps
      High-speed flows: 4 flows (forward direction), RTTs are fixed to 320ms
      Long-Lived backgrond flows: forward direction (iperf 0flows), backward direction (iperf 2flows)[4](RTT is exponentially distributed)
      Mid-sized background flows: Background traffic 1 (forward direction, backward direction) [1][4] (RTT is exponentially distributed)
      Queue size is varied at each experiment 1MB (6.5% BDP), 2MB, 4MB, 8MB (50% BDP)
      Duration: 600sec, Sampled during 10sec-600sec(The highspeed flows start 0 and 10sec, respectively)



  • Link Utilization
    • Buffer Size CUBICBICHTCPHSTCPSTCPFASTSACKWESTWOOD
    1M 0.871549[3] [4] [5] [6] [7] [8] [9] [11] [12] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] 0.884664[1] [2] [3] [4] [5] [6] [7] [8] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [25] 0.797348[2] [3] [4] [5] [6] [7] [8] [9] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] 0.794895[1] [2] [3] [4] [5] [6] [7] [10] [12] [13] [14] [15] [16] [17] [19] [20] [21] [22] [23] [24] 0.857578[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]