draft-ietf-rmcat-eval-test-10.txt   rfc8867.txt 
Network Working Group Z. Sarker Internet Engineering Task Force (IETF) Z. Sarker
Internet-Draft Ericsson AB Request for Comments: 8867 Ericsson AB
Intended status: Informational V. Singh Category: Informational V. Singh
Expires: November 24, 2019 callstats.io ISSN: 2070-1721 callstats.io
X. Zhu X. Zhu
M. Ramalho
Cisco Systems Cisco Systems
May 23, 2019 M. Ramalho
AcousticComms
January 2021
Test Cases for Evaluating RMCAT Proposals Test Cases for Evaluating Congestion Control for Interactive Real-Time
draft-ietf-rmcat-eval-test-10 Media
Abstract Abstract
The Real-time Transport Protocol (RTP) is used to transmit media in The Real-time Transport Protocol (RTP) is used to transmit media in
multimedia telephony applications. These applications are typically multimedia telephony applications. These applications are typically
required to implement congestion control. This document describes required to implement congestion control. This document describes
the test cases to be used in the performance evaluation of such the test cases to be used in the performance evaluation of such
congestion control algorithms in a controlled environment. congestion control algorithms in a controlled environment.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This document is not an Internet Standards Track specification; it is
provisions of BCP 78 and BCP 79. published for informational purposes.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Engineering Task Force
and may be updated, replaced, or obsoleted by other documents at any (IETF). It represents the consensus of the IETF community. It has
time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are candidates for any level of Internet
Standard; see Section 2 of RFC 7841.
This Internet-Draft will expire on November 24, 2019. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc8867.
Copyright Notice Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the Copyright (c) 2021 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology
3. Structure of Test cases . . . . . . . . . . . . . . . . . . . 3 3. Structure of Test Cases
4. Recommended Evaluation Settings . . . . . . . . . . . . . . . 8 4. Recommended Evaluation Settings
4.1. Evaluation metrics . . . . . . . . . . . . . . . . . . . 8 4.1. Evaluation Metrics
4.2. Path characteristics . . . . . . . . . . . . . . . . . . 8 4.2. Path Characteristics
4.3. Media source . . . . . . . . . . . . . . . . . . . . . . 9 4.3. Media Source
5. Basic Test Cases . . . . . . . . . . . . . . . . . . . . . . 10 5. Basic Test Cases
5.1. Variable Available Capacity with a Single Flow . . . . . 10 5.1. Variable Available Capacity with a Single Flow
5.2. Variable Available Capacity with Multiple Flows . . . . . 13 5.2. Variable Available Capacity with Multiple Flows
5.3. Congested Feedback Link with Bi-directional Media Flows . 14 5.3. Congested Feedback Link with Bi-directional Media Flows
5.4. Competing Media Flows with same Congestion Control 5.4. Competing Media Flows with the Same Congestion Control
Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 17 Algorithm
5.5. Round Trip Time Fairness . . . . . . . . . . . . . . . . 19 5.5. Round Trip Time Fairness
5.6. Media Flow Competing with a Long TCP Flow . . . . . . . . 21 5.6. Media Flow Competing with a Long TCP Flow
5.7. Media Flow Competing with Short TCP Flows . . . . . . . . 23 5.7. Media Flow Competing with Short TCP Flows
5.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 25 5.8. Media Pause and Resume
6. Other potential test cases . . . . . . . . . . . . . . . . . 27 6. Other Potential Test Cases
6.1. Media Flows with Priority . . . . . . . . . . . . . . . . 27 6.1. Media Flows with Priority
6.2. Explicit Congestion Notification Usage . . . . . . . . . 27 6.2. Explicit Congestion Notification Usage
6.3. Multiple Bottlenecks . . . . . . . . . . . . . . . . . . 28 6.3. Multiple Bottlenecks
7. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 30 7. Wireless Access Links
8. Security Considerations . . . . . . . . . . . . . . . . . . . 30 8. Security Considerations
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30 9. IANA Considerations
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30 10. References
11. References . . . . . . . . . . . . . . . . . . . . . . . . . 30 10.1. Normative References
11.1. Normative References . . . . . . . . . . . . . . . . . . 30 10.2. Informative References
11.2. Informative References . . . . . . . . . . . . . . . . . 32 Acknowledgments
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32 Authors' Addresses
1. Introduction 1. Introduction
This memo describes a set of test cases for evaluating congestion This memo describes a set of test cases for evaluating congestion
control algorithm proposals in controlled environments for real-time control algorithm proposals in controlled environments for real-time
interactive media. It is based on the guidelines enumerated in interactive media. It is based on the guidelines enumerated in
[I-D.ietf-rmcat-eval-criteria] and the requirements discussed in [RFC8868] and the requirements discussed in [RFC8836]. The test
[I-D.ietf-rmcat-cc-requirements]. The test cases cover basic usage cases cover basic usage scenarios and are described using a common
scenarios and are described using a common structure, which allows structure, which allows for additional test cases to be added to
for additional test cases to be added to those described herein to those described herein to accommodate other topologies and/or the
accommodate other topologies and/or the modelling of different path modeling of different path characteristics. The described test cases
characteristics. The described test cases in this memo should be in this memo should be used to evaluate any proposed congestion
used to evaluate any proposed congestion control algorithm for real- control algorithm for real-time interactive media.
time interactive media.
2. Terminology 2. Terminology
The terminology defined in RTP [RFC3550], RTP Profile for Audio and The terminology defined in RTP [RFC3550], RTP Profile for Audio and
Video Conferences with Minimal Control [RFC3551], RTCP Extended Video Conferences with Minimal Control [RFC3551], RTCP Extended
Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback
(RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP [RFC5506] (RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP [RFC5506]
apply. applies.
3. Structure of Test cases 3. Structure of Test Cases
All the test cases in this document follow a basic structure allowing All the test cases in this document follow a basic structure allowing
implementers to describe a new test scenario without repeatedly implementers to describe a new test scenario without repeatedly
explaining common attributes. The structure includes a general explaining common attributes. The structure includes a general
description section that describes the test case and its motivation. description section that describes the test case and its motivation.
Additionally the test case defines a set of attributes that Additionally the test case defines a set of attributes that
characterize the testbed, for example, the network path between characterize the testbed, for example, the network path between
communicating peers and the diverse traffic sources. communicating peers and the diverse traffic sources.
o Define the test case: Define the test case:
* General description: describes the motivation and the goals of General description: describes the motivation and the goals of
the test case. the test case.
* Expected behavior: describes the desired rate adaptation Expected behavior: describes the desired rate adaptation
behavior. behavior.
* Define a list of metrics to evaluate the desired behavior: this List of metrics to evaluate the desired behavior: this indicates
indicates the minimum set of metrics (e.g., link utilization, the minimum set of metrics (e.g., link utilization, media
media sending rate) that a proposed algorithm needs to measure sending rate) that a proposed algorithm needs to measure to
to validate the expected rate adaptation behavior. It should validate the expected rate adaptation behavior. It should also
also indicate the time granularity (e.g., averaged over 10ms, indicate the time granularity (e.g., averaged over 10 ms, 100
100ms, or 1s) for measuring certain metrics. Typical ms, or 1 s) for measuring certain metrics. Typical measurement
measurement interval is 200ms. interval is 200 ms.
o Define testbed topology: every test case needs to define an Define testbed topology:
evaluation testbed topology. Figure 1 shows such an evaluation
topology. In this evaluation topology, S1..Sn are traffic Every test case needs to define an evaluation testbed topology.
sources. These sources generate media traffic and use the Figure 1 shows such an evaluation topology. In this evaluation
congestion control algorithm(s) under investigation. R1..Rn are topology, S1..Sn are traffic sources. These sources generate
the corresponding receivers. A test case can have one or more media traffic and use the congestion control algorithm(s) under
such traffic sources (S) and their corresponding receivers (R). investigation. R1..Rn are the corresponding receivers. A test
The path from the source to destination is denoted as "forward" case can have one or more such traffic sources (S) and their
and the path from a destination to a source is denoted as corresponding receivers (R). The path from the source to
"backward". The following basic structure of the test case has destination is denoted as "forward", and the path from a
been described from the perspective of media generating endpoints destination to a source is denoted as "backward". The following
attached on the left-hand side of Figure 1. In this setup, the basic structure of the test case has been described from the
media flows are transported in forward direction and corresponding perspective of media-generating endpoints attached on the left-
hand side of Figure 1. In this setup, the media flows are
transported in the forward direction, and the corresponding
feedback/control messages are transported in the backward feedback/control messages are transported in the backward
direction. However, it is also possible to set up the test with direction. However, it is also possible to set up the test with
media in both forward and backward directions. In that case, media in both forward and backward directions. In that case,
unless otherwise specified by the test case, it is expected that unless otherwise specified by the test case, it is expected that
the backward path does not introduce any congestion related the backward path does not introduce any congestion-related
impairments and has enough capacity to accommodate both media and impairments and has enough capacity to accommodate both media and
feedback/control messages. It should be noted that depending on feedback/control messages. It should be noted that, depending on
the test cases it is possible to have different path the test cases, it is possible to have different path
characteristics in either of the directions. characteristics in either of the directions.
+---+ +---+ +---+ +---+
|S1 |====== \ Forward --> / =======|R1 | |S1 |====== \ Forward --> / =======|R1 |
+---+ \\ // +---+ +---+ \\ // +---+
\\ // \\ //
+---+ +-----+ +-----+ +---+ +---+ +-----+ +-----+ +---+
|S2 |=======| A |------------------------------>| B |=======|R2 | |S2 |=======| A |--------------------------->| B |=======|R2 |
+---+ | |<------------------------------| | +---+ +---+ | |<---------------------------| | +---+
+-----+ +-----+ +-----+ +-----+
(...) // \\ (...) (...) // \\ (...)
// <-- Backward \\ // <-- Backward \\
+---+ // \\ +---+ +---+ // \\ +---+
|Sn |====== / \ ======|Rn | |Sn |====== / \ ======|Rn |
+---+ +---+ +---+ +---+
Figure 1: Example of A Testbed Topology Figure 1: Example of a Testbed Topology
In a testbed environment with real equipments, there may exist a In a testbed environment with real equipment, there may exist a
significant amount of unwanted traffic on the portions of the significant amount of unwanted traffic on the portions of the
network path between the endpoints. Some of this traffic may be network path between the endpoints. Some of this traffic may be
generated by other processes on the endpoints themselves (e.g., generated by other processes on the endpoints themselves (e.g.,
discovery protocols) or by other endpoints not presently under discovery protocols) or by other endpoints not presently under
test. Such unwanted traffic should be removed or avoided to the test. Such unwanted traffic should be removed or avoided to the
greatest extent possible. greatest extent possible.
o Define testbed attributes: Define testbed attributes:
* Duration: defines the duration of the test in seconds. Duration: defines the duration of the test in seconds.
* Path characteristics: defines the end-to-end transport level Path characteristics: defines the end-to-end transport level path
path characteristics of the testbed for a particular test case. characteristics of the testbed for a particular test case. Two
Two sets of attributes describe the path characteristics, one sets of attributes describe the path characteristics, one for
for the forward path and the other for the backward path. The the forward path and the other for the backward path. The path
path characteristics for a particular path direction is characteristics for a particular path direction are applicable
applicable to all the Sources "S" sending traffic on that path. to all the sources "S" sending traffic on that path. If only
If only one attribute is specified, it is used for both path one attribute is specified, it is used for both path
directions, however, unless specified the reverse path has no directions; however, unless specified the reverse path has no
capacity restrictions and no path loss. capacity restrictions and no path loss.
+ Path direction: forward or backward. Path direction: forward or backward.
+ Minimum bottleneck-link capacity: defines minimum capacity Minimum bottleneck-link capacity: defines the minimum capacity
of the end-to-end path of the end-to-end path.
+ Reference bottleneck capacity: defines a reference value for Reference bottleneck capacity: defines a reference value for
the bottleneck capacity for test cases with time-varying the bottleneck capacity for test cases with time-varying
bottleneck capacities. All bottleneck capacities will be bottleneck capacities. All bottleneck capacities will be
specified as a ratio with respect to the reference capacity specified as a ratio with respect to the reference capacity
value. value.
+ One-way propagation delay: describes the end-to-end latency One-way propagation delay: describes the end-to-end latency
along the path when network queues are empty, i.e., the time along the path when network queues are empty, i.e., the time
it takes for a packet to go from the sender to the receiver it takes for a packet to go from the sender to the receiver
without encountering any queuing delay. without encountering any queuing delay.
+ Maximum end-to-end jitter: defines the maximum jitter that Maximum end-to-end jitter: defines the maximum jitter that can
can be observed along the path. be observed along the path.
+ Bottleneck queue type: for example, "tail drop" [RFC7567], Bottleneck queue type: for example, "tail drop" [RFC7567],
Flow Queue -CoDel (FQ-CoDel)[RFC8290], or Proportional Flow Queue Controlled Delay (FQ-CoDel) [RFC8290], or
Integral controller Enhanced (PIE)[RFC8033]. Proportional Integral controller Enhanced (PIE) [RFC8033].
+ Bottleneck queue size: defines the size of queue in terms of Bottleneck queue size: defines the size of queue in terms of
queuing time when the queue is full (in milliseconds). queuing time when the queue is full (in milliseconds).
+ Path loss ratio: characterizes the non-congested, additive, Path loss ratio: characterizes the non-congested, additive
losses to be generated on the end-to-end path. This must losses to be generated on the end-to-end path. This must
describe the loss pattern or loss model used to generate the describe the loss pattern or loss model used to generate the
losses. losses.
* Application-related: defines the traffic source behavior for Application-related: defines the traffic source behavior for
implementing the test case implementing the test case:
+ Media traffic Source: defines the characteristics of the Media traffic source: defines the characteristics of the media
media sources. When using more than one media source, the sources. When using more than one media source, the
different attributes are enumerated separately for each different attributes are enumerated separately for each
different media source. different media source.
- Media type: Video/Voice Media type: Video/Voice.
- Media flow direction: forward, backward or both. Media flow direction: forward, backward, or both.
- Number of media sources: defines the total number of Number of media sources: defines the total number of media
media sources sources.
- Media codec: Constant Bit Rate (CBR) or Variable Bit Rate Media codec: Constant Bit Rate (CBR) or Variable Bit Rate
(VBR) (VBR).
- Media source behavior: describes the media encoder Media source behavior: describes the media encoder
behavior. It defines the main parameters that affect the behavior. It defines the main parameters that affect the
adaptation behavior. This may include but is not limited adaptation behavior. This may include but is not limited
to: to the following:
o Adaptability: describes the adaptation options. For Adaptability: describes the adaptation options. For
example, in the case of video it defines the following example, in the case of video, it defines the
ranges of adaptation: bit rate, frame rate, video following ranges of adaptation: bit rate, frame rate,
resolution. Similarly, in the case of voice, it and video resolution. Similarly, in the case of
defines the range of bit rate adaptation, the sampling voice, it defines the range of bit rate adaptation,
rate variation, and the variation in packetization the sampling rate variation, and the variation in
interval. packetization interval.
o Output variation : for a VBR encoder it defines the Output variation: for a VBR encoder, it defines the
encoder output variation from the average target rate encoder output variation from the average target rate
over a particular measurement interval. For example, over a particular measurement interval. For example,
on average the encoder output may vary between 5% to on average the encoder output may vary between 5% to
15% above or below the average target bit rate when 15% above or below the average target bit rate when
measured over a 100 ms time window. The time interval measured over a 100 ms time window. The time interval
over which the variation is specified must be over which the variation is specified must be
provided. provided.
o Responsiveness to a new bit rate request: the lag in Responsiveness to a new bit rate request: the lag in
time between a new bit rate request from the time between a new bit rate request from the
congestion control algorithm and actual rate changes congestion control algorithm and actual rate changes
in encoder output. Depending on the encoder, this in encoder output. Depending on the encoder, this
value may be specified in absolute time (e.g. 10ms to value may be specified in absolute time (e.g., 10 ms
1000ms) or other appropriate metric (e.g. next frame to 1000 ms) or other appropriate metric (e.g., next
interval time). frame interval time).
More detailed discussions on expected media source More detailed discussions on expected media source
behavior, including those from synthetic video traffic behavior, including those from synthetic video traffic
sources, is at [I-D.ietf-rmcat-video-traffic-model]. sources, can be found in [RFC8593].
- Media content: describes the chosen video scenario. For Media content: describes the chosen video scenario. For
example, video test sequences are available at: example, video test sequences are available at [xiph-seq]
[xiph-seq] and [HEVC-seq]. Different video scenarios and [HEVC-seq]. Different video scenarios give different
give different distribution of video frames produced by distributions of video frames produced by the video
the video encoder. Hence, it is important to specify the encoder. Hence, it is important to specify the media
media content used in a particular test. If a synthetic content used in a particular test. If a synthetic video
video traffic souce [I-D.ietf-rmcat-video-traffic-model] traffic source [RFC8593] is used, then the synthetic
is used, then the synthetic video traffic source needs to video traffic source needs to be configured according to
configure according to the characteristics of the media the characteristics of the media content specified.
content specified.
- Media timeline: describes the point when the media source Media timeline: describes the point when the media source
is introduced and removed from the testbed. For example, is introduced and removed from the testbed. For example,
the media source may start transmitting immediately when the media source may start transmitting immediately when
the test case begins, or after a few seconds. the test case begins, or after a few seconds.
- Startup behavior: the media starts at a defined bit rate, Startup behavior: the media starts at a defined bit rate,
which may be the minimum, maximum bit rate, or a value in which may be the minimum, maximum bit rate, or a value in
between (in Kbps). between (in Kbps).
+ Competing traffic source: describes the characteristics of Competing traffic source: describes the characteristics of the
the competing traffic source, the different types of competing traffic source, the different types of competing
competing flows are enumerated in flows are enumerated in [RFC8868].
[I-D.ietf-rmcat-eval-criteria].
- Traffic direction: forward, backward or both. Traffic direction: forward, backward, or both.
- Type of sources: defines the types of competing traffic Type of sources: defines the types of competing traffic
sources. Types of competing traffic flows are listed in sources. Types of competing traffic flows are listed in
[I-D.ietf-rmcat-eval-criteria]. For example, the number [RFC8868]. For example, the number of TCP flows
of TCP flows connected to a web browser, the mean size connected to a web browser, the mean size and
and distribution of the content downloaded. distribution of the content downloaded.
- Number of sources: defines the total number of competing Number of sources: defines the total number of competing
sources of each media type per traffic direction. sources of each media type per traffic direction.
- Congestion control: enumerates the congestion control Congestion control: enumerates the congestion control used
used by each type of competing traffic. by each type of competing traffic.
- Traffic timeline: describes when the competing traffic Traffic timeline: describes when the competing traffic
starts and ends in the test case. starts and ends in the test case.
* Additional attributes: describes attributes essential for Additional attributes: describes attributes essential for
implementing a test case which are not included in the above implementing a test case that are not included in the above
structure. These attributes must be well defined, so that the structure. These attributes must be well defined, so that the
other implementers of that particular test case are able to other implementers of that particular test case are able to
implement it easily. implement it easily.
Any attribute can have a set of values (enclosed within "[]"). Each Any attribute can have a set of values (enclosed within "[]"). Each
member value of such a set must be treated as different value for the member value of such a set must be treated as different value for the
same attribute. It is desired to run separate tests for each such same attribute. It is desired to run separate tests for each such
attribute value. attribute value.
The test cases described in this document follow the above structure. The test cases described in this document follow the above structure.
4. Recommended Evaluation Settings 4. Recommended Evaluation Settings
This section describes recommended test case settings and could be This section describes recommended test case settings and could be
overwritten by the respective test cases. overwritten by the respective test cases.
4.1. Evaluation metrics 4.1. Evaluation Metrics
To evaluate the performance of the candidate algorithms the To evaluate the performance of the candidate algorithms, the
implementers must log enough information to visualize the following implementers must log enough information to visualize the following
metrics at a fine enough time granularity: metrics at a fine enough time granularity:
1. Flow level: 1. Flow level:
A. End-to-end delay for the congestion controlled media flow(s). A. End-to-end delay for the congestion-controlled media flow(s).
For example - end-to-end delay observed on IP packet level, For example, end-to-end delay observed on the IP packet level
video frame level. and the video frame level.
B. Variation in sending bit rate and throughput. Mainly B. Variation in sending bit rate and throughput. Mainly
observing the frequency and magnitude of oscillations. observing the frequency and magnitude of oscillations.
C. Packet losses observed at the receiving endpoint. C. Packet losses observed at the receiving endpoint.
D. Feedback message overhead. D. Feedback message overhead.
E. Convergence time - time to reach steady state for the E. Convergence time. Time to reach steady state for the
congestion controlled media flow(s). Each occurrence of congestion-controlled media flow(s). Each occurrence of
convergence during the test period need to be presented. convergence during the test period needs to be presented.
2. Transport level: 2. Transport level:
A. Bandwidth utilization. A. Bandwidth utilization.
B. Queue length (milliseconds at specified path capacity). B. Queue length (milliseconds at specified path capacity).
4.2. Path characteristics 4.2. Path Characteristics
Each path between a sender and receiver as described in Figure 1 have Each path between a sender and receiver as described in Figure 1 has
the following characteristics unless otherwise specified in the test the following characteristics unless otherwise specified in the test
case. case:
o Path direction: forward and backward. Path direction: forward and backward.
o Reference bottleneck capacity: 1Mbps. Reference bottleneck capacity: 1 Mbps.
o One-Way propagation delay: 50ms. Implementers are encouraged to One-way propagation delay: 50 ms. Implementers are encouraged to
run the experiment with additional propagation delays mentioned in run the experiment with additional propagation delays mentioned in
[I-D.ietf-rmcat-eval-criteria] [RFC8868].
o Maximum end-to-end jitter: 30ms. Jitter models are described in Maximum end-to-end jitter: 30 ms. Jitter models are described in
[I-D.ietf-rmcat-eval-criteria] [RFC8868].
o Bottleneck queue type: "tail drop". Implementers are encouraged Bottleneck queue type: "tail drop". Implementers are encouraged to
to run the experiment with other AQM schemes, such as FQ-CoDel and run the experiment with other Active Queue Management (AQM)
PIE. schemes, such as FQ-CoDel and PIE.
o Bottleneck queue size: 300ms. Bottleneck queue size: 300 ms.
o Path loss ratio: 0%. Path loss ratio: 0%.
Examples of additional network parameters are discussed in Examples of additional network parameters are discussed in [RFC8868].
[I-D.ietf-rmcat-eval-criteria].
For test cases involving time-varying bottleneck capacity, all For test cases involving time-varying bottleneck capacity, all
capacity values are specified as a ratio with respect to a reference capacity values are specified as a ratio with respect to a reference
capacity value, so as to allow flexible scaling of capacity values capacity value, so as to allow flexible scaling of capacity values
along with media source rate range. There exist two different along with media source rate range. There exist two different
mechanisms for inducing path capacity variation: a) by explicitly mechanisms for inducing path capacity variation: a) by explicitly
modifying the value of physical link capacity; or b) by introducing modifying the value of physical link capacity, or b) by introducing
background non-adaptive UDP traffic with time-varying traffic rate. background non-adaptive UDP traffic with time-varying traffic rate.
Implementers are encouraged to run the experiments with both Implementers are encouraged to run the experiments with both
mechanisms for test cases specified in Section 5.1, Section 5.2, and mechanisms for test cases specified in Section 5.1, Section 5.2, and
Section 5.3. Section 5.3.
4.3. Media source 4.3. Media Source
Unless otherwise specified, each test case will include one or more Unless otherwise specified, each test case will include one or more
media sources as described below. media sources as described below:
o Media type: Video Media type: Video
* Media codec: VBR Media codec: VBR
* Media source behavior: Media source behavior:
+ Adaptability: Adaptability:
- Bit rate range: 150 Kbps - 1.5 Mbps. In real-life Bit rate range: 150 Kbps - 1.5 Mbps. In real-life
applications the bit rate range can vary a lot depending applications, the bit rate range can vary a lot depending
on the provided service, for example, the maximum bit on the provided service; for example, the maximum bit
rate can be up to 4Mbps. However, for running tests to rate can be up to 4 Mbps. However, for running tests to
evaluate the congestion control algorithms it is more evaluate the congestion control algorithms, it is more
important to have a look at how they are reacting to important to have a look at how they react to a certain
certain amount of bandwidth change. Also it is possible amount of bandwidth change. Also it is possible that the
that the media traffic generator used in a particular media traffic generator used in a particular simulator or
simulator or testbed is not capable of generating higher testbed is not capable of generating a higher bit rate.
bit rate. Hence we have selected a suitable bit rate Hence, we have selected a suitable bit rate range typical
range typical of consumer-grade video conferencing of consumer-grade video conferencing applications in
applications in designing the test case. If a different designing the test case. If a different bit rate range
bit rate range is used in the test cases, then the end- is used in the test cases, then the end-to-end path
to-end path capacity values will also need to be scaled capacity values will also need to be scaled accordingly.
accordingly.
- Frame resolution: 144p - 720p (or 1080p). This Frame resolution: 144p - 720p (or 1080p). This resolution
resolution range is selected based on the bit rate range. range is selected based on the bit rate range. If a
If a different bit rate range is used in the test cases different bit rate range is used in the test cases, then
then the frame resolution range also need to be selected a suitable frame resolution range also needs to be
suitably. selected.
- Frame rate: 10fps - 30fps. This frame rate range is Frame rate: 10 fps - 30 fps. This frame rate range is
selected based on the bit rate range. If a different bit selected based on the bit rate range. If a different bit
rate range is used in the test cases then the frame rate rate range is used in the test cases, then the frame rate
range also need to be adjusted suitably. range also needs to be suitably adjusted.
+ Variation from target bit rate: +/-5%. Unless otherwise Variation from target bit rate: +/-5%. Unless otherwise
specified in the test case(s), bit rate variation should be specified in the test case(s), bit rate variation should be
calculated over one (1) second period of time. calculated over a one (1) second period of time.
+ Responsiveness to new bit rate request: 100ms Responsiveness to new bit rate request: 100 ms
* Media content: The media content should represent a typical Media content: The media content should represent a typical video
video conversational scenario with head and shoulder movement. conversational scenario with head and shoulder movement. We
We recommend to use Foreman video sequence[xiph-seq]. recommend using the Foreman video sequence [xiph-seq].
* Media startup behavior: 150Kbps. It should be noted that Media startup behavior: 150 Kbps. It should be noted that
applications can use smart ways to select an optimal startup applications can use smart ways to select an optimal startup
bit rate value for a certain network condition. In such cases bit rate value for a certain network condition. In such cases,
the candidate proposals may show the effectiveness of such the candidate proposals may show the effectiveness of such a
smart approach as an additional information for the evaluation smart approach as additional information for the evaluation
process. process.
o Media type: Audio Media type: Audio
* Media codec: CBR Media codec: CBR
* Media bit rate: 20Kbps Media bit rate: 20 Kbps
5. Basic Test Cases 5. Basic Test Cases
5.1. Variable Available Capacity with a Single Flow 5.1. Variable Available Capacity with a Single Flow
In this test case the minimum bottleneck-link capacity between the In this test case, the minimum bottleneck-link capacity between the
two endpoints varies over time. This test is designed to measure the two endpoints varies over time. This test is designed to measure the
responsiveness of the candidate algorithm. This test tries to responsiveness of the candidate algorithm. This test tries to
address the requirements in [I-D.ietf-rmcat-cc-requirements], which address the requirements in [RFC8836], which requires the algorithm
requires the algorithm to adapt the flow(s) and provide lower end-to- to adapt the flow(s) and provide lower end-to-end latency when there
end latency when there exists: exists:
o an intermediate bottleneck * an intermediate bottleneck
o change in available capacity (e.g., due to interface change, * change in available capacity (e.g., due to interface change,
routing change, abrupt arrival/departure of background non- routing change, abrupt arrival/departure of background non-
adaptive traffic). adaptive traffic)
o maximum media bit rate is greater than link capacity. In this * maximum media bit rate is greater than link capacity. In this
case, when the application tries to ramp up to its maximum bit case, when the application tries to ramp up to its maximum bit
rate, since the link capacity is limited to a value lower, the rate, since the link capacity is limited to a lower value, the
congestion control scheme is expected to stabilize the sending bit congestion control scheme is expected to stabilize the sending bit
rate close to the available bottleneck capacity. rate close to the available bottleneck capacity.
It should be noted that the exact variation in available capacity due It should be noted that the exact variation in available capacity due
to any of the above depends on the underlying technologies. Hence, to any of the above depends on the underlying technologies. Hence,
we describe a set of known factors, which may be extended to devise a we describe a set of known factors, which may be extended to devise a
more specific test case targeting certain behaviors in a certain more specific test case targeting certain behaviors in a certain
network environment. network environment.
Expected behavior: the candidate algorithm is expected to detect the Expected behavior: The candidate algorithm is expected to detect the
path capacity constraint, converge to the bottleneck link's capacity path capacity constraint, converge to the bottleneck link's
and adapt the flow to avoid unwanted media rate oscillation when the capacity, and adapt the flow to avoid unwanted media rate
sending bit rate is approaching the bottleneck link's capacity. Such oscillation when the sending bit rate is approaching the
oscillations might occur when the media flow(s) attempts to reach its bottleneck link's capacity. Such oscillations might occur when
maximum bit rate but overshoots the usage of the available bottleneck the media flow(s) attempts to reach its maximum bit rate but
capacity then to rectify, it reduces the bit rate and starts to ramp overshoots the usage of the available bottleneck capacity, then to
up again. rectify, it reduces the bit rate and starts to ramp up again.
Evaluation metrics : as described in Section 4.1. Evaluation metrics: As described in Section 4.1.
Testbed topology: One media source S1 is connected to the Testbed topology: One media source S1 is connected to the
corresponding R1. The media traffic is transported over the forward corresponding R1. The media traffic is transported over the
path and corresponding feedback/control traffic is transported over forward path and corresponding feedback/control traffic is
the backward path. transported over the backward path.
Forward --> Forward -->
+---+ +-----+ +-----+ +---+ +---+ +-----+ +-----+ +---+
|S1 |=======| A |------------------------------>| B |=======|R1 | |S1 |=======| A |------------------------------>| B |=======|R1 |
+---+ | |<------------------------------| | +---+ +---+ | |<------------------------------| | +---+
+-----+ +-----+ +-----+ +-----+
<-- Backward <-- Backward
Figure 2: Testbed Topology for Limited Link Capacity Figure 2: Testbed Topology for Limited Link Capacity
Testbed attributes: Testbed attributes:
o Test duration: 100s Test duration: 100 s
o Path characteristics: as described in Section 4.2 Path characteristics: as described in Section 4.2
o Application-related: Application-related:
* Media Traffic: Media Traffic:
+ Media type: Video Media type: Video
- Media direction: forward. Media direction: forward
- Number of media sources: one (1) Number of media sources: one (1)
- Media timeline: Media timeline:
o Start time: 0s. Start time: 0 s
o End time: 99s. End time: 99 s
+ Media type: Audio Media type: Audio
- Media direction: forward. Media direction: forward
- Number of media sources: one (1) Number of media sources: one (1)
- Media timeline: Media timeline:
o Start time: 0s. Start time: 0 s
o End time: 99s. End time: 99 s
* Competing traffic: Competing traffic:
+ Number of sources : zero (0) Number of sources: zero (0)
o Test Specific Information: Test-specific information:
* One-way propagation delay: [ 50 ms, 100 ms]. on the forward One-way propagation delay: [50 ms, 100 ms]. On the forward path
path direction direction.
* This test uses bottleneck path capacity variation as listed in This test uses bottleneck path capacity variation as listed in
Table 1 Table 1.
* When using background non-adaptive UDP traffic to induce time- When using background non-adaptive UDP traffic to induce a time-
varying bottleneck , the physical path capacity remains at varying bottleneck, the physical path capacity remains at 4 Mbps,
4Mbps and the UDP traffic source rate changes over time as (4 - and the UDP traffic source rate changes over time as (4 - (Y x
(Y x r)), where r is the Reference bottleneck capacity in Mbps r)), where r is the Reference bottleneck capacity in Mbps, and Y
and Y is the path capacity ratio specified in Table 1 is the path capacity ratio specified in Table 1.
+--------------------+--------------+-----------+-------------------+ +=========================+================+=======+===============+
| Variation pattern | Path | Start | Path capacity | | Variation pattern index | Path direction | Start | Path capacity |
| index | direction | time | ratio | | | | time | ratio |
+--------------------+--------------+-----------+-------------------+ +=========================+================+=======+===============+
| One | Forward | 0s | 1.0 | | One | Forward | 0 s | 1.0 |
| Two | Forward | 40s | 2.5 | +-------------------------+----------------+-------+---------------+
| Three | Forward | 60s | 0.6 | | Two | Forward | 40 s | 2.5 |
| Four | Forward | 80s | 1.0 | +-------------------------+----------------+-------+---------------+
+--------------------+--------------+-----------+-------------------+ | Three | Forward | 60 s | 0.6 |
+-------------------------+----------------+-------+---------------+
| Four | Forward | 80 s | 1.0 |
+-------------------------+----------------+-------+---------------+
Table 1: Path capacity variation pattern for forward direction Table 1: Path Capacity Variation Pattern for the Forward Direction
5.2. Variable Available Capacity with Multiple Flows 5.2. Variable Available Capacity with Multiple Flows
This test case is similar to Section 5.1. However in addition this This test case is similar to Section 5.1. However, this test will
test will also consider persistent network load due to competing also consider persistent network load due to competing traffic.
traffic.
Expected behavior: the candidate algorithm is expected to detect the Expected behavior: The candidate algorithm is expected to detect the
variation in available capacity and adapt the media stream(s) variation in available capacity and adapt the media stream(s)
accordingly. The flows stabilize around their maximum bit rate as accordingly. The flows stabilize around their maximum bit rate as
the maximum link capacity is large enough to accommodate the flows. the maximum link capacity is large enough to accommodate the
When the available capacity drops, the flows adapt by decreasing flows. When the available capacity drops, the flows adapt by
their sending bit rate, and when congestion disappears, the flows are decreasing their sending bit rate, and when congestion disappears,
again expected to ramp up. the flows are again expected to ramp up.
Evaluation metrics : as described in Section 4.1. Evaluation metrics: As described in Section 4.1.
Testbed Topology: Two (2) media sources S1 and S2 are connected to Testbed topology: Two (2) media sources S1 and S2 are connected to
their corresponding destinations R1 and R2. The media traffic is their corresponding destinations R1 and R2. The media traffic is
transported over the forward path and corresponding feedback/control transported over the forward path and corresponding feedback/
traffic is transported over the backward path. control traffic is transported over the backward path.
+---+ +---+ +---+ +---+
|S1 |===== \ / =======|R1 | |S1 |===== \ / =======|R1 |
+---+ \\ Forward --> // +---+ +---+ \\ Forward --> // +---+
\\ // \\ //
+-----+ +-----+ +-----+ +-----+
| A |------------------------------>| B | | A |------------------------------>| B |
| |<------------------------------| | | |<------------------------------| |
+-----+ +-----+ +-----+ +-----+
// \\ // \\
// <-- Backward \\ // <-- Backward \\
+---+ // \\ +---+ +---+ // \\ +---+
|S2 |====== / \ ======|R2 | |S2 |====== / \ ======|R2 |
+---+ +---+ +---+ +---+
Figure 3: Testbed Topology for Variable Available Capacity Figure 3: Testbed Topology for Variable Available Capacity
Testbed attributes:
Testbed attributes are similar as described in Section 5.1 except the Testbed attributes: Testbed attributes are similar to those
test specific capacity variation setup. described in Section 5.1, except for the test-specific capacity
variation setup.
Test Specific Information: This test uses path capacity variation as Test-specific information: This test uses path capacity variation as
listed in Table 2 with a corresponding end time of 125 seconds. The listed in Table 2 with a corresponding end time of 125 seconds.
reference bottleneck capacity is 2Mbps. When using background non- The reference bottleneck capacity is 2 Mbps. When using
adaptive UDP traffic to induce time-varying bottleneck for congestion background non-adaptive UDP traffic to induce time-varying
controlled media flows, the physical path capacity is 4Mbps and the bottleneck for congestion-controlled media flows, the physical
UDP traffic source rate changes over time as (4 - (Y x r)), where r path capacity is 4 Mbps, and the UDP traffic source rate changes
is the Reference bottleneck capacity in Mbps and Y is the path over time as (4 - (Y x r)), where r is the Reference bottleneck
capacity ratio specified in Table 2. capacity in Mbps, and Y is the path capacity ratio specified in
Table 2.
+--------------------+--------------+-----------+-------------------+ +=========================+================+=======+===============+
| Variation pattern | Path | Start | Path capacity | | Variation pattern index | Path direction | Start | Path capacity |
| index | direction | time | ratio | | | | time | ratio |
+--------------------+--------------+-----------+-------------------+ +=========================+================+=======+===============+
| One | Forward | 0s | 2.0 | | One | Forward | 0 s | 2.0 |
| Two | Forward | 25s | 1.0 | +-------------------------+----------------+-------+---------------+
| Three | Forward | 50s | 1.75 | | Two | Forward | 25 s | 1.0 |
| Four | Forward | 75s | 0.5 | +-------------------------+----------------+-------+---------------+
| Five | Forward | 100s | 1.0 | | Three | Forward | 50 s | 1.75 |
+--------------------+--------------+-----------+-------------------+ +-------------------------+----------------+-------+---------------+
| Four | Forward | 75 s | 0.5 |
+-------------------------+----------------+-------+---------------+
| Five | Forward | 100 s | 1.0 |
+-------------------------+----------------+-------+---------------+
Table 2: Path capacity variation pattern for forward direction Table 2: Path Capacity Variation Pattern for the Forward Direction
5.3. Congested Feedback Link with Bi-directional Media Flows 5.3. Congested Feedback Link with Bi-directional Media Flows
Real-time interactive media uses RTP hence it is assumed that RTCP, Real-time interactive media uses RTP; hence it is assumed that RTCP,
RTP header extension or such would be used by the congestion control RTP header extension, or such would be used by the congestion control
algorithm in the backchannel. Due to the asymmetric nature of the algorithm in the back channel. Due to the asymmetric nature of the
link between communicating peers it is possible for a participating link between communicating peers, it is possible for a participating
peer to not receive such feedback information due to an impaired or peer to not receive such feedback information due to an impaired or
congested backchannel (even when the forward channel might not be congested back channel (even when the forward channel might not be
impaired). This test case is designed to observe the candidate impaired). This test case is designed to observe the candidate
congestion control behavior in such an event. congestion control behavior in such an event.
Expected behavior: It is expected that the candidate algorithms are Expected behavior: It is expected that the candidate algorithms are
able to cope with the lack of feedback information and adapt to able to cope with the lack of feedback information and to adapt to
minimize the performance degradation of media flows in the forward minimize the performance degradation of media flows in the forward
channel. channel.
It should be noted that for this test case: logs are compared with It should be noted that for this test case, logs are compared with
the reference case, i.e, when the backward channel has no the reference case, i.e., when the backward channel has no
impairments. impairments.
Evaluation metrics : as described in Section 4.1. Evaluation metrics: As described in Section 4.1.
Testbed topology: One (1) media source S1 is connected to Testbed topology: One (1) media source S1 is connected to
corresponding R1, but both endpoints are additionally receiving and corresponding R1, but both endpoints are additionally receiving
sending data, respectively. The media traffic (S1->R1) is and sending data, respectively. The media traffic (S1->R1) is
transported over the forward path and corresponding feedback/control transported over the forward path, and the corresponding feedback/
traffic is transported over the backward path. Likewise media control traffic is transported over the backward path. Likewise,
traffic (S2->R2) is transported over the backward path and media traffic (S2->R2) is transported over the backward path, and
corresponding feedback/control traffic is transported over the the corresponding feedback/control traffic is transported over the
forward path. forward path.
+---+ +---+ +---+ +---+
|S1 |===== \ Forward --> / =======|R1 | |S1 |====== \ Forward --> / =======|R1 |
+---+ \\ // +---+ +---+ \\ // +---+
\\ // \\ //
+-----+ +-----+ +-----+ +-----+
| A |------------------------------>| B | | A |------------------------------>| B |
| |<------------------------------| | | |<------------------------------| |
+-----+ +-----+ +-----+ +-----+
// \\ // \\
// <-- Backward \\ // <-- Backward \\
+---+ // \\ +---+ +---+ // \\ +---+
|R2 |===== / \ ======|S2 | |R2 |===== / \ ======|S2 |
+---+ +---+ +---+ +---+
Figure 4: Testbed Topology for Congested Feedback Link Figure 4: Testbed Topology for Congested Feedback Link
Testbed attributes: Testbed attributes:
o Test duration: 100s Test duration: 100 s
o Path characteristics: Path characteristics:
* Reference bottleneck capacity: 1Mbps. Reference bottleneck capacity: 1 Mbps
o Application-related: Application-related:
* Media Source: Media source:
+ Media type: Video Media type: Video
- Media direction: forward and backward Media direction: forward and backward
- Number of media sources: two (2) Number of media sources: two (2)
- Media timeline: Media timeline:
o Start time: 0s. Start time: 0 s
o End time: 99s. End time: 99 s
+ Media type: Audio Media type: Audio
- Media direction: forward and backward Media direction: forward and backward
- Number of media sources: two (2) Number of media sources: two (2)
- Media timeline: Media timeline:
o Start time: 0s. Start time: 0 s
o End time: 99s. End time: 99 s
* Competing traffic: Competing traffic:
+ Number of sources : zero (0) Number of sources: zero (0)
o Test Specific Information: this test uses path capacity variations Test-specific information: This test uses path capacity variations
to create congested feedback link. Table 3 lists the variation to create a congested feedback link. Table 3 lists the variation
patterns applied to the forward path and Table 4 lists the patterns applied to the forward path, and Table 4 lists the
variation patterns applied to the backward path. When using variation patterns applied to the backward path. When using
background non-adaptive UDP traffic to induce time-varying background non-adaptive UDP traffic to induce a time-varying
bottleneck for congestion controlled media flows, the physical bottleneck for congestion-controlled media flows, the physical
path capacity is 4Mbps for both directions and the UDP traffic path capacity is 4 Mbps for both directions, and the UDP traffic
source rate changes over time as (4-x)Mbps in each direction, source rate changes over time as (4-x) Mbps in each direction,
where x is the bottleneck capacity specified in Table 4. where x is the bottleneck capacity specified in Table 4.
+--------------------+--------------+-----------+-------------------+ +=========================+================+=======+===============+
| Variation pattern | Path | Start | Path capacity | | Variation pattern index | Path direction | Start | Path capacity |
| index | direction | time | ratio | | | | time | ratio |
+--------------------+--------------+-----------+-------------------+ +=========================+================+=======+===============+
| One | Forward | 0s | 2.0 | | One | Forward | 0 s | 2.0 |
| Two | Forward | 20s | 1.0 | +-------------------------+----------------+-------+---------------+
| Three | Forward | 40s | 0.5 | | Two | Forward | 20 s | 1.0 |
| Four | Forward | 60s | 2.0 | +-------------------------+----------------+-------+---------------+
+--------------------+--------------+-----------+-------------------+ | Three | Forward | 40 s | 0.5 |
+-------------------------+----------------+-------+---------------+
| Four | Forward | 60 s | 2.0 |
+-------------------------+----------------+-------+---------------+
Table 3: Path capacity variation pattern for forward direction Table 3: Path Capacity Variation Pattern for the Forward Direction
+--------------------+--------------+-----------+-------------------+ +=========================+================+=======+===============+
| Variation pattern | Path | Start | Path capacity | | Variation pattern index | Path direction | Start | Path capacity |
| index | direction | time | ratio | | | | time | ratio |
+--------------------+--------------+-----------+-------------------+ +=========================+================+=======+===============+
| One | Backward | 0s | 2.0 | | One | Backward | 0 s | 2.0 |
| Two | Backward | 35s | 0.8 | +-------------------------+----------------+-------+---------------+
| Three | Backward | 70s | 2.0 | | Two | Backward | 35 s | 0.8 |
+--------------------+--------------+-----------+-------------------+ +-------------------------+----------------+-------+---------------+
| Three | Backward | 70 s | 2.0 |
+-------------------------+----------------+-------+---------------+
Table 4: Path capacity variation pattern for backward direction Table 4: Path Capacity Variation Pattern for the Backward Direction
5.4. Competing Media Flows with same Congestion Control Algorithm 5.4. Competing Media Flows with the Same Congestion Control Algorithm
In this test case, more than one media flow share the bottleneck link In this test case, more than one media flow share the bottleneck
and each of them uses the same congestion control algorithm. This is link, and each of them uses the same congestion control algorithm.
a typical scenario where a real-time interactive application sends This is a typical scenario where a real-time interactive application
more than one media flow to the same destination and these flows are sends more than one media flow to the same destination, and these
multiplexed over the same port. In such a scenario it is likely that flows are multiplexed over the same port. In such a scenario, it is
the flows will be routed via the same path and need to share the likely that the flows will be routed via the same path and need to
available bandwidth amongst themselves. For the sake of simplicity share the available bandwidth amongst themselves. For the sake of
it is assumed that there are no other competing traffic sources in simplicity, it is assumed that there are no other competing traffic
the bottleneck link and that there is sufficient capacity to sources in the bottleneck link and that there is sufficient capacity
accommodate all the flows individually. While this appears to be a to accommodate all the flows individually. While this appears to be
variant of the test case defined in Section 5.2, it focuses on the a variant of the test case defined in Section 5.2, it focuses on the
capacity sharing aspect of the candidate algorithm. The previous capacity-sharing aspect of the candidate algorithm. The previous
test case, on the other hand, measures adaptability, stability, and test case, on the other hand, measures adaptability, stability, and
responsiveness of the candidate algorithm. responsiveness of the candidate algorithm.
Expected behavior: It is expected that the competing flows will Expected behavior: It is expected that the competing flows will
converge to an optimum bit rate to accommodate all the flows with converge to an optimum bit rate to accommodate all the flows with
minimum possible latency and loss. Specifically, the test introduces minimum possible latency and loss. Specifically, the test
three media flows at different time instances, when the second flow introduces three media flows at different time instances. When
appears there should still be room to accommodate another flow on the the second flow appears, there should still be room to accommodate
bottleneck link. Lastly, when the third flow appears the bottleneck another flow on the bottleneck link. Lastly, when the third flow
link should be saturated. appears, the bottleneck link should be saturated.
Evaluation metrics : as described in Section 4.1. Evaluation metrics: As described in Section 4.1.
Testbed topology: Three media sources S1, S2, S3 are connected to R1, Testbed topology: Three media sources S1, S2, and S3 are connected
R2, R3 respectively. The media traffic is transported over the to R1, R2, and R3, respectively. The media traffic is transported
forward path and corresponding feedback/control traffic is over the forward path, and the corresponding feedback/control
transported over the backward path. traffic is transported over the backward path.
+---+ +---+ +---+ +---+
|S1 |===== \ Forward --> / =======|R1 | |S1 |===== \ Forward --> / =======|R1 |
+---+ \\ // +---+ +---+ \\ // +---+
\\ // \\ //
+---+ +-----+ +-----+ +---+ +---+ +-----+ +-----+ +---+
|S2 |=======| A |------------------------------>| B |=======|R2 | |S2 |=======| A |------------------------------>| B |=======|R2 |
+---+ | |<------------------------------| | +---+ +---+ | |<------------------------------| | +---+
+-----+ +-----+ +-----+ +-----+
// <-- Backward \\ // <-- Backward \\
+---+ // \\ +---+ +---+ // \\ +---+
|S3 |===== / \ ======|R3 | |S3 |===== / \ ======|R3 |
+---+ +---+ +---+ +---+
Figure 5: Testbed Topology for Multiple congestion controlled media Figure 5: Testbed Topology for Multiple Congestion-Controlled
Flows Media Flows
Testbed attributes: Testbed attributes:
o Test duration: 120s Test duration: 120 s
o Path characteristics: Path characteristics:
* Reference bottleneck capacity: 3.5Mbps Reference bottleneck capacity: 3.5 Mbps
* Path capacity ratio: 1.0 Path capacity ratio: 1.0
o Application-related: Application-related:
* Media Source: Media Source:
+ Media type: Video Media type: Video
- Media direction: forward. Media direction: forward
- Number of media sources: three (3) Number of media sources: three (3)
- Media timeline: new media flows are added sequentially, Media timeline: New media flows are added sequentially,
at short time intervals. See test specific setup below. at short time intervals. See the test-specific setup
below.
+ Media type: Audio Media type: Audio
- Media direction: forward.
- Number of media sources: three (3) Media direction: forward
- Media timeline: new media flows are added sequentially, Number of media sources: three (3)
at short time intervals. See test specific setup below.
* Competing traffic: Media timeline: New media flows are added sequentially,
at short time intervals. See the test-specific setup
below.
+ Number of sources : zero (0) Competing traffic:
o Test Specific Information: Table 5 defines the media timeline for Number of sources: zero (0)
both media type.
+---------+------------+------------+----------+ Test-specific information: Table 5 defines the media timeline for
both media types.
+=========+============+============+==========+
| Flow ID | Media type | Start time | End time | | Flow ID | Media type | Start time | End time |
+=========+============+============+==========+
| 1 | Video | 0 s | 119 s |
+---------+------------+------------+----------+ +---------+------------+------------+----------+
| 1 | Video | 0s | 119s | | 2 | Video | 20 s | 119 s |
| 2 | Video | 20s | 119s | +---------+------------+------------+----------+
| 3 | Video | 40s | 119s | | 3 | Video | 40 s | 119 s |
| 4 | Audio | 0s | 119s | +---------+------------+------------+----------+
| 5 | Audio | 20s | 119s | | 4 | Audio | 0 s | 119 s |
| 6 | Audio | 40s | 119s | +---------+------------+------------+----------+
| 5 | Audio | 20 s | 119 s |
+---------+------------+------------+----------+
| 6 | Audio | 40 s | 119 s |
+---------+------------+------------+----------+ +---------+------------+------------+----------+
Table 5: Media Timeline for Video and Audio media sources Table 5: Media Timelines for Video and Audio
Media Sources
5.5. Round Trip Time Fairness 5.5. Round Trip Time Fairness
In this test case, multiple media flows share the bottleneck link, In this test case, multiple media flows share the bottleneck link,
but the one-way propagation delay for each flow is different. For but the one-way propagation delay for each flow is different. For
the sake of simplicity it is assumed that there are no other the sake of simplicity, it is assumed that there are no other
competing traffic sources in the bottleneck link and that there is competing traffic sources in the bottleneck link and that there is
sufficient capacity to accommodate all the flows. While this appears sufficient capacity to accommodate all the flows. While this appears
to be a variant of test case 5.2, it focuses on the capacity sharing to be a variant of test case 5.2 (Section 5.2), it focuses on the
aspect of the candidate algorithm under different RTTs. capacity-sharing aspect of the candidate algorithm under different
RTTs.
Expected behavior: It is expected that the competing flows will Expected behavior: It is expected that the competing flows will
converge to bit rates to accommodate all the flows with minimum converge to bit rates to accommodate all the flows with minimum
possible latency and loss. The effectiveness of the algorithm possible latency and loss. The effectiveness of the algorithm
depends on how fast and fairly the competing flows converge to their depends on how fast and fairly the competing flows converge to
steady states irrespective of the RTT observed. their steady states irrespective of the RTT observed.
Evaluation metrics : as described in Section 4.1. Evaluation metrics: As described in Section 4.1.
Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to Testbed topology: Five (5) media sources S1..S5 are connected to
their corresponding media sinks R1,R2,..,R5. The media traffic is their corresponding media sinks R1..R5. The media traffic is
transported over the forward path and corresponding feedback/control transported over the forward path, and the corresponding feedback/
traffic is transported over the backward path. The topology is the control traffic is transported over the backward path. The
same as in Section 5.4. topology is the same as in Section 5.4.
Testbed attributes: Testbed attributes:
o Test duration: 300s Test duration: 300 s
o Path characteristics: Path characteristics:
* Reference bottleneck capacity: 4Mbps Reference bottleneck capacity: 4 Mbps
* Path capacity ratio: 1.0 Path capacity ratio: 1.0
* One-Way propagation delay for each flow: 10ms for S1-R1, 25ms One-way propagation delay for each flow: 10 ms for S1-R1, 25
for S2-R2, 50ms for S3-R3, 100ms for S4-R4, and 150ms S5-R5. ms for S2-R2, 50 ms for S3-R3, 100 ms for S4-R4, and 150 ms
S5-R5.
o Application-related: Application-related:
* Media Source: Media source:
+ Media type: Video Media type: Video
- Media direction: forward Media direction: forward
- Number of media sources: five (5) Number of media sources: five (5)
- Media timeline: new media flows are added sequentially, Media timeline: New media flows are added sequentially,
at short time intervals. See test specific setup below. at short time intervals. See the test-specific setup
below.
+ Media type: Audio Media type: Audio
- Media direction: forward. Media direction: forward
- Number of media sources: five (5) Number of media sources: five (5)
- Media timeline: new media flows are added sequentially, Media timeline: New media flows are added sequentially,
at short time intervals. See test specific setup below. at short time intervals. See the test-specific setup
below.
* Competing traffic: Competing traffic:
+ Number of sources : zero (0) Number of sources: zero (0)
o Test Specific Information: Table 6 defines the media timeline for Test-specific information: Table 6 defines the media timeline for
both media type. both media types.
+=========+============+============+==========+
| Flow ID | Media type | Start time | End time |
+=========+============+============+==========+
| 1 | Video | 0 s | 299 s |
+---------+------------+------------+----------+ +---------+------------+------------+----------+
| Flow IF | Media type | Start time | End time | | 2 | Video | 10 s | 299 s |
+---------+------------+------------+----------+ +---------+------------+------------+----------+
| 1 | Video | 0s | 299s | | 3 | Video | 20 s | 299 s |
| 2 | Video | 10s | 299s | +---------+------------+------------+----------+
| 3 | Video | 20s | 299s | | 4 | Video | 30 s | 299 s |
| 4 | Video | 30s | 299s | +---------+------------+------------+----------+
| 5 | Video | 40s | 299s | | 5 | Video | 40 s | 299 s |
| 6 | Audio | 0 | 299s | +---------+------------+------------+----------+
| 7 | Audio | 10s | 299s | | 6 | Audio | 0 s | 299 s |
| 8 | Audio | 20s | 299s | +---------+------------+------------+----------+
| 9 | Audio | 30s | 299s | | 7 | Audio | 10 s | 299 s |
| 10 | Audio | 40s | 299s | +---------+------------+------------+----------+
| 8 | Audio | 20 s | 299 s |
+---------+------------+------------+----------+
| 9 | Audio | 30 s | 299 s |
+---------+------------+------------+----------+
| 10 | Audio | 40 s | 299 s |
+---------+------------+------------+----------+ +---------+------------+------------+----------+
Table 6: Media Timeline for Video and Audio media sources Table 6: Media Timeline for Video and Audio
Media Sources
5.6. Media Flow Competing with a Long TCP Flow 5.6. Media Flow Competing with a Long TCP Flow
In this test case, one or more media flows share the bottleneck link In this test case, one or more media flows share the bottleneck link
with at least one long lived TCP flow. Long lived TCP flows download with at least one long-lived TCP flow. Long-lived TCP flows download
data throughout the session and are expected to have infinite amount data throughout the session and are expected to have infinite amount
of data to send and receive. This is a scenario where a multimedia of data to send and receive. This is a scenario where a multimedia
application co-exists with a large file download. The test case application coexists with a large file download. The test case
measures the adaptivity of the candidate algorithm to competing measures the adaptivity of the candidate algorithm to competing
traffic. It addresses the requirement 3 in traffic. It addresses requirement 3 in Section 2 of [RFC8836].
[I-D.ietf-rmcat-cc-requirements].
Expected behavior: depending on the convergence observed in test case Expected behavior: Depending on the convergence observed in test
5.1 and 5.2, the candidate algorithm may be able to avoid congestion cases 5.1 and 5.2, the candidate algorithm may be able to avoid
collapse. In the worst case, the media stream will fall to the congestion collapse. In the worst case, the media stream will
minimum media bit rate. fall to the minimum media bit rate.
Evaluation metrics : following metrics in addition to as described in Evaluation metrics: Includes the following metrics in addition to
Section 4.1. those described in Section 4.1:
1. Flow level: 1. Flow level:
A. TCP throughput. a. TCP throughput
B. Loss for the TCP flow b. Loss for the TCP flow
Testbed topology: One (1) media source S1 is connected to the Testbed topology: One (1) media source S1 is connected to the
corresponding media sink, R1. In addition, there is a long-live TCP corresponding media sink, R1. In addition, there is a long-lived
flow sharing the same bottleneck link. The media traffic is TCP flow sharing the same bottleneck link. The media traffic is
transported over the forward path and corresponding feedback/control transported over the forward path, and the corresponding feedback/
traffic is transported over the backward path. The TCP traffic goes control traffic is transported over the backward path. The TCP
over the forward path from, S_tcp with acknowledgment packets go over traffic goes over the forward path from S_tcp with acknowledgment
the backward path from, R_tcp. packets going over the backward path from R_tcp.
+--+ +--+ +--+ +--+
|S1|===== \ Forward --> / =====|R1| |S1|===== \ Forward --> / =====|R1|
+--+ \\ // +--+ +--+ \\ // +--+
\\ // \\ //
+-----+ +-----+ +-----+ +-----+
| A |---------------------------->| B | | A |---------------------------->| B |
| |<----------------------------| | | |<----------------------------| |
+-----+ +-----+ +-----+ +-----+
// <-- Backward \\ // <-- Backward \\
+-----+ // \\ +-----+ +-----+ // \\ +-----+
|S_tcp|=== / \ ===|R_tcp| |S_tcp|=== / \ ===|R_tcp|
+-----+ +-----+ +-----+ +-----+
Figure 6: Testbed Topology for TCP vs congestion controlled media Figure 6: Testbed Topology for TCP vs Congestion-Controlled Media
Flows Flows
Testbed attributes: Testbed attributes:
o Test duration: 120s Test duration: 120 s
o Path characteristics: Path characteristics:
* Reference bottleneck capacity: 2Mbps Reference bottleneck capacity: 2 Mbps
* Path capacity ratio: 1.0 Path capacity ratio: 1.0
* Bottleneck queue size: [300ms, 1000ms] Bottleneck queue size: [300 ms, 1000 ms]
o Application-related: Application-related:
* Media Source: Media source:
+ Media type: Video Media type: Video
- Media direction: forward Media direction: forward
- Number of media sources: one (1) Number of media sources: one (1)
- Media timeline: Media timeline:
o Start time: 5s. Start time: 5 s
o End time: 119s. End time: 119 s
+ Media type: Audio Media type: Audio
- Media direction: forward
- Number of media sources: one (1) Media direction: forward
- Media timeline: Number of media sources: one (1)
o Start time: 5s. Media timeline:
o End time: 119s. Start time: 5 s
* Additionally, implementers are encouraged to run the experiment End time: 119 s
with multiple media sources.
* Competing traffic: Additionally, implementers are encouraged to run the
experiment with multiple media sources.
+ Number and Types of sources : one (1) and long-lived TCP Competing traffic:
+ Traffic direction : forward Number and types of sources: one (1) and long-lived TCP
+ Congestion control: default TCP congestion control[RFC5681]. Traffic direction: forward
Implementers are also encouraged to run the experiment with
alternative TCP congestion control algorithm.
+ Traffic timeline: Congestion control: default TCP congestion control
[RFC5681]. Implementers are also encouraged to run the
experiment with alternative TCP congestion control
algorithms.
- Start time: 0s. Traffic timeline:
- End time: 119s. Start time: 0 s
o Test Specific Information: none End time: 119 s
Test-specific information: none
5.7. Media Flow Competing with Short TCP Flows 5.7. Media Flow Competing with Short TCP Flows
In this test case, one or more congestion controlled media flow In this test case, one or more congestion-controlled media flows
shares the bottleneck link with multiple short-lived TCP flows. share the bottleneck link with multiple short-lived TCP flows.
Short-lived TCP flows resemble the on/off pattern observed in the web Short-lived TCP flows resemble the on/off pattern observed in web
traffic, wherein clients (for example, browsers) connect to a server traffic, wherein clients (for example, browsers) connect to a server
and download a resource (typically a web page, few images, text and download a resource (typically a web page, few images, text
files, etc.) using several TCP connections. This scenario shows the files, etc.) using several TCP connections. This scenario shows the
performance of a multimedia application when several browser windows performance of a multimedia application when several browser windows
are active. The test case measures the adaptivity of the candidate are active. The test case measures the adaptivity of the candidate
algorithm to competing web traffic, it addresses the requirements 1.E algorithm to competing web traffic, and it addresses requirement 1.E
in [I-D.ietf-rmcat-cc-requirements]. in Section 2 of [RFC8836].
Depending on the number of short TCP flows, the cross-traffic either Depending on the number of short TCP flows, the cross traffic either
appears as a short burst flow or resembles a long TCP flow. The appears as a short burst flow or resembles a long-lived TCP flow.
intention of this test is to observe the impact of short-term burst The intention of this test is to observe the impact of a short-term
on the behavior of the candidate algorithm. burst on the behavior of the candidate algorithm.
Expected behavior: The candidate algorithm is expected to avoid flow Expected behavior: The candidate algorithm is expected to avoid flow
starvation during the presence of short and bursty competing TCP starvation during the presence of short and bursty competing TCP
flows, streaming at least at the minimum media bit rate. After flows, streaming at least at the minimum media bit rate. After
competing TCP flows terminate, the media streams are expected to be competing TCP flows terminate, the media streams are expected to
robust enough to eventually recover to previous steady state be robust enough to eventually recover to previous steady state
behavior, and at the very least, avoid persistent starvation. behavior, and at the very least, avoid persistent starvation.
Evaluation metrics : following metrics in addition to as described in Evaluation metrics: Includes the following metrics in addition to
Section 4.1. those described in Section 4.1:
1. Flow level: 1. Flow level:
A. Variation in the sending rate of the TCP flow. A. Variation in the sending rate of the TCP flow
B. TCP throughput. B. TCP throughput
Testbed topology: The topology described here is same as the one Testbed topology: The topology described here is the same as the one
described in Figure 6. described in Figure 6.
Testbed attributes: Testbed attributes:
o Test duration: 300s Test duration: 300 s
o Path characteristics: Path characteristics:
* Reference bottleneck capacity: 2.0Mbps Reference bottleneck capacity: 2.0 Mbps
* Path capacity ratio: 1.0 Path capacity ratio: 1.0
o Application-related: Application-related:
* Media source: Media source:
+ Media type: Video Media type: Video
- Media direction: forward Media direction: forward
- Number of media sources: two (2) Number of media sources: two (2)
- Media timeline: Media timeline:
o Start time: 5s. Start time: 5 s
o End time: 299s. End time: 299 s
+ Media type: Audio Media type: Audio
- Media direction: forward Media direction: forward
- Number of media sources: two (2)
- Media timeline: Number of media sources: two (2)
o Start time: 5s. Media timeline:
o End time: 299s. Start time: 5 s
* Competing traffic: End time: 299 s
+ Number and Types of sources : ten (10), short-lived TCP Competing traffic:
flows.
+ Traffic direction : forward Number and types of sources: ten (10), short-lived TCP
flows.
+ Congestion algorithm: default TCP Congestion control Traffic direction: forward
[RFC5681]. Implementers are also encouraged to run the
experiment with alternative TCP congestion control
algorithm.
+ Traffic timeline: each short TCP flow is modeled as a Congestion algorithm: default TCP congestion control
sequence of file downloads interleaved with idle periods. [RFC5681]. Implementers are also encouraged to run the
Not all short TCP flows start at the same time, 2 of them experiment with an alternative TCP congestion control
start in the ON state while rest of the 8 flows start in an algorithm.
OFF state. For description of short TCP flow model see test
specific information below.
o Test Specific Information: Traffic timeline: Each short TCP flow is modeled as a
sequence of file downloads interleaved with idle periods.
Not all short TCP flows start at the same time, two of
them start in the ON state, while rest of the eight flows
start in an OFF state. For a description of the short
TCP flow model, see test-specific information below.
* Short-TCP traffic model: The short TCP model to be used in this Test-specific information:
test is described in [I-D.ietf-rmcat-eval-criteria].
Short TCP traffic model: The short TCP model to be used in this
test is described in [RFC8868].
5.8. Media Pause and Resume 5.8. Media Pause and Resume
In this test case, more than one real-time interactive media flows In this test case, more than one real-time interactive media flow
share the link bandwidth and all flows reach to a steady state by share the link bandwidth, and all flows reach to a steady state by
utilizing the link capacity in an optimum way. At this stage one of utilizing the link capacity in an optimum way. At this stage, one of
the media flows is paused for a moment. This event will result in the media flows is paused for a moment. This event will result in
more available bandwidth for the rest of the flows as they are on a more available bandwidth for the rest of the flows as they are on a
shared link. When the paused media flow resumes it would no longer shared link. When the paused media flow resumes, it no longer has
have the same bandwidth share on the link. It has to make its way the same bandwidth share on the link. It has to make its way through
through the other existing flows in the link to achieve a fair share the other existing flows in the link to achieve a fair share of the
of the link capacity. This test case is important specially for link capacity. This test case is important specially for real-time
real-time interactive media which consists of more than one media interactive media, which consists of more than one media flows and
flows and can pause/resume media flows at any point of time during can pause/resume media flows at any point of time during the session.
the session. This test case directly addresses the requirement This test case directly addresses requirement 5 in Section 2 of
number 5 in [I-D.ietf-rmcat-cc-requirements]. One can think it as a [RFC8836]. One can think of it as a variation of the test case
variation of test case defined in Section 5.4. However, it is defined in Section 5.4. However, it is different as the candidate
different as the candidate algorithms can use different strategies to algorithms can use different strategies to increase efficiency, for
increase its efficiency, for example in terms of fairness, example, in terms of fairness, convergence time, oscillation
convergence time, reduce oscillation etc, by capitalizing the fact reduction, etc., by capitalizing on the fact that they have previous
that they have previous information of the link. information of the link.
Expected behavior: During the period where the third stream is Expected behavior: During the period where the third stream is
paused, the two remaining flows are expected to increase their rates paused, the two remaining flows are expected to increase their
and reach the maximum media bit rate. When the third stream resumes, rates and reach the maximum media bit rate. When the third stream
all three flows are expected to converge to the same original fair resumes, all three flows are expected to converge to the same
share of rates prior to the media pause/resume event. original fair share of rates prior to the media pause/resume
event.
Evaluation metrics : following metrics in addition to as described in Evaluation metrics: Includes the following metrics in addition to
Section 4.1. those described in Section 4.1:
1. Flow level: 1. Flow level:
A. Variation in sending bit rate and throughput. Mainly A. Variation in sending bit rate and throughput. Mainly
observing the frequency and magnitude of oscillations. observing the frequency and magnitude of oscillations.
Testbed Topology: Same as test case defined in Section 5.4 Testbed topology: Same as the test case defined in Section 5.4.
Testbed attributes: The general description of the testbed parameters Testbed attributes: The general description of the testbed
are same as Section 5.4 with changes in the test specific setup as parameters are the same as Section 5.4 with changes in the test-
below- specific setup as below:
o Other test specific setup: Other test-specific setup:
* Media flow timeline: Media flow timeline:
+ Flow ID: one (1) Flow ID: one (1)
+ Start time: 0s Start time: 0 s
+ Flow duration: 119s Flow duration: 119 s
+ Pause time: not required Pause time: not required
+ Resume time: not required Resume time: not required
* Media flow timeline: Media flow timeline:
+ Flow ID: two (2) Flow ID: two (2)
+ Start time: 0s Start time: 0 s
+ Flow duration: 119s Flow duration: 119 s
+ Pause time: at 40s Pause time: at 40 s
+ Resume time: at 60s
* Media flow timeline: Resume time: at 60 s
+ Flow ID: three (3) Media flow timeline:
+ Start time: 0s Flow ID: three (3)
+ Flow duration:119s Start time: 0 s
+ Pause time: not required Flow duration: 119 s
+ Resume time: not required Pause time: not required
6. Other potential test cases Resume time: not required
6. Other Potential Test Cases
It has been noticed that there are other interesting test cases It has been noticed that there are other interesting test cases
besides the basic test cases listed above. In many aspects, these besides the basic test cases listed above. In many aspects, these
additional test cases can help further evaluation of the candidate additional test cases can help further evaluation of the candidate
algorithm. They are listed as below. algorithm. They are listed below.
6.1. Media Flows with Priority 6.1. Media Flows with Priority
In this test case media flows will have different priority levels. In this test case, media flows will have different priority levels.
This will be an extension of Section 5.4 where the same test will be This is an extension of Section 5.4 where the same test is run with
run with different priority levels imposed on each of the media different priority levels imposed on each of the media flows. For
flows. For example, the first flow (S1) is assigned a priority of 2 example, the first flow (S1) is assigned a priority of 2, whereas the
whereas the remaining two flows (S2 and S3) are assigned a priority remaining two flows (S2 and S3) are assigned a priority of 1. The
of 1. The candidate algorithm must reflect the relative priorities candidate algorithm must reflect the relative priorities assigned to
assigned to each media flow. In this case, the first flow (S1) must each media flow. In this case, the first flow (S1) must arrive at a
arrive at a steady-state rate approximately twice of that of the steady-state rate approximately twice that of the other two flows (S2
other two flows (S2 and S3). and S3).
The candidate algorithm can use a coupled congestion control The candidate algorithm can use a coupled congestion control
mechanism [I-D.ietf-rmcat-coupled-cc] or use a weighted priority mechanism [RFC8699] or use a weighted priority scheduler for the
scheduler for the bandwidth distribution according to the respective bandwidth distribution according to the respective media flow
media flow priority or use. priority or use.
6.2. Explicit Congestion Notification Usage 6.2. Explicit Congestion Notification Usage
This test case requires to run all the basic test cases with the This test case requires running all the basic test cases with the
availability of Explicit Congestion Notification (ECN) [RFC6679] availability of Explicit Congestion Notification (ECN) [RFC6679]
feature enabled. The goal of this test is to exhibit that the feature enabled. The goal of this test is to exhibit that the
candidate algorithms do not fail when ECN signals are available. candidate algorithms do not fail when ECN signals are available.
With ECN signals enabled the algorithms are expected to perform With ECN signals enabled, the algorithms are expected to perform
better than their delay-based variants. better than their delay-based variants.
6.3. Multiple Bottlenecks 6.3. Multiple Bottlenecks
In this test case one congestion controlled media flow, S1->R1, In this test case, one congestion-controlled media flow, S1->R1,
traverses a path with multiple bottlenecks. As illustrated in traverses a path with multiple bottlenecks. As illustrated in
Figure 7, the first flow (S1->R1) competes with the second congestion Figure 7, the first flow (S1->R1) competes with the second
controlled media flow (S2->R2) over the link between A and B which is congestion-controlled media flow (S2->R2) over the link between A and
close to the sender side; again, that flow (S1->R1) competes with the B, which is close to the sender side. Again, that flow (S1->R1)
third congestion controlled media flow (S3->R3) over the link between competes with the third congestion-controlled media flow (S3->R3)
C and D which is close to the receiver side. The goal of this test over the link between C and D, which is close to the receiver side.
is to ensure that the candidate algorithms work properly in the The goal of this test is to ensure that the candidate algorithms work
presence of multiple bottleneck links on the end to end path. properly in the presence of multiple bottleneck links on the end-to-
end path.
Expected behavior: The candidate algorithm is expected to achieve Expected behavior: The candidate algorithm is expected to achieve
full utilization at both bottleneck links without starving any of the full utilization at both bottleneck links without starving any of
three congestion controlled media flows and ensuring fair share of the three congestion-controlled media flows and ensuring fair
the available bandwidth at each bottlenecks. share of the available bandwidth at each bottleneck.
Forward ----> Forward ---->
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
|S2 | |R2 | |S3 | |R3 | |S2 | |R2 | |S3 | |R3 |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
| | | | | | | |
| | | | | | | |
+---+ +-----+ +-----+ +-----+ +-----+ +---+ +---+ +-----+ +-----+ +-----+ +-----+ +---+
|S1 |=======| A |------>| B |----->| C |---->| D |=======|R1 | |S1 |======| A |------>| B |----->| C |---->| D |======|R1 |
+---+ | |<------| |<-----| |<----| | +---+ +---+ | |<------| |<-----| |<----| | +---+
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
1st 2nd 1st 2nd
Bottleneck (A->B) Bottleneck (C->D) Bottleneck (A->B) Bottleneck (C->D)
<------ Backward <------ Backward
Figure 7: Testbed Topology for Multiple Bottlenecks Figure 7: Testbed Topology for Multiple Bottlenecks
Testbed topology: Three media sources S1, S2, and S3 are connected to Testbed topology: Three media sources S1, S2, and S3 are connected
respective destinations R1, R2, and R3. For all three flows the to respective destinations R1, R2, and R3. For all three flows,
media traffic is transported over the forward path and corresponding the media traffic is transported over the forward path, and the
feedback/control traffic is transported over the backward path. corresponding feedback/control traffic is transported over the
backward path.
Testbed attributes: Testbed attributes:
o Test duration: 300s Test duration: 300 s
o Path characteristics: Path characteristics:
* Reference bottleneck capacity: 2Mbps. Reference bottleneck capacity: 2 Mbps
* Path capacity ratio between A and B: 1.0 Path capacity ratio between A and B: 1.0
* Path capacity ratio between B and C: 4.0. Path capacity ratio between B and C: 4.0
* Path capacity ratio between C and D: 0.75. Path capacity ratio between C and D: 0.75
* One-Way propagation delay: One-way propagation delay:
1. Between S1 and R1: 100ms Between S1 and R1: 100 ms
2. Between S2 and R2: 40ms Between S2 and R2: 40 ms
3. Between S3 and R3: 40ms Between S3 and R3: 40 ms
o Application-related: Application-related:
* Media Source: Media source:
+ Media type: Video Media type: Video
- Media direction: Forward Media direction: Forward
- Number of media sources: Three (3) Number of media sources: Three (3)
- Media timeline: Media timeline:
o Start time: 0s. Start time: 0 s
o End time: 299s. End time: 299 s
+ Media type: Audio Media type: Audio
- Media direction: Forward Media direction: Forward
- Number of media sources: Three (3) Number of media sources: Three (3)
- Media timeline: Media timeline:
o Start time: 0s. Start time: 0 s
o End time: 299s. End time: 299 s
* Competing traffic: Competing traffic:
+ Number of sources : Zero (0) Number of sources: Zero (0)
7. Wireless Access Links 7. Wireless Access Links
Additional wireless network (both cellular network and WiFi network) Additional wireless network (both cellular network and Wi-Fi network)
specific test cases are defined in [I-D.ietf-rmcat-wireless-tests]. specific test cases are defined in [RFC8869].
8. Security Considerations 8. Security Considerations
The security considerations in [I-D.ietf-rmcat-eval-criteria] and the The security considerations in Section 6 of [RFC8868] and the
relevant congestion control algorithms apply. The principles for relevant congestion control algorithms apply. The principles for
congestion control are described in [RFC2914], and in particular any congestion control are described in [RFC2914], and in particular any
new method must implement safeguards to avoid congestion collapse of new method must implement safeguards to avoid congestion collapse of
the Internet. the Internet.
The evaluation of the test cases are intended to be run in a The evaluation of the test cases are intended to be run in a
controlled lab environment. Hence, the applications, simulators and controlled lab environment. Hence, the applications, simulators, and
network nodes ought to be well-behaved and should not impact the network nodes ought to be well-behaved and should not impact the
desired results. Moreover, proper measures must be taken to avoid desired results. Moreover, proper measures must be taken to avoid
leaking non-responsive traffic from unproven congestion avoidance leaking nonresponsive traffic from unproven congestion avoidance
techniques onto the open Internet. techniques onto the open Internet.
9. IANA Considerations 9. IANA Considerations
There are no IANA impacts in this memo. This document has no IANA actions.
10. Acknowledgements
Much of this document is derived from previous work on congestion
control at the IETF.
The content and concepts within this document are a product of the
discussion carried out in the Design Team.
11. References
11.1. Normative References
[I-D.ietf-rmcat-cc-requirements]
Jesup, R. and Z. Sarker, "Congestion Control Requirements
for Interactive Real-Time Media", draft-ietf-rmcat-cc-
requirements-09 (work in progress), December 2014.
[I-D.ietf-rmcat-eval-criteria]
Singh, V., Ott, J., and S. Holmer, "Evaluating Congestion
Control for Interactive Real-time Media", draft-ietf-
rmcat-eval-criteria-08 (work in progress), November 2018.
[I-D.ietf-rmcat-video-traffic-model] 10. References
Zhu, X., Cruz, S., and Z. Sarker, "Video Traffic Models
for RTP Congestion Control Evaluations", draft-ietf-rmcat-
video-traffic-model-07 (work in progress), February 2019.
[I-D.ietf-rmcat-wireless-tests] 10.1. Normative References
Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and
M. Ramalho, "Evaluation Test Cases for Interactive Real-
Time Media over Wireless Networks", draft-ietf-rmcat-
wireless-tests-06 (work in progress), December 2018.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <https://www.rfc-editor.org/info/rfc3550>. July 2003, <https://www.rfc-editor.org/info/rfc3550>.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
Video Conferences with Minimal Control", STD 65, RFC 3551, Video Conferences with Minimal Control", STD 65, RFC 3551,
DOI 10.17487/RFC3551, July 2003, DOI 10.17487/RFC3551, July 2003,
<https://www.rfc-editor.org/info/rfc3551>. <https://www.rfc-editor.org/info/rfc3551>.
skipping to change at page 32, line 10 skipping to change at line 1466
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
<https://www.rfc-editor.org/info/rfc5681>. <https://www.rfc-editor.org/info/rfc5681>.
[RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P., [RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P.,
and K. Carlberg, "Explicit Congestion Notification (ECN) and K. Carlberg, "Explicit Congestion Notification (ECN)
for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August
2012, <https://www.rfc-editor.org/info/rfc6679>. 2012, <https://www.rfc-editor.org/info/rfc6679>.
11.2. Informative References [RFC8593] Zhu, X., Mena, S., and Z. Sarker, "Video Traffic Models
for RTP Congestion Control Evaluations", RFC 8593,
DOI 10.17487/RFC8593, May 2019,
<https://www.rfc-editor.org/info/rfc8593>.
[HEVC-seq] [RFC8836] Jesup, R. and Z. Sarker, Ed., "Congestion Control
HEVC, "Test Sequences", Requirements for Interactive Real-Time Media", RFC 8836,
http://www.netlab.tkk.fi/~varun/test_sequences/ . DOI 10.17487/RFC8836, January 2021,
<https://www.rfc-editor.org/info/rfc8836>.
[I-D.ietf-rmcat-coupled-cc] [RFC8868] Singh, V., Ott, J., and S. Holmer, "Evaluating Congestion
Islam, S., Welzl, M., and S. Gjessing, "Coupled congestion Control for Interactive Real-Time Media", RFC 8868,
control for RTP media", draft-ietf-rmcat-coupled-cc-08 DOI 10.17487/RFC8868, January 2021,
(work in progress), January 2019. <https://www.rfc-editor.org/info/rfc8868>.
[RFC8869] Sarker, Z., Zhu, X., and J. Fu, "Evaluation Test Cases for
Interactive Real-Time Media over Wireless Networks",
RFC 8869, DOI 10.17487/RFC8869, January 2021,
<https://www.rfc-editor.org/info/rfc8869>.
10.2. Informative References
[HEVC-seq] HEVC, "Test Sequences",
<http://www.netlab.tkk.fi/~varun/test_sequences/>.
[RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, [RFC2914] Floyd, S., "Congestion Control Principles", BCP 41,
RFC 2914, DOI 10.17487/RFC2914, September 2000, RFC 2914, DOI 10.17487/RFC2914, September 2000,
<https://www.rfc-editor.org/info/rfc2914>. <https://www.rfc-editor.org/info/rfc2914>.
[RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF [RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF
Recommendations Regarding Active Queue Management", Recommendations Regarding Active Queue Management",
BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015, BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015,
<https://www.rfc-editor.org/info/rfc7567>. <https://www.rfc-editor.org/info/rfc7567>.
skipping to change at page 32, line 42 skipping to change at line 1512
Lightweight Control Scheme to Address the Bufferbloat Lightweight Control Scheme to Address the Bufferbloat
Problem", RFC 8033, DOI 10.17487/RFC8033, February 2017, Problem", RFC 8033, DOI 10.17487/RFC8033, February 2017,
<https://www.rfc-editor.org/info/rfc8033>. <https://www.rfc-editor.org/info/rfc8033>.
[RFC8290] Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys, [RFC8290] Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys,
J., and E. Dumazet, "The Flow Queue CoDel Packet Scheduler J., and E. Dumazet, "The Flow Queue CoDel Packet Scheduler
and Active Queue Management Algorithm", RFC 8290, and Active Queue Management Algorithm", RFC 8290,
DOI 10.17487/RFC8290, January 2018, DOI 10.17487/RFC8290, January 2018,
<https://www.rfc-editor.org/info/rfc8290>. <https://www.rfc-editor.org/info/rfc8290>.
[xiph-seq] [RFC8699] Islam, S., Welzl, M., and S. Gjessing, "Coupled Congestion
Xiph.org, "Video Test Media", Control for RTP Media", RFC 8699, DOI 10.17487/RFC8699,
http://media.xiph.org/video/derf/ . January 2020, <https://www.rfc-editor.org/info/rfc8699>.
[xiph-seq] Xiph.org, "Video Test Media",
<http://media.xiph.org/video/derf/>.
Acknowledgments
Much of this document is derived from previous work on congestion
control at the IETF.
The content and concepts within this document are a product of the
discussion carried out within the Design Team.
Authors' Addresses Authors' Addresses
Zaheduzzaman Sarker Zaheduzzaman Sarker
Ericsson AB Ericsson AB
Torshamnsgatan 23 Torshamnsgatan 23
Stockholm, SE 164 83 SE-164 83 Stockholm
Sweden Sweden
Phone: +46 10 717 37 43 Phone: +46 10 717 37 43
Email: zaheduzzaman.sarker@ericsson.com Email: zaheduzzaman.sarker@ericsson.com
Varun Singh Varun Singh
Nemu Dialogue Systems Oy CALLSTATS I/O Oy
Runeberginkatu 4c A 4 Rauhankatu 11 C
Helsinki 00100 FI-00100 Helsinki
Finland Finland
Email: varun.singh@iki.fi Email: varun.singh@iki.fi
URI: http://www.callstats.io/ URI: http://www.callstats.io/
Xiaoqing Zhu Xiaoqing Zhu
Cisco Systems Cisco Systems
12515 Research Blvd 12515 Research Blvd
Austing, TX 78759 Austin, TX 78759
USA United States of America
Email: xiaoqzhu@cisco.com Email: xiaoqzhu@cisco.com
Michael A. Ramalho Michael A. Ramalho
Cisco Systems, Inc. AcousticComms Consulting
6310 Watercrest Way Unit 203 6310 Watercrest Way Unit 203
Lakewood Ranch, FL 34202-5211 Lakewood Ranch, FL 34202-5211
USA United States of America
Phone: +1 919 476 2038 Phone: +1 732 832 9723
Email: mramalho@cisco.com Email: mar42@cornell.edu
URI: http://ramalho.webhop.info/
 End of changes. 376 change blocks. 
814 lines changed or deleted 848 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/