Internet Engineering Task Force Audio Video Transport WG Internet Draft J.Rosenberg,H.Schulzrinne draft-ietf-avt-rtcptest-01.txt Bell Laboratories,Columbia U. June 25, 1999 Expires: December 1999 Conformance Tests for RTP Scalability Algorithms STATUS OF THIS MEMO This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as work in progress. The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract This document defines a set of tests that can be run on an RTP imple- mentation to determine the level of conformance with the various sca- lability algorithms defined in the draft version of RTP. These tests do not require access to internal implementations, and use black box testing techniques to determine compliance. 1 Introduction The draft standard version of RTP [1] contains a number of algorithms and behaviors related to transmission and reception of RTCP packets. These algorithms include periodic transmission of RTCP packets, SSRC collision detection, and allocation of differing RTCP rates to senders and receivers. Many of these algorithms (forward reconsidera- tion, reverse reconsideration, and BYE reconsideration) relate to J.Rosenberg,H.Schulzrinne [Page 1] Internet Draft RTCP Test June 25, 1999 scalability features for large groups [2,3], and have been added in the transition from proposed to draft standard. Other mechanisms have been added to enhance robustness for high bandwidth sessions, or to allow greater flexibility in choosing RTCP bandwidth usage. It is essential for the evolution of RTP to evaluate the interopera- bility of these features. However, many of these algorithms are local computations which affect timers and other session variables. These do not get reflected in the content of messages that are sent by RTP participants. As a result, correctness and interoperability of this features is difficult to assess. Other features of RTP (such as SSRC collision) occur in unusual circumstances, and will not likely show up in a typical interop test. To get around these problems, this document proposes a suite of con- formance tests that can be run on an implementation to test its com- pliance to various components of the draft standard RTP. These tests are "black box" tests, and do not require access to the internal workings of the implementation being tested. 2 Testing Architecture The basic architecture for performing the tests is shown in Figure 1. -------------- | | | test | | instrument | | | -------------- | -------------------------------- LAN | ---------------- | | | RTP | | implementation | | | ---------------- Figure 1: Testing Architecture The test instrument is connected on the same LAN as the RTP J.Rosenberg,H.Schulzrinne [Page 2] Internet Draft RTCP Test June 25, 1999 implementation being tested. It is assumed that the test instrument is preconfigured with the addresses and ports used by the RTP imple- mentation, and is also aware of the RTCP bandwidth and sender/receiver fractions. The tests can be conducted using either multicast or unicast. The test instrument must be capable of sending arbitrarily crafted RTP and RTCP packets to the RTP implementation. The instrument should also be capable of receiving packets sent by the RTP implementation, parsing them, and computing metrics based on those packets. It is furthermore assumed that a number of basic controls over the RTP implementation exist. These controls are: o the ability to force the implementation to send or not send RTP packets at any desired point in time; o the ability to force the application to terminate its involve- ment in the RTP session, and for this termination to be known immediately to the test instrument; o the ability to set the session bandwidth and RTCP sender and receiver fractions; The second of these is required only for the test of BYE reconsidera- tion, and is the only aspect of these tests not easily implementable by pure automation. It will generally require manual intervention to terminate the session from the RTP implementation and to convey this to the test instrument through some non-RTP means. 3 Basic Behavior The first test is to verify basic correctness of the implementation of the RTCP transmission rules. This basic behavior consists of: o periodic transmission of RTCP packets; o randomization of the interval for RTCP packet transmission; o correct implementation of the randomization interval computa- tions, with unconditional reconsideration. These features are tested in the following manner. The RTP implemen- tation acts as a session receiver, and never sends any RTP packets. The implementation is configured with a large session bandwidth, say 1 Mb/s. This will cause the implementation to use the minimal inter- val of 5s rather than the small interval based on the session bandwidth and membership size. The implementation will generate RTCP J.Rosenberg,H.Schulzrinne [Page 3] Internet Draft RTCP Test June 25, 1999 packets at this minimal interval, on average. The test instrument generates no packets, but receives the RTCP packets generated by the implementation. When an RTCP packet is received, the time is noted by the test instrument. The difference in time between each pair of sub- sequent packets (called the interval) is computed. These intervals are stored, so that statistics based on these intervals can be com- puted. It is recommended that this observation process operate for at least 20 minutes. An implementation passes this test if the intervals have the follow- ing properties: o the minimum interval is never less than 2 seconds or more than 2.5 seconds; o the maximum interval is never more than 7 seconds or less than 5.5 seconds; o the average interval is between 4.5 and 5.5 seconds; o the number of intervals between x and x+500ms is less than the number of intervals between x+500ms and x+1s, for any x. In particular, an implementation fails if the packets are sent with a constant interval. 4 Step join backoff The main purpose of the reconsideration algorithm is to avoid a flood of packets that might occur when a large number of users simultane- ously join an RTP session. Reconsideration therefore exhibits a back- off behavior in sending of RTCP packets when group sizes increase. This aspect of the algorithm can be tested in the following manner. The implementation begins operation. The test instrument waits for the arrival of the first RTCP packet. When it arrives, the test instrument notes the time and then immediately sends 100 RTCP RR packets to the implementation, each with a different SSRC and SDES CNAME. The test instrument should ensure that each RTCP packet is of the same length. The instrument should then wait until the next RTCP packet is received from the implementation, and the time of such reception is noted. Without reconsideration, the next RTCP packet will arrive within a short period of time. With reconsideration, transmission of this packet will be delayed. The earliest it can arrive depends on the RTCP session bandwidth, receiver fraction, and average RTCP packet size. The RTP implementation should be using the exponential J.Rosenberg,H.Schulzrinne [Page 4] Internet Draft RTCP Test June 25, 1999 averaging algorithm defined in the specification to compute the aver- age RTCP packet size. Since this is dominated by the received packets (the implementation has only sent one itself), the average will be roughly equal to the length of the RTCP packets sent by the test instrument. Therefore, the minimum amount of time between the first and second RTCP packets from the implementation is: T > 101 * S / ( B * Fr * (e-1.5) * 2 ) Where S is the size of the RTCP packets sent by the test instrument, B is the RTCP bandwidth (normally five percent of the session bandwidth), Fr is the fraction of RTCP bandwidth allocated to receivers (normally 75 percent), and e is the natural exponent. Without reconsideration, this minimum interval Te would be much smaller: Te > MAX( [ S / ( B * Fr * (e-1.5) * 2 ) ] , [ 2.5 / (e-1.5) ] ) B should be chosen sufficiently small so that T is around 60 seconds. Reasonable choices for these parameters are B = 950 bits per second, and S = 1024 bits. An implementation passes this test if the interval between packets is not less than T above, and not more than 3 times T. The test may be repeated for the case when the RTP implementation is a sender. This is accomplished by having the implementation send RTP packets at least once a second. In this case, the interval between the first and second RTCP packets should be no less than: T > S / ( B * Fs * (e-1.5) * 2 ) Where Fs is the fraction of RTCP bandwidth allocated to senders, usu- ally 25%. Note that this value of T is significantly smaller than the interval for receivers. 5 Steady State Behavior J.Rosenberg,H.Schulzrinne [Page 5] Internet Draft RTCP Test June 25, 1999 In addition to the basic behavior in section 3, an implementation should correctly implement a number of other, slightly more advanced features: o scale the RTCP interval with the group size; o correctly divide bandwidth between senders and receivers; o correctly compute the RTCP interval when the user is a sender The following tests are meant to test these features. 5.1 Scaling the Interval The implementation begins operation as a receiver. The test instru- ment waits for the first RTCP packet from the implementation. When it arrives, the test instrument notes the time, and immediately sends 50 RTCP RR packets and 50 RTCP SR packets to the implementation, each with a different SSRC and SDES CNAME. The test instrument then sends 50 RTP packets, using the 50 SSRC from the RTCP SR packets. The test instrument should ensure that each RTCP packet is of the same length. The instrument should then wait until the next RTCP packet is received from the implementation, and the time of such reception is noted. The difference between the reception of the RTCP packet and the reception of the previous is computed and stored. In addition, after every RTCP packet reception, the 100 RTCP and 50 RTP packets are retransmitted by the test instrument. This ensures that the sender and member status of the 100 users does not time out. The test instrument should collect the interval measurements figures for at least 100 RTCP packets. With 50 senders, the implementation should not try to divide the RTCP bandwidth between senders and receivers, but rather group all users together and divide the RTCP bandwidth equally. The test is deemed successful if the average RTCP interval is within 5% of: T = 101* S/B Where S is the size of the RTCP packets sent by the test instrument, and B is the RTCP bandwidth. B should be chosen sufficiently small so that the value of T is on the order of tens of seconds or more. Rea- sonable values are S=1024 bits and B=3.4 kb/s. 5.2 Compensating for Senders J.Rosenberg,H.Schulzrinne [Page 6] Internet Draft RTCP Test June 25, 1999 The test of section 5.1 is repeated. However, the test instrument sends 10 RTP packets instead of 50, and 10 RTCP SR and 90 RTCP RR instead of 50 of each. In addition, the implementation is made to send at least one RTP packet between transmission of every one of its own RTCP packets. In this case, the average RTCP interval should be within 5% of: T = 11 * S / (B * Fs) Where S is the size of the RTCP packets sent by the test instrument, B is the RTCP bandwidth, and Fs is the fraction of RTCP banwidth allocated for senders (normally 25%). The values for B and S should be chosen small enough so that T is on the order of tens of seconds. Reasonable choices are S=1024 bits and B=1.5 kb/s. 6 Reverse Reconsideration The reverse reconsideration algorithm is effectively the opposite of the normal reconsideration algorithm. It causes the RTCP interval to be reduced more rapidly in response to decreases in the group member- ship. This is advantageous in that it keeps the RTCP information as fresh as possible, and helps avoids some premature timeout problems. These tests verify the basic correctness of reverse reconsideration. 6.1 Test I In this test, the implementation joins the session as a receiver. As soon as it sends its first RTCP packet, the test instrument sends 100 RTCP RR packets, each of the same length S, and a different SDES CNAME and SSRC in each. It then waits for the implementation to send another RTCP packet. Once it does, the test instrument sends 100 BYE packets, each one containing a different SSRC, but matching an SSRC from one of the initial RTCP packets. Each BYE should also be the same size as the RTCP packets sent by the test instrument. This is easily accomplished by using a BYE reason to pad out the length. The time of the next RTCP packet from the implementation is then noted. The delay T between this (the third RTCP packet) and the previous should be no more than: T < 3 * S / (B * Fr * (e-1.5) * 2) J.Rosenberg,H.Schulzrinne [Page 7] Internet Draft RTCP Test June 25, 1999 Where S is the size of the RTCP and BYE packets sent by the test instrument, B is the RTCP bandwidth, Fr is the fraction of the RTCP bandwidth allocated to receivers, and e is the natural exponent. B should be chosen such that T is on the order of tens of seconds. A reasonable choice is S=1024 bits and B=168 bits per second. This test demonstrates basic correctness of implementation. An imple- mentation without reverse reconsideration will not send its next RTCP packet for nearly 100 times as long as the above amount. 6.2 Test II In this test, the implementation joins the session as a receiver. As soon as it sends its first RTCP packet, the test instrument sends 100 RTCP RR packets, each of the same length S, followed by 100 BYE pack- ets, also of length S. Each RTCP packet carries a different SDES CNAME and SSRC, and is matched with precisely one BYE packet with the same SSRC. This will cause the implementation to see a rapid increase and then rapid drop in group membership. The test is deemed succesful if the next RTCP packet shows up T seconds after the first, and T is within: 2.5 / (e-1.5) < T < 7.5 / (e-1.5) This tests correctness of the maintenance of the pmembers variable. An incorrect implementation might try to execute reverse reconsidera- tion every time a BYE is received, as opposed to only when the group membership drops below pmembers. If an implementation did this, it would end up sending an RTCP packet immediately after receiving the stream of BYE's. For this test to work, B must be chosen to be a large value, around 1Mb/s. 7 BYE Reconsideration The BYE reconsideration algorithm works in much the same fashion as regular reconsideration, except applied to BYE packets. When a user leaves the group, instead of sending a BYE immediately, it may delay transmission of its BYE packet if others are sending BYE's. The test for correctness of this algorithm is as follows. The RTP implementation joins the group as a receiver. The test instrument waits for the first RTCP packet. When the test instrument receives this packet, the test instrument immediately sends 100 RTCP RR pack- ets, each of the same length S, and each containing a different SSRC J.Rosenberg,H.Schulzrinne [Page 8] Internet Draft RTCP Test June 25, 1999 and SDES CNAME. Once the test instrument receives the next RTCP packet from the implementation, the RTP implementation is made to leave the RTP session, and this information is conveyed to the test instrument through some non-RTP means. The test instrument then sends 100 BYE packets, each with a different SSRC, and each matching an SSRC from a previously transmitted RTCP packet. Each of these BYE packets is also of size S. Immediately following the BYE packets, the test instrument sends 100 RTCP RR packets, using the same SSRC/CNAMEs as the original 100 RTCP packets. The test is deemed successful if the implementation either never sends a BYE, or if it does, the BYE is not received by the test instrument earlier than T seconds after the implementation left the session, where T is: T = 100 * S / ( 2 * (e-1.5) * B * Fr) S is the size of the RTCP and BYE packets, e is the natural exponent, B is the RTCP bandwidth, and Fr is the RTCP bandwidth fraction for receivers. S and B should be chosen so that T is on the order of 50 seconds. A reasonable choice is S=1024 bits and B=1.1 kb/s. The transmission of the RTCP packets is meant to verify that the implementation is ignoring non-BYE RTCP packets once it decides to leave the group. 8 Timing out members Active RTP participants are supposed to send periodic RTCP packets. When a participant leaves the session, they may send a BYE, however this is not required. Furthermore, BYE reconsideration may cause a BYE to never be sent. As a result, participants must time out other participants who have not sent an RTCP packet in a long time. Accord- ing to the specification, participants who have not sent an RTCP packet in the last 5 intervals are timed out. This test verifies that these timeouts are being performed correctly. The RTP implementation joins a session as a receiver. The test instrument waits for the first RTCP packet from the implementation. Once it arrives, the test instrument sends 100 RTCP RR packets, each with a different SDES and SSRC, and notes the time. This will cause the implementation to believe that there are now 101 group partici- pants, causing it to increase its RTCP interval. The test instrument continues to monitor the RTCP packets from the implementation. As each RTCP packet is received, the time of its reception is noted, and J.Rosenberg,H.Schulzrinne [Page 9] Internet Draft RTCP Test June 25, 1999 the interval between RTCP packets is stored. The 100 participants spoofed by the test instrument should eventually time out at the RTP implementation. This should cause the RTCP interval to be reduced to its minimum. The test is deemed successful if the interval between RTCP packets after the first is no less than: Ti > 101 * S / ( 2 * (e-1.5) * B * Fr) and this minimum interval is sustained no later than Td seconds after the transmission of the 100 RR's, where Td is: Td = 7 * 101 * S / ( B * Fr ) and the interval between RTCP packets after this point is no less than: Tf > 2.5 / (e-1.5) For this test to work, B and S must be chosen so Ti is on the order of minutes. Recommended values are S = 1024 bits and B = 1.9 kbps. 9 SSRC Randomization According to the RTP specification, SSRC's are supposed to be chosen randomly and uniformly over a 32 bit space. This randomization is beneficial for several reasons: o It reduces the probability of collisions in large groups. o It simplifies the process of group sampling [4], which depends on the uniform distribution of SSRC's across the 32 bit space. Unfortunately, verifying that a random number has 32 bits of uniform randomness requires a large number of samples. The procedure below gives only a rough validation to the randomness used for generating the SSRC. J.Rosenberg,H.Schulzrinne [Page 10] Internet Draft RTCP Test June 25, 1999 The test runs as follows. The RTP implementation joins the group as a receiver. The test instrument waits for the first RTCP packet. It notes the SSRC in this RTCP packet. The test is repeated 2500 times, resulting in a collection of 2500 SSRC. The are then placed into 25 bins. An SSRC with value X is placed into bin FLOOR(X/(2**32 / 25)). The idea is to break the 32 bit space into 25 regions, and compute the number of SSRC in each region. Ideally, there should be 40 SSRC in each bin. Of course, the actual number in each bin is a random variable whose expectation is 40. With 2500 SSRC, the coefficient of variation of the number of SSRC in a bin is .1, which means the number should be between 36 and 44. The test is thus deemed successful if each bin has no less than 30 and no more than 50 SSRC. Running this test may require substantial amounts of time, particu- larly if there is no automated way to have the implementation join the session. In such a case, the test can be run fewer times. With 26 tests, half of the SSRC should be less than 2**31, and the other half higher. The cofficient of variation in this case is .2, so the test is successful if there are more than 8 SSRC less than 2**31, and less than 26. In general, if the SSRC is collected N times, and there are B bins, the coefficient of variation of the number of SSRC in each bin is given by: coeff = SQRT( (B-1)/N ) 10 SSRC Collisions RTP has a provision for SSRC collisions. These collisions occur when two different session participants choose the same SSRC. In this case, both participants are supposed to send a BYE, leave the ses- sion, and rejoin with a different SSRC, but the same CNAME. The pur- pose of this test is to verify that this function is present in the implementation. The test is straightforward. The RTP implementation is made to join the multicast group as a receiver. The test instrument waits for the first RTCP packet. Once it arrives, the test instrument notes the CNAME and SSRC from the RTCP packet. The test instrument then gen- erates an RTCP receiver report. This receiver report contains an SDES J.Rosenberg,H.Schulzrinne [Page 11] Internet Draft RTCP Test June 25, 1999 chunk with an SSRC matching that of the RTP implementation, but with a different CNAME. At this point, the implementation should send a BYE RTCP packet (containing an SDES chunk with the old SSRC and CNAME), and then rejoin, causing it to send a receiver report con- taining an SDES chunk, but with a new SSRC and the same CNAME. The test is deemed succesful if the RTP implementation sends the RTCP BYE and RTCP RR as described above within one minute of receiving the colliding RR from the test instrument. 11 Rapid SR's In the draft version of RTP, the minimum interval for RTCP packets can be reduced for large session bandwdiths. The reduction applies to senders only. The recommended algorithm for computing this minimum interval is 360 divided by the RTP session bandwidth, in kbps. For bandwidths larger than 72 kbps, this interval is less than 5 seconds. This test verifies the ability of an implementation to use a lower RTCP minimum interval when it is a sender in a high bandwidth ses- sion. The test can only be run on implementations that support this reduction, since it is optional. The RTP implementation is configured to join the session as a sender. The session is configured to use 360 kbps. If the recommended algo- rithm for computing the reduced minimum interval is used, the result is a 1 second interval. If the RTP implementation uses a different algorithm, the session bandwidth should be set in such a way to cause the reduced minimum interval to be 1 second. Once joining the session, the RTP implementation should begin to send both RTP and RTCP packets. The interval between RTCP packets is meas- ured and stored until 100 intervals have been collected. The test is deemed successful if the smallest interval is no less than 1/2 a second, and the largest interval is no more than 1.5 seconds. The average should be close to 1 second. 12 Changes since -00 The following changes have been made since version -00 of this specification: o The introduction was rewritten to reflect the new broader scope of the tests. Previously, the tests verified only the features new in draft standard. Now, the tests include some features present in the proposed standard. J.Rosenberg,H.Schulzrinne [Page 12] Internet Draft RTCP Test June 25, 1999 o A test for verifying uniform distribution of SSRC was added. o A test for verifying reduction of the minimum interval for high bandwidth sessions was added. o A test for verifying SSRC collision detection was added. o A test for verifying timeouts was added. 13 Author's Addresses Jonathan Rosenberg Lucent Technologies, Bell Laboratories 101 Crawfords Corner Rd. Holmdel, NJ 07733 Rm. 4C-526 email: jdrosen@bell-labs.com Henning Schulzrinne Columbia University M/S 0401 1214 Amsterdam Ave. New York, NY 10027-7003 email: schulzrinne@cs.columbia.edu 14 Bibliography [1] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, "RTP: a transport protocol for real-time applications," Internet Draft, Internet Engineering Task Force, Mar. 1999. Work in progress. [2] J. Rosenberg and H. Schulzrinne, "Timer reconsideration for enhanced RTP scalability," in IEEE Infocom , (San Francisco, Califor- nia), March/April 1998. [3] J. Rosenberg and H. Schulzrinne, "New results in RTP scalabil- ity," Internet Draft, Internet Engineering Task Force, Dec. 1997. Work in progress. [4] J. Rosenberg and H. Schulzrinne, "Sampling of the group member- ship in RTP," Internet Draft, Internet Engineering Task Force, Apr. 1999. Work in progress. J.Rosenberg,H.Schulzrinne [Page 13] Internet Draft RTCP Test June 25, 1999 Full Copyright Statement Copyright (c) The Internet Society (1999). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this docu- ment itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of develop- ing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MER- CHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. J.Rosenberg,H.Schulzrinne [Page 14]