From 7d5c8eb943f92a94b420568778d61e0bd6f7df43 Mon Sep 17 00:00:00 2001 From: claude-bot Date: Mon, 13 Jul 2026 12:40:01 +0000 Subject: Import xiph/opus @ 034c1b61a250457649d788bbf983b3f0fb63f02e Snapshot for re3/reVC vendoring, per @lzcnt. Source: https://github.com/xiph/opus (034c1b61a250457649d788bbf983b3f0fb63f02e). --- doc/draft-ietf-payload-rtp-opus.xml | 960 ++++++++++++++++++++++++++++++++++++ 1 file changed, 960 insertions(+) create mode 100644 doc/draft-ietf-payload-rtp-opus.xml (limited to 'doc/draft-ietf-payload-rtp-opus.xml') diff --git a/doc/draft-ietf-payload-rtp-opus.xml b/doc/draft-ietf-payload-rtp-opus.xml new file mode 100644 index 0000000..c4eb210 --- /dev/null +++ b/doc/draft-ietf-payload-rtp-opus.xml @@ -0,0 +1,960 @@ + + + + + + + + + + + + + + + + + + + + + + ]> + + + + + + + + + + + + + + + + + + RTP Payload Format for the Opus Speech and Audio Codec + + + +
+ jspittka@gmail.com +
+ + + + vocTone +
+ + + + + + + + koenvos74@gmail.com +
+ + + + Mozilla +
+ + 331 E. Evelyn Avenue + Mountain View + CA + 94041 + USA + + jmvalin@jmvalin.ca +
+ + + + + + + This document defines the Real-time Transport Protocol (RTP) payload + format for packetization of Opus encoded + speech and audio data necessary to integrate the codec in the + most compatible way. It also provides an applicability statement + for the use of Opus over RTP. Further, it describes media type registrations + for the RTP payload format. + + + + + +
+ + Opus is a speech and audio codec developed within the + IETF Internet Wideband Audio Codec working group. The codec + has a very low algorithmic delay and it + is highly scalable in terms of audio bandwidth, bitrate, and + complexity. Further, it provides different modes to efficiently encode speech signals + as well as music signals, thus making it the codec of choice for + various applications using the Internet or similar networks. + + + This document defines the Real-time Transport Protocol (RTP) + payload format for packetization + of Opus encoded speech and audio data necessary to + integrate Opus in the + most compatible way. It also provides an applicability statement + for the use of Opus over RTP. + Further, it describes media type registrations for + the RTP payload format. + +
+ +
+ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in . + + + The range of audio frequecies being coded + Constant bitrate + Central Processing Unit + Discontinuous transmission + Forward error correction + Internet Protocol + Speech or audio samples (per channel) + Session Description Protocol + Variable bitrate + + + + Throughout this document, we refer to the following definitions: + + + Abbreviation + Name + Audio Bandwidth (Hz) + Sampling Rate (Hz) + NB + Narrowband + 0 - 4000 + 8000 + + MB + Mediumband + 0 - 6000 + 12000 + + WB + Wideband + 0 - 8000 + 16000 + + SWB + Super-wideband + 0 - 12000 + 24000 + + FB + Fullband + 0 - 20000 + 48000 + + + Audio bandwidth naming + + +
+ +
+ + Opus encodes speech + signals as well as general audio signals. Two different modes can be + chosen, a voice mode or an audio mode, to allow the most efficient coding + depending on the type of the input signal, the sampling frequency of the + input signal, and the intended application. + + + + The voice mode allows efficient encoding of voice signals at lower bit + rates while the audio mode is optimized for general audio signals at medium and + higher bitrates. + + + + Opus is highly scalable in terms of audio + bandwidth, bitrate, and complexity. Further, Opus allows + transmitting stereo signals with in-band signaling in the bit-stream. + + +
+ + Opus supports bitrates from 6 kb/s to 510 kb/s. + The bitrate can be changed dynamically within that range. + All + other parameters being + equal, higher bitrates result in higher audio quality. + +
+ + For a frame size of + 20 ms, these + are the bitrate "sweet spots" for Opus in various configurations: + + + 8-12 kb/s for NB speech, + 16-20 kb/s for WB speech, + 28-40 kb/s for FB speech, + 48-64 kb/s for FB mono music, and + 64-128 kb/s for FB stereo music. + + +
+
+ + For the same average bitrate, variable bitrate (VBR) can achieve higher audio quality + than constant bitrate (CBR). For the majority of voice transmission applications, VBR + is the best choice. One reason for choosing CBR is the potential + information leak that might occur when encrypting the + compressed stream. See for guidelines on when VBR is + appropriate for encrypted audio communications. In the case where an existing + VBR stream needs to be converted to CBR for security reasons, then the Opus padding + mechanism described in is the RECOMMENDED way to achieve padding + because the RTP padding bit is unencrypted. + + + The bitrate can be adjusted at any point in time. To avoid congestion, + the average bitrate SHOULD NOT exceed the available + network bandwidth. If no target bitrate is specified, the bitrates specified in + are RECOMMENDED. + + +
+ +
+ + + Opus can, as described in , + be operated with a variable bitrate. In that case, the encoder will + automatically reduce the bitrate for certain input signals, like periods + of silence. When using continuous transmission, it will reduce the + bitrate when the characteristics of the input signal permit, but + will never interrupt the transmission to the receiver. Therefore, the + received signal will maintain the same high level of audio quality over the + full duration of a transmission while minimizing the average bit + rate over time. + + + + In cases where the bitrate of Opus needs to be reduced even + further or in cases where only constant bitrate is available, + the Opus encoder can use discontinuous + transmission (DTX), where parts of the encoded signal that + correspond to periods of silence in the input speech or audio signal + are not transmitted to the receiver. A receiver can distinguish + between DTX and packet loss by looking for gaps in the sequence + number, as described by Section 4.1 + of . + + + + On the receiving side, the non-transmitted parts will be handled by a + frame loss concealment unit in the Opus decoder which generates a + comfort noise signal to replace the non transmitted parts of the + speech or audio signal. Use of Comfort + Noise (CN) with Opus is discouraged. + The transmitter MUST drop whole frames only, + based on the size of the last transmitted frame, + to ensure successive RTP timestamps differ by a multiple of 120 and + to allow the receiver to use whole frames for concealment. + + + + DTX can be used with both variable and constant bitrate. + It will have a slightly lower speech or audio + quality than continuous transmission. Therefore, using continuous + transmission is RECOMMENDED unless constraints on available network bandwidth + are severe. + + +
+ +
+ +
+ + + Complexity of the encoder can be scaled to optimize for CPU resources in real-time, mostly as + a trade-off between audio quality and bitrate. Also, different modes of Opus have different complexity. + + +
+ +
+ + + The voice mode of Opus allows for embedding "in-band" forward error correction (FEC) + data into the Opus bit stream. This FEC scheme adds + redundant information about the previous packet (N-1) to the current + output packet N. For + each frame, the encoder decides whether to use FEC based on (1) an + externally-provided estimate of the channel's packet loss rate; (2) an + externally-provided estimate of the channel's capacity; (3) the + sensitivity of the audio or speech signal to packet loss; (4) whether + the receiving decoder has indicated it can take advantage of "in-band" + FEC information. The decision to send "in-band" FEC information is + entirely controlled by the encoder and therefore no special precautions + for the payload have to be taken. + + + + On the receiving side, the decoder can take advantage of this + additional information when it loses a packet and the next packet + is available. In order to use the FEC data, the jitter buffer needs + to provide access to payloads with the FEC data. + Instead of performing loss concealment for a missing packet, the + receiver can then configure its decoder to decode the FEC data from the next packet. + + + + Any compliant Opus decoder is capable of ignoring + FEC information when it is not needed, so encoding with FEC cannot cause + interoperability problems. + However, if FEC cannot be used on the receiving side, then FEC + SHOULD NOT be used, as it leads to an inefficient usage of network + resources. Decoder support for FEC SHOULD be indicated at the time a + session is set up. + + +
+ +
+ + + Opus allows for transmission of stereo audio signals. This operation + is signaled in-band in the Opus bit-stream and no special arrangement + is needed in the payload format. An + Opus decoder is capable of handling a stereo encoding, but an + application might only be capable of consuming a single audio + channel. + + + If a decoder cannot take advantage of the benefits of a stereo signal + this SHOULD be indicated at the time a session is set up. In that case + the sending side SHOULD NOT send stereo signals as it leads to an + inefficient usage of network resources. + + +
+ +
+ +
+ The payload format for Opus consists of the RTP header and Opus payload + data. +
+ The format of the RTP header is specified in . + The use of the fields of the RTP header by the Opus payload format is + consistent with that specification. + + The payload length of Opus is an integer number of octets and + therefore no padding is necessary. The payload MAY be padded by an + integer number of octets according to , + although the Opus internal padding is preferred. + + The timestamp, sequence number, and marker bit (M) of the RTP header + are used in accordance with Section 4.1 + of . + + The RTP payload type for Opus is to be assigned dynamically. + + The receiving side MUST be prepared to receive duplicate RTP + packets. The receiver MUST provide at most one of those payloads to the + Opus decoder for decoding, and MUST discard the others. + + Opus supports 5 different audio bandwidths, which can be adjusted during + a stream. + The RTP timestamp is incremented with a 48000 Hz clock rate + for all modes of Opus and all sampling rates. + The unit + for the timestamp is samples per single (mono) channel. The RTP timestamp corresponds to the + sample time of the first encoded sample in the encoded frame. + For data encoded with sampling rates other than 48000 Hz, + the sampling rate has to be adjusted to 48000 Hz. + +
+ +
+ + The Opus encoder can output encoded frames representing 2.5, 5, 10, 20, + 40, or 60 ms of speech or audio data. Further, an arbitrary number of frames can be + combined into a packet, up to a maximum packet duration representing + 120 ms of speech or audio data. The grouping of one or more Opus + frames into a single Opus packet is defined in Section 3 of + . An RTP payload MUST contain exactly one + Opus packet as defined by that document. + + + shows the structure combined with the RTP header. + +
+ + + +
+ + + shows supported frame sizes in + milliseconds of encoded speech or audio data for the speech and audio modes + (Mode) and sampling rates (fs) of Opus and shows how the timestamp is + incremented for packetization (ts incr). If the Opus encoder + outputs multiple encoded frames into a single packet, the timestamp + increment is the sum of the increments for the individual frames. + + + + Mode + fs + 2.5 + 5 + 10 + 20 + 40 + 60 + ts incr + all + 120 + 240 + 480 + 960 + 1920 + 2880 + voice + NB/MB/WB/SWB/FB + x + x + o + o + o + o + audio + NB/WB/SWB/FB + o + o + o + o + x + x + + +
+ +
+ +
+ + The target bitrate of Opus can be adjusted at any point in time, thus + allowing efficient congestion control. Furthermore, the amount + of encoded speech or audio data encoded in a + single packet can be used for congestion control, since the transmission + rate is inversely proportional to the packet duration. A lower packet + transmission rate reduces the amount of header overhead, but at the same + time increases latency and loss sensitivity, so it ought to be used with + care. + + Since UDP does not provide congestion control, applications that use + RTP over UDP SHOULD implement their own congestion control above the + UDP layer . Work in the rmcat working group + describes the + interactions and conceptual interfaces necessary between the application + components that relate to congestion control, including the RTP layer, + the higher-level media codec control layer, and the lower-level + transport interface, as well as components dedicated to congestion + control functions. +
+ +
+ One media subtype (audio/opus) has been defined and registered as + described in the following section. + +
+ Media type registration is done according to and . + + Type name: audio + Subtype name: opus + + Required parameters: + + the RTP timestamp is incremented with a + 48000 Hz clock rate for all modes of Opus and all sampling + rates. For data encoded with sampling rates other than 48000 Hz, + the sampling rate has to be adjusted to 48000 Hz. + + + + Optional parameters: + + + + a hint about the maximum output sampling rate that the receiver is + capable of rendering in Hz. + The decoder MUST be capable of decoding + any audio bandwidth but due to hardware limitations only signals + up to the specified sampling rate can be played back. Sending signals + with higher audio bandwidth results in higher than necessary network + usage and encoding complexity, so an encoder SHOULD NOT encode + frequencies above the audio bandwidth specified by maxplaybackrate. + This parameter can take any value between 8000 and 48000, although + commonly the value will match one of the Opus bandwidths + (). + By default, the receiver is assumed to have no limitations, i.e. 48000. + + + + + a hint about the maximum input sampling rate that the sender is likely to produce. + This is not a guarantee that the sender will never send any higher bandwidth + (e.g. it could send a pre-recorded prompt that uses a higher bandwidth), but it + indicates to the receiver that frequencies above this maximum can safely be discarded. + This parameter is useful to avoid wasting receiver resources by operating the audio + processing pipeline (e.g. echo cancellation) at a higher rate than necessary. + This parameter can take any value between 8000 and 48000, although + commonly the value will match one of the Opus bandwidths + (). + By default, the sender is assumed to have no limitations, i.e. 48000. + + + + the maximum duration of media represented + by a packet (according to Section 6 of + ) that a decoder wants to receive, in + milliseconds rounded up to the next full integer value. + Possible values are 3, 5, 10, 20, 40, 60, or an arbitrary + multiple of an Opus frame size rounded up to the next full integer + value, up to a maximum value of 120, as + defined in . If no value is + specified, the default is 120. + + + the preferred duration of media represented + by a packet (according to Section 6 of + ) that a decoder wants to receive, in + milliseconds rounded up to the next full integer value. + Possible values are 3, 5, 10, 20, 40, 60, or an arbitrary + multiple of an Opus frame size rounded up to the next full integer + value, up to a maximum value of 120, as defined in . If no value is + specified, the default is 20. + + + specifies the maximum average + receive bitrate of a session in bits per second (b/s). The actual + value of the bitrate can vary, as it is dependent on the + characteristics of the media in a packet. Note that the maximum + average bitrate MAY be modified dynamically during a session. Any + positive integer is allowed, but values outside the range + 6000 to 510000 SHOULD be ignored. If no value is specified, the + maximum value specified in + for the corresponding mode of Opus and corresponding maxplaybackrate + is the default. + + + specifies whether the decoder prefers receiving stereo or mono signals. + Possible values are 1 and 0 where 1 specifies that stereo signals are preferred, + and 0 specifies that only mono signals are preferred. + Independent of the stereo parameter every receiver MUST be able to receive and + decode stereo signals but sending stereo signals to a receiver that signaled a + preference for mono signals may result in higher than necessary network + utilization and encoding complexity. If no value is specified, + the default is 0 (mono). + + + + specifies whether the sender is likely to produce stereo audio. + Possible values are 1 and 0, where 1 specifies that stereo signals are likely to + be sent, and 0 specifies that the sender will likely only send mono. + This is not a guarantee that the sender will never send stereo audio + (e.g. it could send a pre-recorded prompt that uses stereo), but it + indicates to the receiver that the received signal can be safely downmixed to mono. + This parameter is useful to avoid wasting receiver resources by operating the audio + processing pipeline (e.g. echo cancellation) in stereo when not necessary. + If no value is specified, the default is 0 + (mono). + + + + specifies if the decoder prefers the use of a constant bitrate versus + variable bitrate. Possible values are 1 and 0, where 1 specifies constant + bitrate and 0 specifies variable bitrate. If no value is specified, + the default is 0 (vbr). When cbr is 1, the maximum average bitrate can still + change, e.g. to adapt to changing network conditions. + + + specifies that the decoder has the capability to + take advantage of the Opus in-band FEC. Possible values are 1 and 0. + Providing 0 when FEC cannot be used on the receiving side is + RECOMMENDED. If no + value is specified, useinbandfec is assumed to be 0. + This parameter is only a preference and the receiver MUST be able to process + packets that include FEC information, even if it means the FEC part is discarded. + + + specifies if the decoder prefers the use of + DTX. Possible values are 1 and 0. If no value is specified, the + default is 0. + + + Encoding considerations: + + The Opus media type is framed and consists of binary data according + to Section 4.8 in . + + + Security considerations: + + See of this document. + + + Interoperability considerations: none + Published specification: RFC [XXXX] + Note to the RFC Editor: Replace [XXXX] with the number of the published + RFC. + + Applications that use this media type: + + Any application that requires the transport of + speech or audio data can use this media type. Some examples are, + but not limited to, audio and video conferencing, Voice over IP, + media streaming. + + + Fragment identifier considerations: N/A + + Person & email address to contact for further information: + + SILK Support silksupport@skype.net + Jean-Marc Valin jmvalin@jmvalin.ca + + + Intended usage: COMMON + + Restrictions on usage: + + + For transfer over RTP, the RTP payload format ( of this document) SHALL be + used. + + + Author: + + Julian Spittka jspittka@gmail.com + Koen Vos koenvos74@gmail.com + Jean-Marc Valin jmvalin@jmvalin.ca + + + Change controller: IETF Payload Working Group delegated from the IESG +
+
+ +
+ The information described in the media type specification has a + specific mapping to fields in the Session Description Protocol (SDP) + , which is commonly used to describe RTP + sessions. When SDP is used to specify sessions employing Opus, + the mapping is as follows: + + + + The media type ("audio") goes in SDP "m=" as the media name. + + The media subtype ("opus") goes in SDP "a=rtpmap" as the encoding + name. The RTP clock rate in "a=rtpmap" MUST be 48000 and the number of + channels MUST be 2. + + The OPTIONAL media type parameters "ptime" and "maxptime" are + mapped to "a=ptime" and "a=maxptime" attributes, respectively, in the + SDP. + + The OPTIONAL media type parameters "maxaveragebitrate", + "maxplaybackrate", "stereo", "cbr", "useinbandfec", and + "usedtx", when present, MUST be included in the "a=fmtp" attribute + in the SDP, expressed as a media type string in the form of a + semicolon-separated list of parameter=value pairs (e.g., + maxplaybackrate=48000). They MUST NOT be specified in an + SSRC-specific "fmtp" source-level attribute (as defined in + Section 6.3 of ). + + The OPTIONAL media type parameters "sprop-maxcapturerate", + and "sprop-stereo" MAY be mapped to the "a=fmtp" SDP attribute by + copying them directly from the media type parameter string as part + of the semicolon-separated list of parameter=value pairs (e.g., + sprop-stereo=1). These same OPTIONAL media type parameters MAY also + be specified using an SSRC-specific "fmtp" source-level attribute + as described in Section 6.3 of . + They MAY be specified in both places, in which case the parameter + in the source-level attribute overrides the one found on the + "a=fmtp" line. The value of any parameter which is not specified in + a source-level source attribute MUST be taken from the "a=fmtp" + line, if it is present there. + + + + + Below are some examples of SDP session descriptions for Opus: + + Example 1: Standard mono session with 48000 Hz clock rate +
+ + + +
+ + + Example 2: 16000 Hz clock rate, maximum packet size of 40 ms, + recommended packet size of 40 ms, maximum average bitrate of 20000 bps, + prefers to receive stereo but only plans to send mono, FEC is desired, + DTX is not desired + +
+ + + +
+ + Example 3: Two-way full-band stereo preferred + +
+ + + +
+ + +
+ + When using the offer-answer procedure described in to negotiate the use of Opus, the following + considerations apply: + + + + Opus supports several clock rates. For signaling purposes only + the highest, i.e. 48000, is used. The actual clock rate of the + corresponding media is signaled inside the payload and is not + restricted by this payload format description. The decoder MUST be + capable of decoding every received clock rate. An example + is shown below: + +
+ + + +
+ + + The "ptime" and "maxptime" parameters are unidirectional + receive-only parameters and typically will not compromise + interoperability; however, some values might cause application + performance to suffer. defines the SDP offer-answer handling of the + "ptime" parameter. The "maxptime" parameter MUST be handled in the + same way. + + + The "maxplaybackrate" parameter is a unidirectional receive-only + parameter that reflects limitations of the local receiver. When + sending to a single destination, a sender MUST NOT use an audio + bandwidth higher than necessary to make full use of audio sampled at + a sampling rate of "maxplaybackrate". Gateways or senders that + are sending the same encoded audio to multiple destinations + SHOULD NOT use an audio bandwidth higher than necessary to + represent audio sampled at "maxplaybackrate", as this would lead + to inefficient use of network resources. + The "maxplaybackrate" parameter does not + affect interoperability. Also, this parameter SHOULD NOT be used + to adjust the audio bandwidth as a function of the bitrate, as this + is the responsibility of the Opus encoder implementation. + + + The "maxaveragebitrate" parameter is a unidirectional receive-only + parameter that reflects limitations of the local receiver. The sender + of the other side MUST NOT send with an average bitrate higher than + "maxaveragebitrate" as it might overload the network and/or + receiver. The "maxaveragebitrate" parameter typically will not + compromise interoperability; however, some values might cause + application performance to suffer, and ought to be set with + care. + + The "sprop-maxcapturerate" and "sprop-stereo" parameters are + unidirectional sender-only parameters that reflect limitations of + the sender side. + They allow the receiver to set up a reduced-complexity audio + processing pipeline if the sender is not planning to use the full + range of Opus's capabilities. + Neither "sprop-maxcapturerate" nor "sprop-stereo" affect + interoperability and the receiver MUST be capable of receiving any signal. + + + + The "stereo" parameter is a unidirectional receive-only + parameter. When sending to a single destination, a sender MUST + NOT use stereo when "stereo" is 0. Gateways or senders that are + sending the same encoded audio to multiple destinations SHOULD + NOT use stereo when "stereo" is 0, as this would lead to + inefficient use of network resources. The "stereo" parameter does + not affect interoperability. + + + + The "cbr" parameter is a unidirectional receive-only + parameter. + + + The "useinbandfec" parameter is a unidirectional receive-only + parameter. + + The "usedtx" parameter is a unidirectional receive-only + parameter. + + Any unknown parameter in an offer MUST be ignored by the receiver + and MUST be removed from the answer. + + + + + The Opus parameters in an SDP Offer/Answer exchange are completely + orthogonal, and there is no relationship between the SDP Offer and + the Answer. + +
+ +
+ + For declarative use of SDP such as in Session Announcement Protocol + (SAP), , and RTSP, , for + Opus, the following needs to be considered: + + + + The values for "maxptime", "ptime", "maxplaybackrate", and + "maxaveragebitrate" ought to be selected carefully to ensure that a + reasonable performance can be achieved for the participants of a session. + + + The values for "maxptime", "ptime", and of the payload + format configuration are recommendations by the decoding side to ensure + the best performance for the decoder. + + + All other parameters of the payload format configuration are declarative + and a participant MUST use the configurations that are provided for + the session. More than one configuration can be provided if necessary + by declaring multiple RTP payload types; however, the number of types + ought to be kept small. + +
+
+ +
+ + Use of variable bitrate (VBR) is subject to the security considerations in + . + + RTP packets using the payload format defined in this specification + are subject to the security considerations discussed in the RTP + specification , and in any applicable RTP profile such as + RTP/AVP , RTP/AVPF , + RTP/SAVP or RTP/SAVPF . + However, as "Securing the RTP Protocol Framework: + Why RTP Does Not Mandate a Single Media Security Solution" + discusses, it is not an RTP payload + format's responsibility to discuss or mandate what solutions are used + to meet the basic security goals like confidentiality, integrity and + source authenticity for RTP in general. This responsibility lays on + anyone using RTP in an application. They can find guidance on + available security mechanisms and important considerations in Options + for Securing RTP Sessions [I-D.ietf-avtcore-rtp-security-options]. + Applications SHOULD use one or more appropriate strong security + mechanisms. + + This payload format and the Opus encoding do not exhibit any + significant non-uniformity in the receiver-end computational load and thus + are unlikely to pose a denial-of-service threat due to the receipt of + pathological datagrams. +
+ +
+ Many people have made useful comments and suggestions contributing to this document. + In particular, we would like to thank + Tina le Grand, Cullen Jennings, Jonathan Lennox, Gregory Maxwell, Colin Perkins, Jan Skoglund, + Timothy B. Terriberry, Martin Thompson, Justin Uberti, Magnus Westerlund, and Mo Zanaty. +
+ + + + + &rfc2119; + &rfc3389; + &rfc3550; + &rfc3711; + &rfc3551; + &rfc6838; + &rfc4855; + &rfc4566; + &rfc3264; + &rfc2326; + &rfc5576; + &rfc6562; + &rfc6716; + + + + &rfc2974; + &rfc4585; + &rfc5124; + &rfc5405; + &rfc7202; + + + + rmcat documents + + + + + + + + + + + + -- cgit v1.2.3