Network Working Group J. Uberti Internet-Draft Google Intended status: Standards Track C. Jennings Expires:September 4,December 6, 2012 Cisco Systems, Inc.March 3,June 4, 2012 Javascript Session Establishment Protocoldraft-ietf-rtcweb-jsep-00draft-ietf-rtcweb-jsep-01 Abstract This document proposes a mechanism for allowing a Javascript application to fully control the signaling plane of a multimedia session, and discusses how this would work with existing signaling protocols. This document is an input document for discussion. It should be discussed in the RTCWEB WG list, rtcweb@ietf.org. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on July 26, 2012. Copyright Notice Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . .. . .5 2. JSEP Approach . . . . . . . . . . . . . . . . . . . . . . . . . 5 3. Other Approaches Considered . . . . . . . . . . . . . . . . . . 6 4. Semantics and Syntax . . . . . . . . . . . . . . . . . . . . . 7 4.1. Signaling Model . . . . . . . . . . . . . . . . . . . . . . 7 4.2. Session Descriptions. . . . . . . . .and State Machine . . . . . . . . . . 7 4.3. Session Description Format . . . . . . . . . . . . . . . .89 4.4. Separation of Signaling and ICE State Machines . . . . . .910 4.5. ICE Candidate Trickling . . . . . . . . . . . . . . . . . .910 4.6. ICE Candidate Format . . . . . . . . . . . . . . . . . . .10 5. Media Setup Overview11 4.7. Interactions With Forking . . . . . . . . . . . . . . . . . 11 4.7.1. Serial Forking . . . .10 5.1. Initiating the Session. . . . . . . . . . . . . . . . 11 4.7.2. Parallel Forking . .10 5.1.1. Generating An Offer. . . . . . . . . . . . . . . . . 12 4.8. Session Rehydration .11 5.1.2. Applying the Offer. . . . . . . . . . . . . . . . . .11 5.1.3. Initiating ICE. 12 5. Interface . . . . . . . . . . . . . . . . . . .11 5.1.4. Serializing the Offer and Candidates. . . . . . . . 13 5.1. Methods .11 5.2. Receiving the Session. . . . . . . . . . . . . . . . . . .12 5.2.1. Receiving the Offer. . . . . . 13 5.1.1. createOffer . . . . . . . . . . . .12 5.2.2. Initiating ICE. . . . . . . . . . 13 5.1.2. createAnswer . . . . . . . . . .12 5.2.3. Handling ICE Messages. . . . . . . . . . . 14 5.1.3. SessionDescriptionType . . . . . .12 5.2.4. Generating the Answer. . . . . . . . . . 14 5.1.4. setLocalDescription . . . . . . .12 5.2.5. Applying the Answer. . . . . . . . . . . 15 5.1.5. setRemoteDescription . . . . . . .13 5.2.6. Serializing the Answer. . . . . . . . . . 15 5.1.6. localDescription . . . . . .13 5.3. Completing the Session. . . . . . . . . . . . . 16 5.1.7. remoteDescription . . . . .13 5.3.1. Receiving the Answer. . . . . . . . . . . . . . 16 5.1.8. updateIce . . .13 5.4. Updates to the Session. . . . . . . . . . . . . . . . . .13 6. Proposed WebRTC API changes. . 16 5.1.9. addIceCandidate . . . . . . . . . . . . . . . .14 6.1. PeerConnection API. . . . 17 5.2. Configurable SDP Parameters . . . . . . . . . . . . . . . .14 6.1.1 MediaHints17 6. Media Setup Overview . . . . . . . . . . . . . . . . . . . . . 17 6.1. Initiating the Session . .15 6.1.2 createOffer. . . . . . . . . . . . . . . . 18 6.1.1. Generating An Offer . . . . . .16 6.1.3 createAnswer. . . . . . . . . . . . 18 6.1.2. Applying the Offer . . . . . . . . . .16 6.1.4 SDP_OFFER, SDP_PRANSWER, and SDP_ANSWER. . . . . . . .17 6.1.5 setLocalDescription18 6.1.3. Handling ICE Callbacks . . . . . . . . . . . . . . . . 18 6.1.4. Serializing the Offer and Candidates . . .17 6.1.6 setRemoteDescription. . . . . . 19 6.2. Receiving the Session . . . . . . . . . . . .18 6.1.7 localDescription. . . . . . . 19 6.2.1. Receiving the Offer . . . . . . . . . . . . .18 6.1.8 remoteDescription. . . . . 19 6.2.2. Handling ICE Messages . . . . . . . . . . . . . .18 6.1.9 IceOptions. . . 19 6.2.3. Generating the Answer . . . . . . . . . . . . . . . . . 20 6.2.4. Applying the Answer . . . .19 6.1.10 startIce. . . . . . . . . . . . . . 20 6.2.5. Serializing the Answer . . . . . . . . .19 6.1.11 processIceMessage. . . . . . . 20 6.3. Completing the Session . . . . . . . . . . . .19 7. Example API Flows. . . . . . 20 6.3.1. Receiving the Answer . . . . . . . . . . . . . . . . . 207.1. Call using ROAP6.4. Updates to the Session . . . . . . . . . . . . . . . . . . 20 7. Security Considerations . . . .20 7.2. Call using XMPP. . . . . . . . . . . . . . . . 21 8. IANA Considerations . . . . . .20 7.3. Adding video to a call, using XMPP. . . . . . . . . . . .22 7.4. Simultaneous add of video streams, using XMPP. . . . 21 9. Acknowledgements . . .22 7.5. Call using SIP. . . . . . . . . . . . . . . . . . . . 21 10. References . .23 7.6. Handling early media (e.g. 1-800-FEDEX), using SIP. . . .24 8. Example Application. . . . . . . . . . . . . . . . . . . . 21 10.1. Normative References . .24 9. Security Considerations. . . . . . . . . . . . . . . . . 21 10.2. Informative References . . . . .26 10. IANA Considerations. . . . . . . . . . . . . 21 Appendix A. JSEP Implementation Examples . . . . . . . .26 11. Acknowledgements. . . . . 22 A.1. Example API . . . . . . . . . . . . . . . . . .26 12. References. . . . . . 22 A.2. Example API Flows . . . . . . . . . . . . . . . . . . . .26 12.1. Normative References. 23 A.2.1. Call using ROAP . . . . . . . . . . . . . . . . . .26 12.2. Informative References. . 23 A.2.2 Call using XMPP . . . . . . . . . . . . . . . . .27 Appendix A. Open Issues. . . 24 A.2.3. Adding video to a call, using XMPP . . . . . . . . . . 25 A.2.4. Simultaneous add of video streams, using XMPP . . . . . 26 A.2.5. Call using SIP . . . . . . . . . . . . . . . . . . . . 27 A.2.6. Handling early media (e.g. 1-800-FEDEX), using SIP . . 28 A.3. Full Example Application . . . . . . . . . . . . . . . . . 28 Appendix B. Change log . . . . . . . . . . . . . . . . . . . . . .2730 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . .2730 1. Introduction Thegeneralthinking behind WebRTC call setup has been to fully specify and control the media plane, but to leave the signaling plane up to the application as much as possible. The rationale is that different applications may prefer to use different protocols, such as the existing SIP or Jingle call signaling protocols, or something custom to the particular application, perhaps for a novel use case. In this approach, the key information that needs to be exchanged is the multimedia session description, which specifies the necessary transport and media configuration information necessary to establish the media plane. The original spec for WebRTC attempted to implement this protocol- agnostic signaling by providing a mechanism to exchange session descriptions in the form of SDP blobs. Upon starting a session, the browser would generate a SDP blob, which would be passed to the application for transport over its preferred signaling protocol. On the remote side, this blob would be passed into the browser from the application, and the browser would then generate a blob of its own in response. Upon transmission back to the initiator, this blob would be plugged into their browser, and the handshake would be complete. Experimentation with this mechanism turned up several shortcomings, which generally stemmed from there being insufficient context at the browser to fully determine the meaning of a SDP blob. For example, determining whether a blob is an offer or an answer, or differentiating a new offer from a retransmit. The ROAP proposal, specified inhttp://tools.ietf.org/html/draft- jennings-rtcweb-signaling-01,[I-D.draft-jennings-rtcweb-signaling- 01], attempted to resolve these issues by providing additional structure in the messaging - in essence, to create a generic signaling protocol that specifies how the browser signaling state machine should operate. However, even though the protocol is abstracted, the state machine forces aleast-common- denominatorleast-common-denominator approach on the signaling interactions. For example, in Jingle, the call initiator can provide additional ICE candidates even after the initial offer has been sent, which allows the offer to be sent immediately for quicker call startup. However, in the browser state machine, there is no notion of sending an updated offer before the initial offer has been responded to, rendering this functionality impossible. While specific concerns like this could be addressed by modifying the generic protocol, others would likely be discovered later. The main reason this mechanism is inflexible is because it embeds a signaling state machine within the browser. Since the browser generates the session descriptions on its own, and fully controls the possible states and advancement of the signaling state machine, modification of the session descriptions or use of alternate state machines becomes difficult or impossible. The browser environment also has its own challenges that cause problems for an embedded signaling state machine. One of these is that the user may reload the web page at any time. If this happens, and the state machine is being run at a server, the server can simply push the current state back down to the page and resume the call where it left off. If instead the state machine is run at the browser end, and is instantiated within, for example, the PeerConnection object, that state machine will be reinitialized when the page is reloaded and the JavaScript re-executed. This actually complicates the design of any interoperability service, as all cases where an offer or answer has already been generated but is now "forgotten" must now be handled by trying to move the client state machine forward to the same state it had been in previously in order to match what has already been delivered to and/or answered by the far side, or handled by ensuring that aborts are cleanly handled from every state and the negotiation rapidly restarted. 1.1. Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 2. JSEP Approach To resolvethese issues,the issues mentioned above, this document proposes the Javascript Session Establishment Protocol (JSEP) that pulls the signaling state machine out of the browser and into Javascript. This mechanism effectively removes the browser almost completely from the core signaling flow; the only interface needed is a way for the application to pass in the local and remote session descriptions negotiated by whatever signaling mechanism is used, and a way to interact with the ICE state machine. JSEP's handling of session descriptions is simple and straightforward. Whenever an offer/answer exchange is needed, the initiating side creates an offer by calling a createOffer()API on PeerConnection.API. The application can do massaging of that offer, if it wants to, and then uses it to set up its local config via a setLocalDescription() API. The offer is then sent off to the remote side over its preferred signaling mechanism (e.g. WebSockets); upon receipt of that offer, the remote party installs it using a setRemoteDescription() API. When the call is accepted, the callee uses a createAnswer() API to generate an appropriate answer, applies it using setLocalDescription(), and sends the answer back to the initiator over the signaling channel. When the offerer gets that answer, it installs it using setRemoteDescription(), and initial setup is complete. This process can be repeated for additional offer/answer exchanges. Regarding ICE,in this approach we decoupleJSEP decouples the ICE state machine from the overall signaling statemachine;machine, as the ICE state machine must remain in the browser,given thatsince only the browser has the necessary knowledge of candidates and other transport info.While transport has typically been lumped in with session descriptions, performingPerforming this separation it provides additionalflexibility. Inflexibility; in protocols that decouple session descriptions from transport, such as Jingle, the transport information can be sent separately; in protocols that don't, such as SIP, the information can be easily aggregated and recombined. Sending transport information separately can allow for faster ICE and DTLS startup, since the necessary roundtrips can occur while waiting for the remote side to accept the session. The JSEP approach does come with a minor downside. As the application now is responsible for driving the signaling state machine, slightly more application code is necessary to perform call setup; the application must call the right APIs at the right times, and convert the sessiondesciptionsdescriptions and ICE information into the defined messages of its chosen signaling protocol, instead of simply forwarding the messages emitted from the browser. One way to mitigate this is to provide a Javascript library that hides this complexity from the developer, which would implement the state machine and serialization of the desired signaling protocol. For example, this library could convert easily adapt the JSEP API into the exact ROAP API, thereby implementing the ROAP signaling protocol. Such a library could of course also implement other popular signaling protocols, including SIP or Jingle. In this fashion we can enable greater control for the experienced developer without forcing any additional complexity on the novice developer. 3. Other Approaches Considered Another approach that was considered for JSEP was to move the mechanism for generating offers and answers out of the browser as well.ThisInstead of providing createOffer/createAnswer methods within the browser, this approach wouldaddinstead expose a getCapabilities API which would provide the application with the information it needed in order to generate its own session descriptions. This increases the amount of work that the application needs to do; it needs to know how to generate session descriptions from capabilities, and especially how to generate the correct answer from an arbitrary offer andavailablethe supported capabilities. While this could certainly be addressed by using a library like the one mentioned above,some experimentation also indicates that coming up with a sufficiently complete getCapabilities API is a nontrivial undertaking. Nevertheless, if we wanted to go down this road, JSEP makesitsignificantly easier; if a getCapabilities API is added in the future,basically forces theapplication can generate session descriptions accordingly and pass thoseuse of said library even for a simple example. Exposing createOffer/createAnswer avoids that problem, but still allows applications to generate their own offers/answers if they choose, using thesetLocalDescription/setRemoteDescription APIs addeddescription generated byJSEP. (Even with JSEP, an application could still perform its own browser fingerprinting and generate approximate session descriptionscreateOffer asa result.)an indication of the browser's capabilities. Note also that while JSEP transfers more control to Javascript, it is not intended to be an example of a "low-level" API. The general argument against a low-level API is that there are too many necessary API points, and they can be called in any order, leading to something that is hard to specify and test. In the approach proposed here, control is performed via session descriptions; this requires only a few APIs to handle these descriptions, and they are evaluated in a specific fashion, which reduces the number of possible states and interactions. 4. Semantics and Syntax 4.1. Signaling Model JSEP does not specify a particular signaling model or state machine, other than the generic need to exchange RFC 3264 offers and answers in order for both sides of the session to know how to conduct the session. JSEP provides mechanisms to create offers and answers, as well as to apply them to aPeerConnection.session. However, the actual mechanism by which these offers and answers are communicated to the remote side, including addressing, retransmission, forking, and glare handling, is left entirely up to the application. +-----------+ +-----------+ | Web App |<--- App-Specific Signaling --->| Web App | +-----------+ +-----------+ | | | SDP | SDP V V +-----------+ +-----------+ | Browser |<----------- Media ------------>| Browser | +-----------+ +-----------+ Figure 1: JSEP Signaling Model 4.2. Session Descriptions and State Machine In order to establish the media plane,PeerConnectionthe user agent needs specific parameters to indicate what to transmit to the remote side, as well as how to handle the media that is received. These parameters are determined by the exchange of session descriptions in offers and answers, and there are certain details to this process that must be handled in the JSEP APIs. Whether a session description was sent or received affects the meaning of that description. For example, the list of codecs sent to a remote party indicates what the local side is willing to decode, and what the remote party should send. Not all parameters follow this rule; for example, the SRTP parameters [RFC4568] sent to a remote party indicate what the local side will use to encrypt, and thereby how the remote party should expect to receive. In addition, various RFCs put different conditions on the format of offers versus answers. For example, a offer may propose multiple SRTP configurations, but an answer may only contain a single SRTP configuration. Lastly, while the exact media parameters are only known only after a offer and an answer have been exchanged, it is possible for the offerer to receive media after they have sent an offer and before they have received an answer. To properly process incoming media in this case, the offerer's media handler must be aware of the details of the offerer before the answer arrives. Therefore, in order to handle session descriptions properly,PeerConnectionthe user agent needs: 1. To know if a session description pertains to the local or remote side. 2. To know if a session description is an offer or an answer. 3. To allow the offer to be specified independently of the answer. JSEP addresses this by adding both a setLocalDescription and a setRemoteDescription method, and both these methods takeasafirstparametereitherto indicate thevalue SDP_OFFER, SDP_PRANSWER (for a non-final answer) or SDP_ANSWER (for a final answer).type of session description being supplied. This satisfies the requirements listed above for both the offerer, who first callssetLocalDescription(SDP_OFFER,setLocalDescription("offer", sdp) and then latersetRemoteDescription(SDP_ANSWER,setRemoteDescription("answer", sdp), as well as for the answerer, who first callssetRemoteDescription(SDP_OFFER,setRemoteDescription("offer", sdp) and then latersetLocalDescription(SDP_ANSWER,setLocalDescription("answer", sdp). While it could be possible to implicitly determine the value of the offer/answerargument inside of PeerConnection,argument, requiring it to be specified explicitlyseems substantiallyis more robust, allowing invalid combinations (i.e. an answer before an offer) to generate an appropriate error. It also allows for an answer to be treated as provisional by the application. Provisional answers provide a way for an answerer to communicate session parameters back to the offerer, in order for the session to begin, while allowing a final answer to be specified later. This concept of a final answer is important to the offer/answer model; when such an answer is received, any extra resources allocated by the caller can be released, now that the exact session configuration is known. These "resources" can include things like extra ICE components, TURN candidates, or video decoders. Provisional answers, on the other hand, do no such deallocation; as a result, multiple dissimilar provisional answers can be received and applied during call setup. As in [RFC3264], an offerer can send an offer, and update it as long as it has not been answered. The answerer can send back zero or more provisional answers, and finally end the offer-answer exchange by sending a final answer. The state machine for this is as follows: +-----------+ | | | | | Stable |<---------------\ | | | | | | +-----------+ | ^ | | | | OFFER | ANSWER | | | ANSWER | V | +-----------+ +-----------+ | | | | | | PRANSWER | | | Offer |--------->| Pranswer | | | | | | |----\ | |----\ +-----------+ | +-----------+ | ^ | ^ | | | | | \-----/ \-----/ OFFER PRANSWER Figure 2: JSEP State Machine Aside from these state transitions, there is no other difference between the handling of provisional ("pranswer") and final ("answer") answers. 4.3. Session Description Format In the current WebRTC specification, session descriptions are formatted as SDP messages. While this format is not optimal for manipulation from Javascript, it is widely accepted, and frequently updated with new features. Any alternate encoding of session descriptions would have to keep pace with the changes to SDP, at least until the time that this new encoding eclipsed SDP in popularity. As a result, JSEP continues to use SDP as the internal representation for its session descriptions. However, to simplify Javascript processing, and provide for future flexibility, the SDP syntax is encapsulated within a SessionDescription object, which can be constructed from SDP, and be serialized out to SDP. If we were able to agree on a JSON format for session descriptions, we could easily enable this object to generate/expect JSON. Other methods may be added to SessionDescription in the future to simplify handling of SessionDescriptions from Javascript. 4.4. Separation of Signaling and ICE State MachinesPreviously, PeerConnection operated two state machines, referred to in the spec as an "ICE Agent", which handles the establishment of peer-to-peer connectivity, and an "SDP Agent", which handles the state of the offer-answer signaling. The states of these state machines were exposed through the iceState and sdpState attributes on PeerConnection, with an additional readyState attribute that reflected the high-level state of the PeerConnection.JSEP does away with the SDP Agent within thebrowser;browser, and this functionality is now controlled directly by the application, which uses the setLocalDescription and setRemoteDescription APIs to tellPeerConnectionthe browser what SDP has been negotiated. The ICE Agent remains in the browser, as it still needs toperform gatheringdrive the process of gathering candidates, connectivitychecking,checks, and related ICE functionality.The net effect of thisWhen a new ICE candidate isthat sdpState goes away, and processSignalingMessage becomes processIceMessage, which now specifically handles incomingavailable, the ICEcandidates. To allowAgent will notify the application via a callback; these candidates will automatically be added tocontrol exactly when it wants to start ICE negotiation (e.g. either on receipt of the call, or only after acceptingthecall), a startIce method haslocal session description. When all candidates have beenadded.gathered, the callback will also be invoked to signal that the gathering process is complete. 4.5. ICE Candidate Trickling Candidate trickling is a technique through which a caller may incrementally provide candidates to the callee after the initial offer has been dispatched. This allows the callee to begin acting upon the call and setting up the ICE (and perhaps DTLS) connections immediately, without having to wait for the caller to allocate all possible candidates, resulting in faster call startup in many cases. JSEP supports optional candidate trickling by providing APIs that provide control and feedback on the ICE candidate gathering process. Applications that support candidate trickling can send the initial offer immediately and send individual candidates when they getathe onicecandidate callback with a new candidate; applications that do not support this feature can simply wait for the final onicecandidate callback that indicates gathering is complete, andsimplycreate and send their offer, with all the candidates, at this time.To be clear, aplicationsUpon receipt of trickled candidates, the receiving application can supply them to its ICE Agent by calling an addIceCandidate method. This triggers the ICE Agent to start using this remote candidate for connectivity checks. Applications that do not make use of candidate tricking can ignoreprocessIceMessageaddIceCandidate entirely, and useIceCallbackthe onicecandidate callback solely to indicate when candidate gathering is complete. 4.6. ICE Candidate Format As with session descriptions, we choose to provide an IceCandidate object that provides some abstraction, but can be easily converted to/from SDP a=candidate lines. The IceCandidate object hasa fieldfields to indicate which m= line it should be associated with, and a method to convert to a SDP representation, ex: a=candidate:1 1 UDP 1694498815 66.77.88.99 10000 typ host Currently, a=candidate lines are the onlythingSDP information thatareis contained within IceCandidate, asthis isthey represent the only informationthat isneeded that is not present in the initial offer (i.e. for trickle candidates).5. Media Setup Overview The example here shows4.7. Interactions With Forking 4.7.1. Serial Forking Serial forking involves atypicalcallsetup using the JSEP model. We assume the following architecture in this example,being dispatched to multiple remote callees, whereUA is synonymous with "browser", and JS is synonymous with "web application": OffererUA <-> OffererJS <-> WebServer <-> AnswererJS <-> AnswererUA 5.1. Initiatingeach callee can accept theSession The initiator createscall, but only one active session ever exists at aPeerConnection, installs its IceCallback, and adds the desired MediaStreams (presumably obtained via getUserMedia). The PeerConnectiontime; no mixing of received media isinperformed. JSEP handles serial forking well, allowing theNEW state. OffererJS->OffererUA: var pc = new PeerConnection(config, iceCb); OffererJS->OffererUA: pc.addStream(stream); 5.1.1. Generating An Offer The initiator then creates a session description to offerapplication to easily control thecallee. This description includespolicy for selecting thecodecs and other necessary session parameters, as well as information about eachdesired remote endpoint. When an answer arrives from one of thestreams that has been added (e.g. SSRC, CNAME, etc.) The created description includes all parameters thatcallees, theofferer's UA supports; if the initiator wants to influence the created offer, theyapplication canpass in a MediaHints object to createOffer that allows for customization (e.g. if the initiator wantschoose toreceive but not send video). The initiator can also directly manipulate the created session description as well, perhaps ifapply itwants to changeeither as a provisional answer, leaving open theprioritypossibility of using a different answer in theoffered codecs. OffererJS->OffererUA: var offer = pc.createOffer(null); 5.1.2. Applyingfuture, or apply it as a final answer, ending theOffer The initiator then instructssetup flow. In a "first-one-wins" situation, thePeerConnection to use this offerfirst answer will be applied as a final answer, and thelocal description forapplication will send a terminate message to any subsequent answers. In SIP parlance, thissession, i.e. what codecs itwould be ACK + BYE. In a "last-one-wins" situation, all answers would be applied as provisional answers, and any previous call leg willuse for received media, what SRTP keys itbe terminated. At some point, the application willuse for sending media (if using SDES), etc. In order thatend theUA handlesetup process, perhaps with a timer; At this point, thedescription properly,application could reapply theinitiator marks itexisting remote description asan offer when calling setLocalDescription; this indicatesa final answer. 4.7.2. Parallel Forking Parallel forking involves a call being dispatched to multiple remote callees, where each callee can accept theUA thatcall, and multiplecapabilities have been offered, but this set maysimultaneous active sessions can bepared back later, whenestablished as a result. If multiple callees send media, this media is mixed and played out at theanswer arrives. Sincecaller side. JSEP can handle parallel forking by "cloning" thelocal user agent must be prepared to receive media upon applying the offer, this operation will cause local decoder resourcessession when needed tobe allocated, based oncreate multiple parallel sessions. When thecodecs indicated infirst answer is received, theoffer. OffererJS->OffererUA: pc.setLocalDescription(SDP_OFFER, offer); 5.1.3. Initiating ICE The initiatorcaller cannow startclone theICE process of candidate generationexisting session, andconnectivity checking. This results in callbacks to the application's IceCallback. Candidates are provided tothen apply theIceCallbackanswer asthey are allocated, with the |moreToFollow| argument seta final answer totrue if there are still allocations pending; whenthelast allocation completes or times out, this callback will be invoked with |moreToFollow| set to false. OffererJS->OffererUA: pc.startIce(); OffererUA->OffererJS: iceCallback(candidate, ...); 5.1.4. Serializingoriginal session. Upon receiving theOffernext answer, the cloned session is cloned again, andCandidates At this point,theoffererreceived answer isreadyapplied as a final answer tosend its offerthe first clone. This process repeats until the caller decides to end thecallee using its preferred signaling protocol. Depending onsetup flow, and closes theprotocol, it can either sendfinal cloned session. Cloned sessions inherit theinitiallocal session descriptionfirst,andthen "trickle" the ICEcandidatesas they are given to the application, or itfrom their parent, and an empty remote description; only sessions that have not yet applied an answer canwait for all the ICE candidates tobecollected, and then send the offer and list ofcloned. Each cloned session may discover new peer-reflexive candidates; these candidatesall at once. 5.2. Receivingwill be supplied via theSession Throughonicecandidate callback to that specific session. Since thechosen signaling protocol,clone uses therecipientsame local description as its parent, creating a clone will fail if it isnotifiednot possible to reserve the same resources for the clone as have already been reserved by the parent. As a result ofan incoming session request. It createsthis cloning, the application will end up with N parallel sessions, each with aPeerConnection,local andinstalls itsremote description and their ownIceCallback. AnswererJS->AnswererUA: var pc = new PeerConnection(config, iceCb); 5.2.1. Receiving the Offerlocal and remote addresses. Therecipient converts the received offermedia flow fromits signaling protocol intothese sessions can be managed by specifying SDPformat, and supplies itdirection attributes in the descriptions, or the application can choose toits PeerConnection, again marking it as an offer. As a remote description,play out theoffer indicates what codecsmedia from all sessions mixed together. Of course, if theremote sideapplication wants touse for receiving, as well as what SRTP keysonly keep a single session, itwill use for sending. The setting ofcan simply terminate theremote description causes callbacks to be issued, informingsessions that it no longer needs. 4.8. Session Rehydration In the event that the local application state is reinitialized, either due to a user reload ofwhat kinds of streams are present intheoffer. This step will also cause encoder resourcespage, or a decision within the application tobe allocated, basedreload itself (perhaps to update to a new version), it is possible to keep an existing session alive via a process called "rehydration". With rehydration, the current local session description is persisted somewhere outside of the page, perhaps on thecodecs specifiedapplication server, or in|offer|. AnswererJS->AnswererUA: pc.setRemoteDescription(SDP_OFFER, offer); AnswererUA->AnswererJS: onAddStream(stream); 5.2.2. Initiating ICEbrowser local storage. Therecipientpage is thenstarts its ownreloaded, and a new session object is created in Javascript. The saved local session is now retrieved, but the previous ICEstate machine, to allow connectivity tocandidates will no longer beestablished as quickly as possible. AnswererJS->AnswererUA: pc.startIce(); AnswererUA->AnswererJS: iceCallback(candidate, ...); 5.2.3. Handlingvalid in this case, so we will need to perform an ICEMessages Ifrestart; to do so, we simply generate a new ICEcandidates fromufrag/pwd combo for theremote site were included in the offer, the ICE Agent will automatically start trying to use them. Otherwise, if ICE candidates are sent separately, they are passed into the PeerConnection when they arrive. AnswererJS->AnswererUA: pc.processIceMessage(candidate); 5.2.4. Generating the Answer Once the recipient has decided to accept the session, it generates an answer sessionlocal description.This process performs the appropriate intersection of codecsThe modified local description is then installed via setLocalDescription, andother parameterssent off as an offer togeneratethecorrect answer. Asremote side, who will reply withthe offer, MediaHintsan answer that can beprovidedsupplied toinfluence the answer thatsetRemoteDescription. ICE processing proceeds as usual, and as soon as connectivity isgenerated, and/or the application can post-processestablished, theanswer manually. AnswererJS->AnswererUA: pc.createAnswer(offer, null); 5.2.5. Applyingsession will be back up and running again. 5. Interface This section details theAnswerbasic operations that must be present to implement JSEP functionality. Therecipient then instructsactual API exposed in thePeerConnectionW3C API may have somewhat different syntax, but should map easily tousethese concepts. 5.1. Methods 5.1.1. createOffer The createOffer method generates a blob of SDP that contains a RFC 3264 offer with theanswer as its local descriptionsupported configurations forthis session, i.e. what codecs it will use to receive media, etc. It also marks the description as an answer, which tellstheUA that these parameters are final. This causessession, including descriptions of thePeerConnection to movelocal MediaStreams attached to this PeerConnection, theACTIVE state,codec/RTP/RTCP options supported by this implementation, andtransmission of mediaany candidates that have been gathered by theanswererICE Agent. A constraints parameters may be supplied tostart. AnswererJS->AnswererUA: pc.setLocalDescription(SDP_ANSWER, answer); AnswererUA->OffererUA: <media> 5.2.6. Serializing the Answer As withprovide additional control over the generated offer,the answer (withe.g. to get a full set of session capabilities, orwithout candidates) is now convertedto request a new set of ICE credentials. In the initial offer, the generated SDP will contain all desiredsignaling format and sent tofunctionality for theinitiator. 5.3. Completingsession (certain parts that are supported but not desired by default may be omitted); for each SDP line, theSession 5.3.1. Receivinggeneration of theAnswer The initiator convertsSDP must follow theanswer fromappropriate process for generating an offer. In thesignaling protocol and applies it asevent createOffer is called after theremote description, marking it assession is established, createOffer will generate ananswer. This causesoffer that is compatible with thePeerConnection to movecurrent session, incorporating any changes that have been made to theACTIVE state, and transmissionsession since the last complete offer-answer exchange, such as addition or removal ofmedia bystreams. If no changes have been made, theofferer to start. OffererJS->OffererUA: pc.setRemoteDescription(SDP_ANSWER, answer); OffererUA->AnswererUA: <media> 5.4. Updatesoffer will be identical to the current local description. SessionUpdates to the session are handled withdescriptions generated by createOffer must be immediately usable by setLocalDescription; if anew offer/answer exchange. However, since mediasystem has limited resources (e.g. a finite number of decoders), createOffer should return an offer that reflects the current state of the system, so that setLocalDescription willalready be flowing atsucceed when it attempts to acquire those resources. Because thispoint,method may need to inspect thenew offerer needssystem state tosupport both its old session description as well asdetermine thenew onecurrently available resources, ithas offered, until themay be implemented as an async operation. Calling this method does not change state; its use isaccepted by the remote side. Note alsonot required. 5.1.2. createAnswer The createAnswer method generates a blob of SDP thatin an update scenario,contains a RFC 3264 SDP answer with theroles may be reversed, i.e.supported configuration for theupdate offerer can be different thansession that is compatible with theoriginal offerer. 6. Proposed WebRTC API changes 6.1. PeerConnection API The text below indicatesparameters supplied in |offer|. Like createOffer, therecommended changes toreturned blob contains descriptions of thePeerConnection APIlocal MediaStreams attached toimplementthis PeerConnection, theJSEP functionality. Methods marked with a [+] are new/proposed; methods marked with a [-]codec/RTP/RTCP options negotiated for this session, and any candidates that have beenremoved in this proposal. [Constructor (in DOMString configuration, in IceCallback iceCb)] interface PeerConnection { // creates a blob of SDP togathered by the ICE Agent. A constraints parameter may beprovided assupplied to provide additional control over the generated answer. As anoffer. [+] SessionDescription createOffer (MediaHints hints); // createsanswer, the generated SDP will contain ablobspecific configuration that specifies how the media plane should be established. For each SDP line, the generation of the SDPto be provided asmust follow the appropriate process for generating an answer.[+] SessionDescriptionSession descriptions generated by createAnswer(DOMString offer, MediaHints hints); // actions, for setLocalDescription/setRemoteDescription [+] const unsigned short SDP_OFFER = 0x100; [+] const unsigned short SDP_PRANSWER = 0x200; [+] const unsigned short SDP_ANSWER = 0x300; // sets the local session description [+] void setLocalDescription (unsigned short action, SessionDescription desc); // setsmust be immediately usable by setLocalDescription; like createOffer, theremote sessionreturned description[+] void setRemoteDescription (unsigned short action, SessionDescription desc); // returnsshould reflect the currentlocal session description [+] readonly SessionDescription localDescription; // returnsstate of thecurrent remote session description [+] readonly SessionDescription remoteDescription; [-] void processSignalingMessage (DOMString message); const unsigned short NEW = 0; // initialsystem. Because this method may need to inspect the system state[+] const unsigned short OPENING = 1; //to determine the currently available resources, it may need to be implemented as an async operation. Calling this method does not change state; its use is not required. 5.1.3. SessionDescriptionType The strings "offer", "pranswer", and "answer" serve as type arguments to setLocalDescription and setRemoteDescription. They provide information as to how the description parameter should be parsed, and how the media state should be changed. "offer" indicates that a description should be parsed as an offer; said description may include many possible media configurations. A description used as an "offer" may be applied anytime the PeerConnection is in a stable state, or as an update to a previously sent but unanswered "offer". "pranswer" indicates that a description should be parsed as an answer, but not a final answer, and so should not result in the freeing of allocated resources. It may result in the start of media transmission, if the answer does not specify an inactive media direction. A description used as a "pranswer" may be applied as a response to an "offer", or an update to a previously sent "answer". "answer" indicates that a description should be parsed as an answer, the offer-answer exchange should be considered complete, and any resources (decoders, candidates) that are no longer needed can be released. A description used as an "answer" may be applied as a response to a "offer", or an update to a previously sent "pranswer". The application can use some discretion on whether an answer should be applied as provisional or final. For example, in a serial forking scenario, an application may receive multiple "final" answers, one from each remote endpoint. The application could accept the initial answers as provisional answers, and only apply an answer as final when it receives one that meets its criteria (e.g. a live user instead of voicemail). 5.1.4. setLocalDescription The setLocalDescription method instructs the PeerConnection to apply the supplied SDP blob as its local configuration. The type parameter indicates whether the blob should be processed as an offer, provisional answer, or final answer; offers and answers are checked differently, using the various rules that exist for each SDP line. This API changes the local media state; among other things, it sets up local resources for receiving and decoding media. In order to successfully handle scenarios where the application wants to offer to change from one media format to a different, incompatible format, the PeerConnection must be able to simultaneously support use of both the old and new local descriptions (e.g. support codecs that exist in both descriptions) until a final answer is received, at which point the PeerConnection can fully adopt the new local description, or roll back to the old description if the remote side denied the change. If setRemoteDescription was previous called with an offer, and setLocalDescription is called with an answer (provisional or final), and the media directions are compatible, this will result in the starting of media transmission. 5.1.5. setRemoteDescription The setRemoteDescription method instructs the PeerConnection to apply the supplied SDP blob as the desired remote configuration. As in setLocalDescription, the |type| parameter indicates how the blob should be processed. This API changes the local media state; among other things, it sets up local resources for sending and encoding media. If setRemoteDescription was previous called with an offer, and setLocalDescription is called with an answer (provisional orremote desc set const unsigned short ACTIVE = 2; // localfinal), andremote desc set const unsigned short CLOSED = 3; // ended state readonly attribute unsigned short readyState; // starts ICE connection/handshaking [+] void startIce (optional IceOptions options); // processes received ICE information [+] void processIceMessage (IceCandidate candidate); const unsigned short ICE_GATHERING = 0x100; const unsigned short ICE_WAITING = 0x200; const unsigned short ICE_CHECKING = 0x300; const unsigned short ICE_CONNECTED = 0x400; const unsigned short ICE_COMPLETED = 0x500; const unsigned short ICE_FAILED = 0x600; const unsigned short ICE_CLOSED = 0x700; readonly attribute unsigned short iceState; [-] const unsigned short SDP_IDLE = 0x1000; [-] const unsigned short SDP_WAITING = 0x2000; [-] const unsigned short SDP_GLARE = 0x3000; [-] readonly attribute unsigned short sdpState; void addStream (MediaStream stream, MediaStreamHints hints); void removeStream (MediaStream stream); readonly attribute MediaStream[] localStreams; readonly attribute MediaStream[] remoteStreams; void close (); [ restthe media directions are compatible, this will result in the starting of media transmission. 5.1.6. localDescription The localDescription method returns a copy ofinterface omitted ] }; [Constructor (in DOMString sdp)] interface SessionDescription { // addsthespecified candidatecurrent local configuration, i.e. what was most recently passed to setLocalDescription, plus any local candidates that have been generated by the ICE Agent. A null object will be returned if the local descriptionvoid addCandidate(IceCandidate candidate); // serializeshas not yet been established. 5.1.7. remoteDescription The remoteDescription method returns a copy of the current remote configuration, i.e. what was most recently passed to setRemoteDescription, plus any remote candidates that have been supplied via processIceMessage. A null object will be returned if the remote description has not yet been established. 5.1.8. updateIce The updateIce method allows the configuration of the ICE Agent toSDP DOMString toSdp(); }; [Constructor (in DOMString label, in DOMString candidateLine)] interface IceCandidate { //be changed during them= line this candidate is associated with readonly DOMString label; // creates a SDP-ized formsession, primarily for changing which types of local candidates are provided to the application and used for connectivity checks. A callee may initially configure the ICE Agent to use only relay candidates, to avoid leaking location information, but update thiscandidate DOMString toSdp(); }; 6.1.1 MediaHints MediaHintsconfiguration to use all candidates once the call isan object that canaccepted. Regardless of the configuration, the gathering process collects all available candidates, but excluded candidates will not bepassed into createOffersurfaced in onicecallback orcreateAnswerused for connectivity checks. This call may result in a change toaffectthetypestate ofoffer/answer that is generated. The following properties can be set on MediaHints: has_audio: boolean Indicates whether we want to receive audio; defaultsthe ICE Agent, and may result in a change totruemedia state ifwe have audio streams, else false has_video: boolean Indicates whether we wantit results in connectivity being established. 5.1.9. addIceCandidate The addIceCandidate method provides a remote candidate toreceive video; defaultsthe ICE Agent, which will be added totrue if we have video streams, else false As an example, MediaHints couldthe remote description. Connectivity checks will beusedsent tocreatethe new candidate. This call will result in asession that transmits only audio, but is ablechange toreceive video from the remote side, by forcingtheinclusionstate of the ICE Agent, and may result in am=video line even when no video sources are provided. 6.1.2 createOfferchange to media state if it results in connectivity being established. 5.2. Configurable SDP Parameters ThecreateOffer method generatesfollowing is ablobpartial list of SDP parameters thatcontainsan application may want to control, in either local or remote descriptions, using this API. - remove or reorder codecs (m=) - change codec attributes (a=fmtp; ptime) - enable/disable BUNDLE (a=group) - enable/disable RTCP mux (a=rtcp-mux) - remove or reorder SRTP crypto-suites (a=crypto) - change SRTP parameters or keys (a=crypto) - change send resolution or framerate (TBD) - change desired recv resolution or framerate (TBD) - change total bandwidth (b=) - remove desired AVPF mechanisms (a=rtcp-fb) - remove RTP header extensions (a=rtphdr-ext) - add/change SSRC grouping (e.g. FID, RTX, etc) (a=ssrc-group) - add SSRC attributes (a=ssrc) - change ICE ufrag/password (a=ice-ufrag/pwd) - change media send/recv state (a=sendonly/recvonly/inactive) For example, an application could implement call hold by adding an a=inactive attribute to its local description, and then applying and signaling that description. 6. Media Setup Overview The example here shows aRFC 3264 offer with the supported configurations fortypical call setup using thesession, including descriptions ofJSEP model, indicating thelocal MediaStreams attached to this PeerConnection,functions that are called and thecodec/RTP/RTCP options supported bystate changes that occur. We assume the following architecture in thisimplementation,example, where UA is synonymous with "browser", andany candidates that have been gathered byJS is synonymous with "web application": OffererUA <-> OffererJS <-> WebServer <-> AnswererJS <-> AnswererUA 6.1. Initiating theICE Agent.Session The|hints| parameter may be suppliedinitiator creates a PeerConnection, hooks up toprovide additional control over the generated offer. As an offer, the generated SDP will contain the full set of capabilities supported byits ICE callback, and adds thesession (as opposed to an answer, which will include only a specific negotiated subsetdesired MediaStreams (presumably obtained via getUserMedia). The ICE gathering process begins touse);gather candidates foreach SDP line, the generationa default number of streams, as theSDP must follow the appropriate process for generating an offer. Inexact number will not be known until theevent createOfferlocal description iscalled afterapplied. The PeerConnection is in the NEW state. OffererJS->OffererUA: var pc = new PeerConnection(config, null); OffererJS->OffererUA: pc.onicecandidate = onIceCandidate; OffererJS->OffererUA: pc.addStream(stream); 6.1.1. Generating An Offer The initiator then creates a sessionis established, createOffer will generate andescription to offerthat is compatible with the current session, incorporating any changes that have been madeto thesession sincecallee. This description includes thelast complete offer-answer exchange, suchcodecs and other necessary session parameters, asaddition or removalwell as information about each ofstreams. If no changes have been made,theoffer will be identical tostreams that has been added (e.g. SSRC, CNAME, etc.) The created description includes all parameters that thecurrent local description. Session descriptions generated by createOffer must be immediately usable by setLocalDescription;offerer's UA supports; if the initiator wants to influence the created offer, they can pass in asystem has limited resources (e.g. a finite number of decoders),MediaConstraints object to createOffershould return an offerthatreflectsallows for customization (e.g. if thecurrent state ofinitiator wants to receive but not send video). The initiator can also directly manipulate thesystem, so that setLocalDescription will succeed whencreated session description as well, perhaps if itattemptswants toacquire those resources. Calling this method does notchange thestatepriority of thePeerConnection; its use is not required. A TBD exception is thrown ifoffered codecs. OffererJS->OffererUA: var offer = pc.createOffer(null); 6.1.2. Applying the|hints| parameter is malformed. 6.1.3 createAnswerOffer ThecreateAnswer method generates a blob of SDP that contains a RFC 3264 SDP answer withinitiator then instructs thesupported configuration forPeerConnection to use this offer as thesessionlocal description for this session, i.e. what codecs it will use for received media, what SRTP keys it will use for sending media (if using SDES), etc. In order thatis compatible withtheparameters supplied in |offer|. Like createOffer,UA handle thereturned blob contains descriptions ofdescription properly, thelocal MediaStreams attached toinitiator marks it as an offer when calling setLocalDescription; thisPeerConnection,indicates to thecodec/RTP/RTCP options negotiated for this session, and any candidatesUA that multiple capabilities have beengathered byoffered, but this set may be pared back later, when the answer arrives. Since the local user agent must be prepared to receive media upon applying the offer, this operation will cause local decoder resources to be allocated, based on the codecs indicated in the offer. OffererJS->OffererUA: pc.setLocalDescription("offer", offer); 6.1.3. Handling ICEAgent.Callbacks The|hints| parameter may be suppliedinitiator starts to receive callbacks on its onicecandidate handler. Candidates are provided toprovide additional control overthegenerated answer. As an answer,IceCallback as they are allocated; when thegenerated SDPlast allocation completes or times out, this callback willcontainbe invoked with aspecific configuration that specifies hownull argument. OffererUA->OffererJS: onIceCandidate(candidate); 6.1.4. Serializing themedia plane should be established. For each SDP line,Offer and Candidates At this point, thegeneration ofofferer is ready to send its offer to theSDP must followcallee using its preferred signaling protocol. Depending on theappropriate process for generating an answer. Session descriptions generated by createAnswer must be immediately usable by setLocalDescription; like createOffer,protocol, it can either send thereturnedinitial session descriptionshould reflectfirst, and then "trickle" thecurrent state ofICE candidates as they are given to thesystem. Calling this method does not changeapplication, or it can wait for all thestateICE candidates to be collected, and then send the offer and list of candidates all at once. 6.2. Receiving thePeerConnection; its use is not required. A TBD exception is thrown ifSession Through the|hints| parameter is malformed, orchosen signaling protocol, the|offer| parameterrecipient ismissing or malformed. 6.1.4 SDP_OFFER, SDP_PRANSWER,notified of an incoming session request. It creates a PeerConnection, andSDP_ANSWERsets up its own ICE callback. TheSDP_XXXX enums serve as arguments to setLocalDescription and setRemoteDescription. They provide information asICE gathering process begins tohowgather candidates for a default number of streams. AnswererJS->AnswererUA: var pc = new PeerConnection(config, null); AnswererJS->AnswererUA: pc.onicecandidate = onIceCandidate; 6.2.1. Receiving the|description| parameter should be parsed,Offer The recipient converts the received offer from its signaling protocol into SDP format, andhowsupplies it to its PeerConnection, again marking it as an offer. As a remote description, themedia state should be changed. SDP_OFFERoffer indicatesthat a description should be parsedwhat codecs the remote side wants to use for receiving, asan offer; said description may include many possible media configurations. A description usedwell asa SDP_OFFER maywhat SRTP keys it will use for sending. The setting of the remote description causes callbacks to beapplied anytimeissued, informing thePeerConnection isapplication of what kinds of streams are present ina stable state, or as an updatethe offer. This step will also cause encoder resources toa previously sent but unanswered SDP_OFFER. SDP_PRANSWER indicates that a description shouldbeparsed as an answer, but not a final answer, and so should not resultallocated, based on the codecs specified in |offer|. AnswererJS->AnswererUA: pc.setRemoteDescription("offer", offer); AnswererUA->AnswererJS: onAddStream(stream); 6.2.2. Handling ICE Messages If ICE candidates from thestarting of media transmission. A description used as a SDP_PRANSWER may be applied as a response to a SDP_OFFER, or an updateremote site were included in the offer, the ICE Agent will automatically start trying toa previouslyuse them. Otherwise, if ICE candidates are sentSDP_PRANSWER. SDP_ANSWER indicates that a description should be parsed asseparately, they are passed into the PeerConnection when they arrive. AnswererJS->AnswererUA: pc.addIceCandidate(candidate); 6.2.3. Generating the Answer Once the recipient has decided to accept the session, it generates ananswer,answer session description. This process performs the appropriate intersection of codecs and other parameters to generate theoffer-answer exchange should be considered complete. A description used as a SDP_ANSWER maycorrect answer. As with the offer, MediaConstraints can beapplied as a response to a SDP_OFFER, or an updateprovided toa previously send SDP_PRANSWER. 6.1.5 setLocalDescriptioninfluence the answer that is generated, and/or the application can post-process the answer manually. AnswererJS->AnswererUA: pc.createAnswer(offer, null); 6.2.4. Applying the Answer ThesetLocalDescription methodrecipient then instructs the PeerConnection toapplyuse thesupplied SDP blobanswer as its localconfiguration. The |type| parameter indicates whetherdescription for this session, i.e. what codecs it will use to receive media, etc. It also marks theblob should be processeddescription as anoffer (SDP_OFFER), provisional answer (SDP_PRANSWER), or final answer (SDP_ANSWER); offers and answers are checked differently, usinganswer, which tells thevarious rulesUA thatexist for each SDP line.these parameters are final. ThisAPI changes the local media state; among other things, it sets up local resources for receiving and decoding media. In order to successfully handle scenarios wherecauses theapplication wantsPeerConnection tooffermove tochange from onethe ACTIVE state, and transmission of mediaformat to a different, incompatible format,by thePeerConnection must be ableanswerer tosimultaneously support use ofstart (assuming boththe old and new local descriptions (e.g. support codecs that existsides have indicated this inboth descriptions) until a final answer is received, at which pointtheir descriptions). AnswererJS->AnswererUA: pc.setLocalDescription("answer", answer); AnswererUA->OffererUA: <media> 6.2.5. Serializing thePeerConnection can fully adoptAnswer As with thenew local description,offer, the answer (with orroll backwithout candidates) is now converted to theold description ifdesired signaling format and sent to theremote side deniedinitiator. 6.3. Completing thechange. Changes toSession 6.3.1. Receiving thestate of media transmission will only occur when a final answer is successfully applied. A TBD exception is thrown if |description| is invalid. A TBD exception is thrown if there are insufficient local resources to apply |description|. 6.1.6 setRemoteDescriptionAnswer ThesetRemoteDescription method instructsinitiator converts thePeerConnection to applyanswer from thesupplied SDP blobsignaling protocol and applies it as thedesiredremoteconfiguration. As in setLocalDescription, the |type| parameter indicates how the blob should be processed.description, marking it as an answer. ThisAPI changescauses thelocal media state; among other things, it sets up local resources for sending and encoding media. ChangesPeerConnection to move to thestate of mediaACTIVE state, and transmissionwill only occur when a final answer is successfully applied. A TBD exception is thrown if |description| is invalid. A TBD exception is thrown if there are insufficient local resources to apply |description|. 6.1.7 localDescription The localDescription method returns a copyof media by thecurrent local configuration, i.e. what was most recently passedofferer tosetLocalDescription, plus any local candidates thatstart (assuming both sides havebeen generated byindicated this in their descriptions). OffererJS->OffererUA: pc.setRemoteDescription("answer", answer); OffererUA->AnswererUA: <media> 6.4. Updates to theICE Agent. A null objectSession Updates to the session are handled with a new offer/answer exchange. However, since media will already bereturned ifflowing at this point, thelocalnew offerer needs to support both its old session description as well as the new one it hasnot yet been established. 6.1.8 remoteDescription The remoteDescription method returns a copy ofoffered, until the change is accepted by thecurrent remote configuration, i.e. what was most recently passed to setRemoteDescription, plus anyremotecandidatesside. Note also thathave been supplied via processIceMessage. A null object willin an update scenario, the roles may bereturned ifreversed, i.e. the update offerer can be different than the original offerer. 7. Security Considerations TODO 8. IANA Considerations This document requires no actions from IANA. 9. Acknowledgements Harald Alvestrand, Dan Burnett, Neil Stratford, Eric Rescorla, Anant Narayanan, and Adam Bergkvist all provided valuable feedback on this proposal. Matthew Kaufman provided theremote description has not yet been established. 6.1.9 IceOptions IceOptions is an objectobservation thatcan be passed into startIcekeeping state out of the browser allows a call torestrictcontinue even if thecandidates that arepage is reloaded. Richard Ejzak providedtotheapplication and usedspecifics on session cloning. 10. References 10.1. Normative References [RFC2119] Bradner, S., "Key words forconnectivity checks. This can be useful if the application wants to onlyuseTURN candidatesin RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC3264] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with Session Description Protocol (SDP)", RFC 3264, June 2002. [RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session Description Protocol", RFC 4566, July 2006. 10.2. Informative References [RFC4568] Andreasen, F., Baugher, M., and D. Wing, "Session Description Protocol (SDP) Security Descriptions forprivacy reasons, or only local + STUN candidatesMedia Streams", RFC 4568, July 2006. [RFC5245] Rosenberg, J., "Interactive Connectivity Establishment (ICE): A Protocol forcost reasons.Network Address Translator (NAT) Traversal for Offer/Answer Protocols", RFC 5245, April 2010. [webrtc-api] Bergkvist, Burnett, Jennings, Narayanan, "WebRTC 1.0: Real-time Communication Between Browsers", May 2011. Available at http://dev.w3.org/2012/webrtc/editor/webrtc.html Appendix A. JSEP Implementation Examples A.1. Example API Thefollowing properties caninterface below shows a basic Javascript API that could beset on IceOptions: use_candidates: "all", "no_relay", "only_relay" Indicates what typesused to expose the functionality discussed in this document. This API is used for the examples in the following parts oflocal candidates shouldthis Appendix. // actions, for setLocalDescription/setRemoteDescription enum SessionDescriptionType { "offer", "pranswer", "answer" } // constraints that can beused; defaultssupplied to"all" 6.1.10 startIce The startIce method startsthe ctor orupdatescreateXXXX enum MediaConstraints { "offerConfig", // controls theICE Agent processkind ofgathering local candidates and pinging remote candidates. The |options| argumentoffer created; // "default" (normal offer) // "caps" (all capabilities) // "new" (brand new description) // "iceRestart" (new ICE creds) "iceTransports", // controls ICE candidates; can beused to restrict which types of local candidates are provided to// "none" (no candidates) // "relay" (only relay candidates) // "all" (all available candidates) } [Constructor (int index, DOMString id, in DOMString candidateLine)] interface IceCandidate { // theapplication and usedm= line index forpinging;thiscan be used to limitcandidate readonly attribute int mLineIndex // theuse of TURN candidates by a callee to avoid leaking location information prior tomid for thecall being accepted. This call may result inm= line for this candidate readonly attribute DOMString mLineId; // creates achangeSDP-ized form of this candidate stringifier DOMString (); }; [Constructor (DOMString sdp)] interface SessionDescription { // adds the specified candidate to thestate ofdescription void addCandidate(IceCandidate candidate); // serializes theICE Agent, and may result indescription to SDP stringifier DOMString (); }; [Constructor (DOMString configuration, optional MediaConstraints constraints)] interface PeerConnection { // creates achangeblob of SDP tomedia state if it results in connectivity being established. A TBD exception willbethrown if |options| is malformed. 6.1.11 processIceMessage The processIceMessage method providesprovided as an offer. SessionDescription createOffer ( SessionDescriptionCallback successCb, optional ErrorCallback errorCb, optional MediaContraints constraints); // creates aremote candidateblob of SDP tothe ICE Agent, which willbeadded toprovided as an answer. SessionDescription createAnswer ( SessionDescription offer, SessionDescriptionCallback successCb, optional ErrorCallback errorCb, optional MediaContraints constraints); // sets the local session description void setLocalDescription ( SessionDescriptionType action, SessionDescription desc); // sets the remotedescription. If startIce has been called, connectivity checks will be sent tosession description void setRemoteDescription ( SessionDescriptionType action, SessionDescription desc) // returns thenew candidates. This call will result incurrent local session description readonly attribute SessionDescription localDescription; // returns the current remote session description readonly attribute SessionDescription remoteDescription; // updates the constraints for ICE processing void updateIce ( optional DOMString configuration, optional MediaConstraints constraints); // starts using achange toreceived remote ICE candidate void addIceCandidate ( IceCandidate candidate); // notifies thestateapplication ofthe ICE Agent, and may result inachange to media state if it results in connectivity being established. A TBD exception will be thrown if |candidate| is missing or malformed. 7.new local ICE candidate attribute Function? onicecandidate; }; A.2. Example API Flows Below are several sample flows for the new PeerConnection and library APIs, demonstrating when the various APIs are called in different situations and with various transport protocols.7.1.For clarity and simplicity, the createOffer/createAnswer calls are assumed to be synchronous in these examples, whereas the actual APIs are async. A.2.1. Call using ROAP This example demonstrates a ROAP call, without the use of trickle candidates. // Call is initiated toward Answerer OffererJS->OffererUA: pc = new PeerConnection(); OffererJS->OffererUA: pc.addStream(localStream, null);OffererJS->OffererUA: pc.startIce();OffererUA->OffererJS:iceCallback(candidate, false);iceCallback(candidate); OffererJS->OffererUA: offer = pc.createOffer(null); OffererJS->OffererUA:pc.setLocalDescription(SDP_OFFER, offer.toSdp());pc.setLocalDescription("offer", offer); OffererJS->AnswererJS: {"type":"OFFER","sdp":"<offer>"}"sdp":offer } // OFFER arrives at Answerer AnswererJS->AnswererUA: pc = new PeerConnection(); AnswererJS->AnswererUA:pc.setRemoteDescription(SDP_OFFER,pc.setRemoteDescription("offer", msg.sdp); AnswererUA->AnswererJS: onaddstream(remoteStream);AnswererJS->AnswererUA: pc.startIce();AnswererUA->OffererUA:iceCallback(candidate, false);iceCallback(candidate); // Answerer accepts call AnswererJS->AnswererUA: peer.addStream(localStream, null); AnswererJS->AnswererUA: answer =peer.createAnswer(msg.offer,peer.createAnswer(msg.sdp, null); AnswererJS->AnswererUA:peer.setLocalDescription(SDP_ANSWER,peer.setLocalDescription("answer", answer); AnswererJS->OffererJS:{"type":"ANSWER","sdp":"<answer>"}{"type":"ANSWER","sdp":answer } // ANSWER arrives at Offerer OffererJS->OffererUA:peer.setRemoteDescription(ANSWER,peer.setRemoteDescription("answer", answer); OffererUA->OffererJS: onaddstream(remoteStream); // ICE Completes (at Answerer) AnswererUA->AnswererJS: onopen(); AnswererUA->OffererUA: Media // ICE Completes (at Offerer) OffererUA->OffererJS: onopen(); OffererJS->AnswererJS: {"type":"OK" } OffererUA->AnswererUA: Media7.2.A.2.2 Call using XMPP This example demonstrates an XMPP call, making use of trickle candidates. // Call is initiated toward Answerer OffererJS->OffererUA: pc = new PeerConnection(); OffererJS->OffererUA: pc.addStream(localStream, null); OffererJS->OffererUA: offer = pc.createOffer(null); OffererJS->OffererUA:pc.setLocalDescription(SDP_OFFER,pc.setLocalDescription("offer", offer); OffererJS: xmpp = createSessionInitiate(offer); OffererJS->AnswererJS: <jingle action="session-initiate"/> OffererJS->OffererUA: pc.startIce(); OffererUA->OffererJS:iceCallback(cand);onicecandidate(cand); OffererJS:createTransportInfo(cand, ...);createTransportInfo(cand); OffererJS->AnswererJS: <jingle action="transport-info"/> // session-initiate arrives at Answerer AnswererJS->AnswererUA: pc = new PeerConnection(); AnswererJS: offer = parseSessionInitiate(xmpp); AnswererJS->AnswererUA:pc.setRemoteDescription(SDP_OFFER,pc.setRemoteDescription("offer", offer); AnswererUA->AnswererJS: onaddstream(remoteStream); // transport-infos arrive at Answerer AnswererJS->AnswererUA:candidatescandidate = parseTransportInfo(xmpp); AnswererJS->AnswererUA:pc.processIceMessage(candidates); AnswererJS->AnswererUA: pc.startIce();pc.addIceCandidate(candidate); AnswererUA->AnswererJS:iceCallback(cand, ...)onicecandidate(cand) AnswererJS: createTransportInfo(cand); AnswererJS->OffererJS: <jingle action="transport-info"/> // transport-infos arrive at Offerer OffererJS->OffererUA: candidates = parseTransportInfo(xmpp); OffererJS->OffererUA:pc.processIceMessage(candidates);pc.addIceCandidate(candidates); // Answerer accepts call AnswererJS->AnswererUA: peer.addStream(localStream, null); AnswererJS->AnswererUA: answer = peer.createAnswer(offer, null); AnswererJS: xmpp = createSessionAccept(answer); AnswererJS->AnswererUA:pc.setLocalDescription(SDP_ANSWER,pc.setLocalDescription("answer", answer); AnswererJS->OffererJS: <jingle action="session-accept"/> // session-accept arrives at Offerer OffererJS: answer = parseSessionAccept(xmpp); OffererJS->OffererUA:peer.setRemoteDescription(ANSWER,peer.setRemoteDescription("answer", answer); OffererUA->OffererJS: onaddstream(remoteStream); // ICE Completes (at Answerer) AnswererUA->AnswererJS: onopen(); AnswererUA->OffererUA: Media // ICE Completes (at Offerer) OffererUA->OffererJS: onopen(); OffererUA->AnswererUA: Media7.3.A.2.3. Adding video to a call, using XMPP This example demonstrates an XMPP call, where the XMPP content-add mechanism is used to add video media to an existing session. For simplicity, candidate exchange is not shown. Note that the offerer for the change to the session may be different than the original call offerer. // Offerer adds video stream OffererJS->OffererUA: pc.addStream(videoStream) OffererJS->OffererUA: offer = pc.createOffer(null); OffererJS: xmpp = createContentAdd(offer); OffererJS->OffererUA:pc.setLocalDescription(SDP_OFFER,pc.setLocalDescription("offer", offer); OffererJS->AnswererJS: <jingle action="content-add"/> // content-add arrives at Answerer AnswererJS: offer = parseContentAdd(xmpp); AnswererJS->AnswererUA:pc.setRemoteDescription(SDP_OFFER,pc.setRemoteDescription("offer", offer); AnswererJS->AnswererUA: answer = pc.createAnswer(offer, null); AnswererJS->AnswererUA:pc.setLocalDescription(SDP_ANSWER,pc.setLocalDescription("answer", answer); AnswererJS: xmpp = createContentAccept(answer); AnswererJS->OffererJS: <jingle action="content-accept"/> // content-accept arrives at Offerer OffererJS: answer = parseContentAccept(xmpp); OffererJS->OffererUA:pc.setRemoteDescription(SDP_ANSWER,pc.setRemoteDescription("answer", answer);7.4.A.2.4. Simultaneous add of video streams, using XMPP This example demonstrates an XMPP call, where new video sources are added at the same time to a call that already has video; since adding these sources only affects one side of the call, there is no conflict. The XMPP description-info mechanism is used to indicate the new sources to the remote side. // Offerer and "Answerer" add video streams at the same time OffererJS->OffererUA: pc.addStream(offererVideoStream2) OffererJS->OffererUA: offer = pc.createOffer(null); OffererJS: xmpp = createDescriptionInfo(offer); OffererJS->OffererUA:pc.setLocalDescription(SDP_OFFER,pc.setLocalDescription("offer", offer); OffererJS->AnswererJS: <jingle action="description-info"/> AnswererJS->AnswererUA: pc.addStream(answererVideoStream2) AnswererJS->AnswererUA: offer = pc.createOffer(null); AnswererJS: xmpp = createDescriptionInfo(offer); AnswererJS->AnswererUA:pc.setLocalDescription(SDP_OFFER,pc.setLocalDescription("offer", offer); AnswererJS->OffererJS: <jingle action="description-info"/> // description-info arrives at "Answerer", and is acked AnswererJS: offer = parseDescriptionInfo(xmpp); AnswererJS->OffererJS: <iq type="result/> // ack // description-info arrives at Offerer, and is acked OffererJS: offer = parseDescriptionInfo(xmpp); OffererJS->AnswererJS: <iq type="result/> // ack // ack arrives at Offerer; remote offer is used as an answer OffererJS->OffererUA:pc.setRemoteDescription(SDP_ANSWER,pc.setRemoteDescription("answer", offer); // ack arrives at "Answerer"; remote offer is used as an answer AnswererJS->AnswererUA:pc.setRemoteDescription(SDP_ANSWER,pc.setRemoteDescription("answer", offer);7.5.A.2.5. Call using SIP This example demonstrates a simple SIP call (e.g. where the client talks to a SIP proxy over WebSockets). // Call is initiated toward Answerer OffererJS->OffererUA: pc = new PeerConnection(); OffererJS->OffererUA: pc.addStream(localStream, null);OffererJS->OffererUA: pc.startIce();OffererUA->OffererJS:iceCallback(candidate, false);onicecandidate(candidate); OffererJS->OffererUA: offer = pc.createOffer(null); OffererJS->OffererUA:pc.setLocalDescription(SDP_OFFER,pc.setLocalDescription("offer", offer); OffererJS: sip =createInvite(offer);-createInvite(offer); OffererJS->AnswererJS: SIP INVITE w/ SDP // INVITE arrives at Answerer AnswererJS->AnswererUA: pc = new PeerConnection(); AnswererJS: offer = parseInvite(sip); AnswererJS->AnswererUA:pc.setRemoteDescription(SDP_OFFER,pc.setRemoteDescription("offer", offer); AnswererUA->AnswererJS: onaddstream(remoteStream);AnswererJS->AnswererUA: pc.startIce();AnswererUA->OffererUA:iceCallback(candidate, false);onicecandidate(candidate); // Answerer accepts call AnswererJS->AnswererUA: peer.addStream(localStream, null); AnswererJS->AnswererUA: answer = peer.createAnswer(offer, null); AnswererJS: sip = createResponse(200, answer); AnswererJS->AnswererUA:peer.setLocalDescription(SDP_ANSWER,peer.setLocalDescription("answer", answer); AnswererJS->OffererJS: 200 OK w/ SDP // 200 OK arrives at Offerer OffererJS: answer = parseResponse(sip); OffererJS->OffererUA:peer.setRemoteDescription(ANSWER,peer.setRemoteDescription("answer", answer); OffererUA->OffererJS: onaddstream(remoteStream); OffererJS->AnswererJS: ACK // ICE Completes (at Answerer) AnswererUA->AnswererJS: onopen(); AnswererUA->OffererUA: Media // ICE Completes (at Offerer) OffererUA->OffererJS: onopen(); OffererUA->AnswererUA: Media7.6.A.2.6. Handling early media (e.g. 1-800-FEDEX), using SIP This example demonstrates how early media could be handled; for simplicity, only the offerer side of the call is shown. // Call is initiated toward Answerer OffererJS->OffererUA: pc = new PeerConnection(); OffererJS->OffererUA: pc.addStream(localStream, null);OffererJS->OffererUA: pc.startIce();OffererUA->OffererJS:iceCallback(candidate, false);onicecandidate(candidate); OffererJS->OffererUA: offer = pc.createOffer(null); OffererJS->OffererUA:pc.setLocalDescription(SDP_OFFER,pc.setLocalDescription("offer", offer); OffererJS: sip = createInvite(offer); OffererJS->AnswererJS: SIP INVITE w/ SDP // 180 Ringing is received by offerer, w/ SDP OffererJS: answer = parseResponse(sip); OffererJS->OffererUA:pc.setRemoteDescription(SDP_PRANSWER,pc.setRemoteDescription("pranswer", answer); OffererUA->OffererJS: onaddstream(remoteStream); // ICE Completes (at Offerer) OffererUA->OffererJS: onopen(); OffererUA->AnswererUA: Media // 200 OK arrives at Offerer OffererJS: answer = parseResponse(sip); OffererJS->OffererUA:pc.setRemoteDescription(SDP_ANSWER,pc.setRemoteDescription("answer", answer); OffererJS->AnswererJS: ACK8.A.3. Full Example Application The following example demonstrates a simple video calling application,roughly correspondingusing both trickle candidates and provisional answers to speed up call setup. // Usage: // Caller calls start(true) // Callee calls start(false) to prepare theflow in Example 7.1.call/start connecting, // and then accept() to start transmitting. var signalingChannel = createSignalingChannel(); var pc = null; varhasCandidateslocalStream =false;null; signalingChannel.onmessage = handleMessage; // Set up the call, get access to local media, // and establish connectivity. function start(isCaller) { //createCreate a PeerConnection and hook up theIceCallbackIceCallback. pc = newwebkitPeerConnection( "", function (candidate, moreToFollow) { if (!moreToFollow) { hasCandidateswebkitPeerConnection(null, null); pc.onicecandidate =true; maybeSignal(isCaller); } });function(evt) { sendMessage("candidate", evt.candidate); }; //getGet the local stream and show it in the local videoelementelement; // if we're the caller, ship off an offer once we get the stream. navigator.webkitGetUserMedia( {"audio": true, "video": true}, function(localStream)(stream) { selfView.src =webkitURL.createObjectURL(localStream); pc.addStream(localStream); maybeSignal(isCaller); }webkitURL.createObjectURL(stream); localStream = stream; if (isCaller) { pc.addStream(stream); pc.createOffer(function(sdp) { setLocalAndSendMessage("offer", sdp); }); }); //onceWhen the remote stream arrives, show it in the remote // videoelementelement. pc.onaddstream = function(evt) { remoteView.src = webkitURL.createObjectURL(evt.stream); }; } //if we'reThe callee has accepted thecaller, create and install our offer,call, attach their media // andstart candidate generation if (isCaller) { offer = pc.createOffer(null); pc.setLocalDescription(SDP_OFFER, offer); pc.startIce(); } }send a final answer. functionmaybeSignal(isCaller)accept() { //only signal onceThe addStream could also be done for the pranswer, // although that would delay the pranswer // (due to the need for user consent) pc.addStream(localStream); // assumes we havea localthe streamand local candidates if (localStreams.size() == 0 || !hasCandidates) return; if (isCaller)already pc.createAnswer(msg.sdp, function(sdp) {offer = pc.localDescription; signalingChannel.send( JSON.stringify({ "type": "offer", "sdp": offer }));setLocalAndSendMessage("answer", sdp); }); }else {//if we're the callee, generate, apply,-- internal methods -- // Apply SDP locally and send it to theanswer answer = pc.createAnswer(pc.remoteDescription, null); pc.setLocalDescription(SDP_ANSWER, answer);remote side. function setLocalAndSendMessage(type, sdp) { pc.setLocalDescription(type, sdp); sendMessage(type, sdp); } // Send a signaling message to the remote side. function sendMessage(type, obj) { signalingChannel.send( JSON.stringify({ "type":"answer",type, "sdp":answerobj })); }} signalingChannel.onmessage = function(evt)// Handle incoming signaling messages. function handleMessage(str) { var msg =JSON.parse(evt.data); if (msg.type == "offer")JSON.parse(str); switch (msg.type) { case "offer": // create the PeerConnection start(false); // feed the received offer into the PeerConnectionand // start candidate generation pc.setRemoteDescription(PeerConnection.SDP_OFFER,pc.setRemoteDescription(msg.type, msg.sdp);pc.startIce(); } else if (msg.type == "answer") {//feed thecreate provisional answerinto the PeerConnectiontocomplete setup pc.setRemoteDescription(PeerConnection.SDP_ANSWER,allow ICE/DTLS to start pc.createAnswer(msg.sdp, function(sdp) { setDirection(sdp, "recvonly"); setLocalAndSendMessage("pranswer", sdp); }); break; case "pranswer": case "answer": pc.setRemoteDescription(msg.type, msg.sdp); break; case "candidate": pc.addIceCandidate(msg.sdp); break; }9. Security Considerations TODO 10. IANA Considerations This document requires no actions from IANA. 11. Acknowledgements Harald Alvestrand, Dan Burnett, Neil Stratford, Eric Rescorla,} Appendix B. Change log 01: Added diagrams for architecture andAnant Narayanan all provided valuable feedback on this proposal. Matthew Kaufman provided the observation that keepingstateout of the browser allows a call to continue even if the page is reloaded. Adam Bergvist provided a code example that served as the basis for the example in Section 8. 12. References 12.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC3264] Rosenberg, J.machine. Added sections on forking andH. Schulzrinne, "An Offer/Answer Model with Session Description Protocol (SDP)", RFC 3264, June 2002. [RFC4566] Handley, M., Jacobson, V.,rehydration. Clarified meaning of "pranswer" andC. Perkins, "SDP: Session Description Protocol", RFC 4566, July 2006. 12.2. Informative References [RFC4568] Andreasen, F., Baugher, M.,"answer". Reworked how ICE restarts andD. Wing, "Session Description Protocol (SDP) Security Descriptions for Media Streams", RFC 4568, July 2006. [RFC5245] Rosenberg, J., "Interactive Connectivity Establishment (ICE): A Protocol for Network Address Translator (NAT) Traversal for Offer/Answer Protocols", RFC 5245, April 2010. [webrtc-api] Bergkvist, Burnett, Jennings, Narayanan, "WebRTC 1.0: Real-time Communication Between Browsers", October 2011. Available at http://dev.w3.org/2011/webrtc/editor/webrtc.html Appendix A. Open Issues - Determinemedia directions are controlled. Added list ofexceptionsparameters that can bethrown by each method. Leaning toward something like a PCException,changed in ala https://developer.mozilla.org/en/IndexedDB/IDBDatabaseException - Need callbackdescription. Updated suggested API and examples toindicate that the transport is down, e.g. ICE_DISCONNECTED or ondisconnected(). Appendix B. Change logmatch latest thinking. Suggested API and examples have been moved to an appendix. 00: Migrated from draft-uberti-rtcweb-jsep-02. Authors' Addresses Justin Uberti Google 5 Cambridge Center Cambridge, MA 02142 Email: justin@uberti.name Cullen Jennings Cisco 170 West Tasman Drive San Jose, CA 95134 USA Email: fluffy@cisco.com