DetNet J. Korhonen, Ed. Internet-Draft Intended status: Standards Track B. Varga, Ed. Expires:April 24,September 11, 2019 EricssonOctober 21, 2018March 10, 2019 DetNet MPLS Data Plane Encapsulationdraft-ietf-detnet-dp-sol-mpls-01draft-ietf-detnet-dp-sol-mpls-02 Abstract This document specifies the Deterministic Networking data planeencapsulation solutions. The described data plane solutions is appliedwhen operating over an MPLS Packet Switched Networks. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire onApril 24,September 11, 2019. Copyright Notice Copyright (c)20182019 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1. TermsusedUsed inthis documentThis Document . . . . . . . . . . . . . . . 4 2.2. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 4 3. RequirementslanguageLanguage . . . . . . . . . . . . . . . . . . . .65 4.MPLSDetNetdata plane overviewMPLS Data Plane Overview . . . . . . . . . . . . . . . 6 4.1. Layers of DetNetdata planeData Plane . . . . . . . . . . . . . . . 6 4.2.MPLSDetNetdata plane scenariosMPLS Data Plane Scenarios . . . . . . . . . . . . 74.3. Packet flow example (service protection) . . . . . . . . 11 5.4.2.1. IP Over DetNet MPLS DataConsiderations . . . . . . . .Plane Scenarios . . . . . . 9 4.2.2. IEEE 802.1 TSN Over DetNet MPLS Data Plane Scenario . 125.1. End-system specific considerations . . . .4.3. Packet Flow Example with Service Protection . . . . . . .13 5.2.14 5. DetNetdomain specific considerations .MPLS Data Plane Considerations . . . . . . . . .15 5.2.1. DetNet Layer Two Service. . . 15 5.1. End-System Specific Considerations . . . . . . . . . . . 165.2.2. DetNet Routing Service (IP over MPLS) . . . . . . . . 17 5.3. DetNet Inter-Working Function (DN-IWF) . . . . . . . . . 18 5.3.1. Networks with multiple technology segments . .5.2. Sub-Network Considerations . . .18 5.3.2. DN-IWF related considerations. . . . . . . . . . . .1917 6.MPLS-basedMPLS-Based DetNetdata plane solutionData Plane Solution . . . . . . . . . . . .2018 6.1. DetNetoverOver MPLS Encapsulation Components . . . . . . . .2018 6.2. MPLSdata plane encapsulationData Plane Encapsulation . . . . . . . . . . . . . .22 6.3.19 6.2.1. DetNetcontrol word . . .Control Word and the DetNet Sequence Number . 20 6.2.2. S-Labels . . . . . . . . . . . . . . .23 6.4. Flow Identification. . . . . . . 21 6.2.3. F-Labels . . . . . . . . . . . .24 6.5. Indication of the DetNet Payload Type. . . . . . . . . . 246.6.6.3. OAM Indication . . . . . . . . . . . . . . . . . . . . .25 6.7.26 6.4. Flow Aggregation . . . . . . . . . . . . . . . . . . . .25 6.7.1.27 6.4.1. Aggregation at the LSP . . . . . . . . . . . . . . .26 6.7.2.28 6.4.2. Aggregating DetNetflowsFlows as a new DetNet flow . . . .26 6.7.3.28 6.4.3. Simple Aggregation at the DetNetlayerLayer . . . . . . .27 6.8.29 6.5. ServiceLayerSub-Layer Considerations . . . . . . . . . . . .. . 28 6.8.1.29 6.5.1. Edgenode processingNode Processing . . . . . . . . . . . . . . . .28 6.8.2.30 6.5.2. Relaynode processingNode Processing . . . . . . . . . . . . . . . .29 6.9. Other DetNet data plane considerations31 6.6. Forwarding Sub-Layer Considerations . . . . . . . . .30 6.9.1.. . 31 6.6.1. Class of Service . . . . . . . . . . . . . . . . . .30 6.9.2.31 6.6.2. Quality of Service . . . . . . . . . . . . . . . . .31 6.9.3.32 6.6.3. Cross-DetNetflow resource aggregationFlow Resource Aggregation . . . . . . . 326.9.4.6.6.4. Layer 2addressingAddressing and QoS Considerations . . . . . . 336.9.5.6.6.5. Time Synchronization . . . . . . . . . . . . . . . .3334 7.ManagementController Plane (Management andcontrol considerations . . . . . . .Control) Considerations . . . . .34 7.1. MPLS-based data plane. . . . . . . . . . . . . . . . . . 347.1.1.7.1. S-Labelassignmentanddistribution . . . . . . . . . 34 7.1.2. Explicit routes . . . . . . . . . . . . . .F-Label Assignment and Distribution . . . . .3435 7.2. PacketreplicationReplication, Elimination, andelimination . . . . . . . . .Ordering (PREOF) . .3536 7.3.Congestion protectionContention Loss andlatency controlJitter Reduction . . . . . . . . . . 36 7.4. BidirectionaltrafficTraffic . . . . . . . . . . . . . . . . . .3637 7.5. Flowaggregation controlAggregation Control . . . . . . . . . . . . . . . .36 8.38 7.6. DetNetIP Operation overController (Control and Management) Plane Requirements . 38 8. DetNet MPLSService . . . . . . . . 36 9.Operation Over IEEE 802.1 TSNInterconnection over DetNet MPLS ServiceSub-Networks . . .37 10. DetNet MPLS Transport Layer Operation over IEEE 802.139 8.1. Mapping of TSNSub-NetworksStream ID and Sequence Number . . . . . . 41 8.2. TSN Usage of FRER . . . . . . . . . . . . . . . . . .37 10.1. Mapping of TSN Stream ID. . 42 8.3. Management andSequence NumberControl Implications . . . . . .38 10.2. TSN Usage of FRER. . . . . 42 9. DetNet MPLS Operation over DetNet IP PSNs . . . . . . . . . . . . . .40 10.3. Management and Control Implications. . . . . . . . . .40 11. DetNet MPLS Transport Layer Operation over IP DetNet PSNs. .40 12.. 43 10. SecurityconsiderationsConsiderations . . . . . . . . . . . . . . . . . . .42 13.45 11. IANAconsiderationsConsiderations . . . . . . . . . . . . . . . . . . . . .42 14.45 12. Contributors . . . . . . . . . . . . . . . . . . . . . . . .43 15.45 13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . .45 16.46 14. References . . . . . . . . . . . . . . . . . . . . . . . . .45 16.1.47 14.1. NormativereferencesReferences . . . . . . . . . . . . . . . . . .45 16.2.47 14.2. InformativereferencesReferences . . . . . . . . . . . . . . . . .4849 Appendix A. Example of DetNetdata plane operationData Plane Operation . . . . . . .5052 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . .5052 1. Introduction Deterministic Networking (DetNet) is a service that can be offered by a network to DetNet flows. DetNet provides these flows with a low packet loss rates and assured maximum end-to-end delivery latency. General background and concepts of DetNet can be found in [I-D.ietf-detnet-architecture]. The DetNet Architecture decomposes the DetNet related data plane functions into two sub-layers: a service sub-layer and a forwarding sub-layer. The service sub-layer is used to provide DetNet service protection and reordering. The forwarding sub-layer is used to provides congestion protection (low loss, assured latency, and limited reordering) leveraging MPLS Traffic Engineering mechanisms. This document specifies the DetNet data plane operation and theon-wireon- wire encapsulation of DetNet flows over an MPLS-based Packet Switched Network (PSN). The specified encapsulation provides the building blocks to enable the DetNet servicelayerand forwarding sub-layer functions andallowsupports flow identification as described in the DetNet Architecture.The DetNet transport layer functionality that provides congestion protection for DetNet flows is assumed to be in place in a DetNet node. Furthermore,As part of the service sub-layer functions, this documentalsodescribeshow DetNet flows are identified, and how aDetNetRelay/Edge/Transit nodes works.node data plane operation. It also describes the function and operation of the Packet Replication (PRF) Packet Elimination (PEF) and Packet Ordering (POF) functionsin thewith an MPLS data plane. It also describes an MPLS-based DetNet forwarding sub-layer that eliminates (or reduces) contention loss and provides bounded latency for DetNet flows. MPLS encapsulated DetNet flows can be carried over network technologies that can provide the DetNet required level of service. This document defines examples of such, specifically carrying DetNet MPLS flows over IEEE 802.1 TSN sub-networks, and over DetNet IP PSN. The intent is for this document to support different traffic types being mapped over DetNet MPLS, but this is out side the scope of this document. An example of such can be found in [I-D.ietf-detnet-dp-sol-ip]. This document also allows for, but does notdefine thedefine, associatedcontrolcontroller planefunctions, orand Operations, Administration, and Maintenance(OAM). It also does not specify traffic handling capabilities required to deliver congestion protection and latency control for DetNet flows at the DetNet transport layer.(OAM) functions. 2. Terminology 2.1. TermsusedUsed inthis documentThis Document This document uses the terminology established in the DetNet architecture[I-D.ietf-detnet-architecture][I-D.ietf-detnet-architecture], and theDetNet Data Plane Solution Alternatives [I-D.ietf-detnet-dp-alt]. T-Labelreader is assumed to be familiar with that document and its terminology. The following terminology is introduced in this document: F-Label A Detnet "forwarding" labelused to identifythat identifies the LSP used totransportforward a DetNet flow across an MPLS PSN, e.g., a hop-by-hop label used between label switching routers (LSR). S-Label A DetNet "service" label that is used between DetNet nodes that implement also the DetNet servicelayersub-layer functions. An S-Label is also used to identify a DetNet flow at DetNet servicelayer. PEFsub-layer. d-CW APacket Elimination Function (PEF) eliminates duplicate copies of packets received by an edge or a relay node to prevent excess packets flooding the network, or to preventDetNet Control Word (d-CW) is used for sequencing and identifying duplicate packetsbeing sent outofthe DetNet domain. PRF A Packet Replication Function (PRF) replicatesa DetNet flowpackets and forwards them to one or more next hops in the DetNet domain. The number of packet copies sent to each next hop is a DetNet flow specific parameter at the node doing the replication. PRF can be implemented by an edge node, a relay node, or an end system. POF A Packet Ordering Function (POF) re-orders packets within a DetNet flow that are received out of order. This function can be implemented by an edge node, a relay node, or an end system. PREOF Collective name for Packet Replication, Elimination, and Ordering Functions. d-CW A DetNet Control Word (d-CW) is used for sequencing and identifying duplicate packets of a DetNet flow atat the DetNet servicelayer.sub-layer. 2.2. Abbreviations The following abbreviations are used in this document: AC Attachment Circuit. CE Customer Edge equipment. CoS Class of Service. CW Control Word.d-CW DetNet Control Word.DetNet Deterministic Networking. DF DetNet Flow. DN-IWF DetNet Inter-Working Function. L2 Layer 2. L2VPN Layer 2 Virtual Private Network. L3 Layer 3. LSR Label Switching Router. MPLS Multiprotocol Label Switching. MPLS-TE Multiprotocol Label Switching - Traffic Engineering. MPLS-TP Multiprotocol Label Switching - Transport Profile. MS-PW Multi-Segment PseudoWire (MS-PW). NSP Native Service Processing. OAM Operations, Administration, and Maintenance. PE Provider Edge. PEF Packet Elimination Function. PRF Packet Replication Function. PREOF Packet Replication, Elimination and Ordering Functions. POF Packet Ordering Function. PSN Packet Switched Network. PW PseudoWire. QoS Quality of Service. S-PE Switching Provider Edge. T-PE Terminating Provider Edge. TSN Time-Sensitive Network. 3. RequirementslanguageLanguage The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 4.MPLSDetNetdata plane overviewMPLS Data Plane Overview 4.1. Layers of DetNetdata planeData Plane This document describes how DetNet flows are carried over MPLS networks. The DetNet Architecture, [I-D.ietf-detnet-architecture], decomposes the DetNet data plane into twolayers:sub-layers: a service sub- layer and atransport layer.forwarding sub-layer. The basic approach defined in this document supports the DetNet servicelayersub-layer based on existing pseudowire (PW) encapsulations and mechanisms, and supports the DetNettransport layerforwarding sub-layer based on existing MPLS Traffic Engineering encapsulations and mechanisms. Background on PWs can be found in [RFC3985] and [RFC3031]. Background on MPLS Traffic Engineering can be found in [RFC3272] and [RFC3209]. DetNet MPLS . .+-----------++------------+ | Service | d-CW, S-Label+-----------++------------+ |TransportForwarding |T-Label(s) +-----------+F-Label(s) +------------+ . . Figure 1: DetNetadaptationAdaptation to MPLSdata planeData Plane TheMPLSDetNet MPLS data plane approach defined in this document is shown in Figure 1. The servicelayersub-layer is supported by a DetNet control word (d-CW) which conforms to the Generic PW MPLS Control Word (PWMCW) defined in [RFC4385]. A d-CW identifying service label (S-Label) is also used.The transport layer is supported by one or labels (T-Labels).A node operating on a DetNet flow in the Detnetlayer,service sub-layer, i.e. a node processing a DetNet packet which has theS-labelS-Label as top of stack uses the local context associated with thatS-labelS-Label, for example a received F-Label, to determine what local DetNet operation(s) are applied to that packet.The S-label has toAn S-Label may be uniqueon each edge and relay node, which is achieved by using a labelwhen taken from the platform label space[RFC3031]. 4.2. MPLS[RFC3031], which would enable correct DetNet flow identification regardless of which input interface or LSP the packet arrives on. The DetNet MPLS data planescenarios TSN Edgebuilds on MPLS Traffic Engineering encapsulations and mechanisms to provide a forwarding sub-layer that is responsible for providing resource allocation and explicit routes. The forwarding sub-layer is supported by one or more forwarding labels (F-Labels). 4.2. DetNet MPLS Data Plane Scenarios DetNet MPLS Relay TransitEdge TSNRelay DetNet MPLS End System Node Node Node End System (T-PE) (S-PE) (LSR) (S-PE) (T-PE)+---------+ +.........+ +.........+ +---------++----------+ +----------+ | Appl.|<--:Svc Proxy:--End|<------------ End to EndSvc.--:Svc Proxy:-->|Service ----------->| Appl. | +----------+ +---------+ +---------++---------+ +---------+ | TSN+----------+ ||TSN| |Svc|<--Service |<--| Service |-- DetNet flow-->:--| Service |-->| Service:-->| TSN| +----------+ +---------++---+ +---+ +---------++----------+ +---------++---------+ |Transport| |Trp| |Trp| |Transport| |Trp| |Trp| |Transport| +-------.-+ +--.++----------+ |Forwarding| |Fwd| |Fwd| |Forwarding| |Fwd| |Fwd| |Forwarding| +-------.--+ +-.-++--.----.-++-.-+ +----.---.-+ +-.-++---.-----++-.-+ +---.------+ : Link : / ,-----. \ : Link : / ,-----. \ +........+ +-[ Sub ]-++........++......+ +-[ Sub ]-+ [Network] [Network] `-----' `-----' |<-TSN ->| |<-----LSP -->| |<-------- LSP -----------| |<--- LSP -->| |<----------------- DetNet MPLS---->| |<-- TSN --->|--------------------->| Figure 2: ATSN overDetNet MPLSEnabledNetwork Figure 2shows several node types defined in [I-D.ietf-detnet-architecture]. DetNet Edge Nodes sit at the boundary ofillustrates a hypothetical DetNetdomain. They are responsible for mapping non-MPLS-only network composed of DetNet awaretraffic toMPLS enabled end systems, operating over a DetNetservices. They also support the impositionaware MPLS network. In this figure, relay nodes sit at MPLS LSP boundaries anddisposition of the required DetNet encapsulation. These are functionally similar to pseudowire (PW) Terminating Provider Edge (T-PE)transit nodeswhich use MPLS-TE LSPs. Native TSN floware LSRs. DetNet end system and relay nodes are DetNetMPLS flow differ not only byservice sub-layer aware, understand theadditional MPLS specific encapsulation, butparticular needs of DetNetMPLSflowshave on each DetNet node an associatedand provide both DetNetspecific data structure, what defines flow related characteristicsservice andrequiredforwarding sub-layer functions.Edge Nodes MUST provide a Service Proxy entity that "associates" the DetNet flowsThey add, remove andnative flows atprocess d-CWs, S-Labels and F-labels as needed. MPLS enabled end system and relay nodes can enhance theedgereliability of delivery by enabling the replication of packets where multiple copies, possibly over multiple paths, are forwarded through the DetNet domain.It ensures that the DN Flow is properly served at the Edge node (and inside the domain). Transit nodes are normal MPLS Label Switching Routers (LSRs).Theyare generally unaware of the special requirementscan also eliminate surplus previously replicated copies of DetNetflows, although they need to provide traffic engineering services and proper QoS to the LSPs associated withpackets. DetNetflowsMPLS nodes provide functionality similar toenhanceT-PEs when they sit at theprospectedge of an MPLS domain, and functionality similar to S-PEs when they are in theLSPs meetingmiddle of an MPLS domain, see [RFC6073]. End system and relay nodes also include DetNet forwarding sub-layer functions, support for notably explicit routes, and resources allocation to eliminate (or reduce) congestion loss and jitter. DetNet transit nodes reside wholly within a DetNet domain, and also provide DetNet forwarding sub-layer functions in accordance with the performance required by a DetNet flow carried over an LSP. Unlike other DetNet node types, transit nodes provide no servicerequirements. Some implementations ofsub-layer processing. In a DetNet MPLS network, transit nodes may be DetNetaware, butservice aware or may be DetNet unaware MPLS Label Switching Routers (LSRs). In this latter case, suchnodes just supportLSRs would be unaware of the special requirements of the DetNettransport layer.service sub-layer, but would still provide traffic engineering services and the QoS need to ensure that the (TE) LSPs meet the service requirements of the carried DetNet flows. TheMPLS LSPLSPs may be provided by any MPLSmethod (provisioned, RSVP- TE, MPLS- TP,controller method. For example they may be provisioned via a management plane, RSVP-TE, MPLS-TP, or MPLS Segment Routing(SR)). IP DetNet Relay Transit Relay IP(when extended to support resource allocation). Figure 3 illustrates how an end to end MPLS-based DetNetEnd System Node Node Node End System (T-PE) (LSR) (T-PE) +---------+ +---------+ | Appl. |<--------------- Endservice is provided in a more detail. In this figure, the end systems, CE1 and CE2, are able toEnd Service ---------->| Appl. | +---------+ .....-----+ +-----..... +---------+ | Service |<---: Service |--send and receive MPLS encapsulated DetNetflow ---| Service :-->| Service | +---------+ +---------+ +---------+ +---------+ +---------+ |Transport| |Trp| |Trp| |Transport| |Trp| |Trp| |Transport| +-------.-+ +-.-+ +-.-+ +---.---.-+ +-.-+ +-.-+ +---.-----+ : Link : / ,-----. \ : Link : / ,-----. \ +........+ +-[ Sub ]-+ +........+ +-[ Sub ]-+ [Network] [Network] `-----' `-----' |<-DN IP->| |<---- DetNet MPLS ---->| |<-DN IP->| Figure 3: DetNet (DN) IP Over MPLS Network Figure 3flows, andFigure 4, show different cases where relay nodes may be used. Relay nodes are similar to edge nodes in that both are aware of the needs of particular DetNet flowsR1, R2 andtake care to process them in accordance with the required performance needs. They differ in that relay nodes sit within a DetNet domain while edge nodes always sit at DetNet domain boundaries. Both node types can enhance the reliability of delivery by enabling the replication of packets so that multiple copies, possibly over multiple pathsR3 areforwarded through the DetNet domain. They also reduce the impact of replication by eliminating surplus copies of DetNet packets. Relayrelay nodesmayas they sit in theboundary of an MPLS domain when the non-MPLS domain is DetNet aware. Relay nodes are functionally similar to PW S-PEs or, when at the edgemiddle ofan MPLS network, T-PEs [RFC6073]. Figure 4 illustrates how DetNet can provide services for IEEE 802.1TSN end systems, CE1 and CE2, overa DetNetenablednetwork. Theedge nodes, E1 and E2, insert and remove required DetNet data plane encapsulation. The'X' in theedge nodesend systems, and relaynode, R1, represent anodes represents potential DetNet compound flow packet replication and eliminationpoint. This conceptually parallels L2VPN services,points. In this example, service protection is supported over four DetNet member flows andcould leverage existing related solutions as discussed below. TSN |<------- End to End DetNet Service ------>| TSN Service | Transit Transit | Service TSN (AC) | |<-Tnl->| |<-Tnl->| | (AC) TSN End | V V 1 V V 2 V V | End System | +--------+ +--------+ +--------+ | System +---+ | | E1 |=======| R1 |=======| E2 | | +---+ | |--|----|._X_....|..DF1..|.._ _...|..DF3..|...._X_.|---|---| | |CE1| | | \ | | X | | / | | |CE2| | | | \_.|..DF2..|._/ \_..|..DF4..|._/ | | | +---+ | |=======| |=======| | +---+ ^ +--------+ +--------+ +--------+ ^ | Edge Node Relay Node Edge Node | | (T-PE) (S-PE) (T-PE) | | | |<-TSN-> <------- TSN Over DetNet MPLS ---------> <-TSN->| | | |<--- Emulated Time Sensitive Networking (TSN) Service --->| DFx = DetNet Flow x Figure 4: IEEE 802.1TSN over DetNet [Editor's note: TSN Over DetNet MPLS arrows extended beyond the 'X' (the PREF points).] Figure 5 illustrates how an end to end MPLS-based DetNet service is provided inTE LSPs. For amore detail. In this case, the end systems, CE1 and CE2, are able to send and receive DetNet flows, andunidirectional flow, R1andsupports PRF, R2are relay nodes as they sit in the middle of a DetNet network. For example, an end system sends data encapsulated in MPLS. The 'X' in the end systems,supports PREOF andrelay nodes represents potential DetNet flow packet replicationR3 supports PEF andelimination points. In this figure,POF. Note that the relay nodes may change the underlyingtransport,forwarding sub-layer, for example tunneling MPLS overIPIEEE 802.1 TSN Section11,8, or simply over interconnect networksegments.links. DetNet DetNet MPLS Service Transit Transit Service MPLS DetNet | |<-Tnl->| |<-Tnl->| | DetNet End | V 1 V V 2 V | End System | +--------+ +--------+ +--------+ | System +---+ | | R1 |=======| R2 |=======| R3 | | +---+ | X...DFa...|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|.DFa..|.X | |CE1|========| \ | | X | | / |======|CE2| | | | | \_.|..DF2..|._/ \__.|..DF4..|._/ | | | | +---+ | |=======| |=======| | +---+ ^ +--------+ +--------+ +--------+ ^ | Relay Node Relay Node Relay Node | | (S-PE) (S-PE) (S-PE) | | | |<---------------------- DetNet MPLS --------------------->| | | |<--------------- End to End DetNet Service -------------->|Figure 5: MPLS-Based Native-------------------------- Data Flow -------------------------> X = Optional service protection (none, PRF, PREOF, PEF/POF) DFx = DetNetFigure 6 illustratesmember flow x over a TE LSP Figure 3: MPLS-Based Native DetNet As previously mentioned, this document specifies howan endMPLS is used toend MPLS-basedsupport DetNetservice is provided where the end systems are not ableflows using an MPLS data plane as well as how such can be mapped tosendIEEE 802.1 TSN andreceiveIP DetNetflows. InPSNs. An equally import scenario is when IP is supported over DetNet MPLS and thisexample, the nodes labeled CE1is covered in [I-D.ietf-detnet-dp-sol-ip]. Another important scenario is where an Ethernet Layer 2 service is supported over DetNet MPLS andCE2 could be non-DetNet awarethis is covered in [TBD-TSN-OVER-DETNET]. 4.2.1. IProuters or hosts. Note that E1 and E2 are edge nodes as they sit boundaries of theOver DetNetenabled domain.MPLS Data Plane Scenarios [Author's note: this section to be moved to IP sol draft] IPNon Service Transit Transit Service NonDetNet|<-Tnl->| |<-Tnl->|Relay Transit Relay IP DetNet End| V 1 V V 2 V | EndSystem| +--------+ +--------+ +--------+ | System +---+ | | E1 |=======| R2 |=======| E3 | | +---+ | |--------|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|------| | |CE1| | | \ | | X | | / | | |CE2| | | | | \_.|..DF2..|._/ \__.|..DF4..|._/ | | | | +---+ | |=======| |=======| | +---+ +--------+ +--------+ +--------+ ^ EdgeNodeRelayNodeEdge Node^ |Node End System (T-PE)(S-PE)(LSR) (T-PE) +----------+ +----------+ || | <--IP-->| <-------- IP Over DetNet MPLS ---------> |<--IP--> | | |<------Appl. |<------------ End to EndDetNetService------->| Figure 6: MPLS-Based DetNet (non-MPLS End System) Figure 7 illustrates how end to end----------->| Appl. | +----------+ .....-----+ +-----..... +----------+ | Service |<--: Service |-- DetNetservice is provided where the end systems are ableflow --| Service :-->| Service | +----------+ +---------+ +----------+ +---------+ +----------+ |Forwarding| |Fwd| |Fwd| |Forwarding| |Fwd| |Fwd| |Forwarding| +-------.--+ +-.-+ +-.-+ +----.---.-+ +-.-+ +-.-+ +---.------+ : Link : / ,-----. \ : Link : / ,-----. \ +........+ +-[ Sub ]-+ +......+ +-[ Sub ]-+ [Network] [Network] `-----' `-----' |<- DN IP->| |<---- DetNet MPLS ---->| |< -DN IP ->| Figure 4: DetNet IP Over MPLS Network Figure 4 illustrates DetNet enabled End Systems (hosts), connected tosend and receiveDetNet (DN) enabled IP networks, operating over a DetNetflows, e.g., per [I-D.ietf-detnet-dp-sol-ip], andaware MPLS network. In this figure, Relay nodes sit at the boundary of the MPLS domain since the non-MPLS domain is DetNet aware. This figure is very similar to Figure 2. The primary difference is that the Relay nodesoptionallyare at the edges of the MPLS domain and therefore function as T-PEs, and that service sub-layer functions are not provided over the DetNet IP network. There is no difference in transit node function. Figure 5 illustrates how relay nodes can provide serviceprotection.protection over the MPLS domain. In thiscase R1case, CE1 andR3CE2 areT-PEsIP DetNet end systems which are interconnected via a MPLS domain such as previously shown in Figure 3. Note that R1 andR2 isR3 sit at the edges of anS-PEMPLS domain and therefore are similar to T-PEs, while R2 sits in theDetNet servicemiddle of the domain and isend-to-end.therefore similar to an S-PE. DetNet DetNet IP Service Transit Transit Service IP DetNet |<-Tnl->| |<-Tnl->| DetNet End | V 1 V V 2 V | End System | +--------+ +--------+ +--------+ | System +---+ | | R1 |=======| R2 |=======| R3 | | +---+ | |-------|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|-------| | |CE1| | | \ | | X | | / | | |CE2| | | | | \_.|..DF2..|._/ \__.|..DF4..|._/ | | | | +---+ | |=======| |=======| | +---+ ^ +--------+ +--------+ +--------+ ^ | Relay Node Relay Node Relay Node | | (T-PE) (S-PE) (T-PE) | | | |<-DN IP-> <-------- DetNet MPLS ---------------> <-DN IP->| | | |<-------------- End to End DetNet Service --------------->| -------------------------- Data Flow -------------------------> X = Service protection (PRF, PREOF, PEF/POF) DFx = DetNet member flow x over a TE LSP Figure7:5: DetNet IPoverOver DetNet(DN) MPLS 4.3. Packet flow example (service protection) An exampleMPLSDetNet network fragment and packet flow is illustrated in Figure 8. 1 1.1 1.1 1.2.1 1.2.1 1.2.2 CE1----EN1--------R1-------R2-------R3--------EN2-----CE2 \ 1.2.1 /Network IP Edge Edge IP End System Node Node End System (T-PE) (LSR) (T-PE) +----------+ +....-----+ +-----....+ +----------+ | Appl. |<--:Svc Proxy|-- E2E Service --|Svc Proxy:-->| Appl. | +----------+ +.....+---+ +---+.....+ +----------+ | IP |<--:IP : |Svc|-- IP/DN Flow ---|Svc| :IP :-->| IP | +----------+ +---+ +---+ +----------+ +---+ +---+ +----------+ |Forwarding| |Fwd| |Fwd| |Forwarding| |Fwd| |Fwd| |Forwarding| +-------.--+ +-.-+ +-.-+ +----.---.-+ +-.-+ +-.-+ +---.------+ : Link : /\1.2 /-----+,-----. \ : Link : /+------R4------------------------+ 1.2.2,-----. \ +........+ +-[ Sub ]-+ +......+ +-[ Sub ]-+ [Network] [Network] `-----' `-----' |<--- IP --->| |<----- DetNet MPLS ----->| |<--- IP --->| Figure8: Example Packet flow in6: Non-DetNet Aware IP Over DetNetEnabledMPLS NetworkInFigure8 the numbers are used6 illustrates non-DetNet enabled End Systems (hosts), connected toidentify the instance of a packet. Packet 1 is the original packet, and packets 1.1, and 1.2 are two first generation copies of packet 1. Packet 1.2.1 is a second generation copy of packet 1.2 etc. Note that these numbers never appearDetNet (DN) enabled MPLS network. It differs from Figure 4 in that thepacket,hosts and edge IP networks are notto be confused with sequence numbers, labels or any other identifier that appears in the packet. They simply indicateDetNet aware. In this case, edge nodes sit at thegeneration numberboundary of theoriginal packet so that its passage through the network fragment can be identified to the reader. Customer Equipment CE1 sends a packet into the DetNet enabledMPLSnetwork. Thisdomain since it ispacket (1). Edge Node EN1 encapsulates the packet asalso a DetNetPacketdomain boundary. The edge nodes provide DetNet service proxies for the end applications by initiating andsends it to Relay node R1 (packet 1.1). EN1 makes a copy ofterminating DetNet service for thepacket (1.2), encapsulatesapplication's IP flows. See [I-D.ietf-detnet-dp-sol-ip] for more information. Figure 7 illustrates how itand sends this copy to Relay node R4. Note that along the MPLS path from EN1is still possible toR1 there may be zero or more LSRs which,provided DetNet service protection forclarity, are not shown. The samenon-DetNet aware end systems. This figures istrue for any other path between two DetNet entities shown inbasically the same as Figure8. Relay node R4 has been configured to send one copy of5, with thepacket to Relay Node R2 (packet 1.2.1)exception that CE1 andone copy to Edge Node EN2 (packet 1.2.2). R2 receives packet copy 1.2.1 before packet copy 1.1 arrives, and, having been configured to perform packet elimination on this DetNet flow, forwards packet 1.2.1 to Relay Node R3. Packet copy 1.1 is of no further useCE2 are non-DetNet aware end systems andso is discarded by R2. Edge Node EN2 receives packet copy 1.2.2 from R4 before it receives packet copy 1.2.1 from R2 via relay Node R3. EN2 therefore strips any DetNet encapsulation from packet copy 1.2.2E1 andforwards the packet to CE2. When EN2 receives the later packet copy 1.2.1 this is discarded. The above is of course illustrative of many network scenarios that can be configured. Between a pair of relay nodes there may be one or more transportE3 are edge nodes thatsimply forwardreplace the relay nodes R1 and R3. IP IP Non Service Transit Transit Service Non DetNettraffic, but these are omitted for clarity. 5.|<-Tnl->| |<-Tnl->| DetNetMPLS Data Considerations This sectionEnd | V 1 V V 2 V | End System | +--------+ +--------+ +--------+ | System +---+ | | E1 |=======| R2 |=======| E3 | | +---+ | |--------|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|------| | |CE1| | | \ | | X | | / | | |CE2| | | | | \_.|..DF2..|._/ \__.|..DF4..|._/ | | | | +---+ | |=======| |=======| | +---+ +--------+ +--------+ +--------+ ^ Edge Node Relay Node Edge Node^ | (T-PE) (S-PE) (T-PE) | | | <--IP-->| <-------- IP Over DetNet MPLS ---------> |<--IP--> | | |<------ End to End DetNet Service ------->| X = Optional service protection (none, PRF, PREOF, PEF/POF) DFx = DetNet member flow x over a TE LSP Figure 7: MPLS-Based DetNet (non-MPLS End System) 4.2.2. IEEE 802.1 TSN Over DetNet MPLS Data Plane Scenario [Author's note: this section to be moved to TSN over mpls sol draft - TBD-TSN-OVER-DETNET] TSN Edge Transit Edge TSN End System Node Node Node End System (T-PE) (LSR) (T-PE) +----------+ +.........+ +.........+ +----------+ | Appl. |<-:Svc Proxy:--End to End Svc.--:Svc Proxy:->| Appl. | +----------+ +---------+ +---------+ +----------+ | TSN | |TSN| |Svc|<-- DetNet flow -->|Svc| |TSN| | TSN | +----------+ +---+ +---+ +----------+ +---+ +---+ +----------+ |Forwarding| |Fwd| |Fwd| |Forwarding| |Fwd| |Fwd| |Forwarding| +------.---+ +--.+ +-.-+ +---.----.-+ +--.+ +-.-+ +----.-----+ : Link : / ,-----. \ : Link : / ,-----. \ +.........+ +-[ Sub ]-+ +........+ +-[ Sub ]-+ [Network] [Network] `-----' `-----' |<- TSN ->| |<------ DetNet MPLS ------>| |<-- TSN --->| Figure 8: A TSN over DetNet MPLS Enabled Network Figure 8 shows IEEE 802.1 TSN end stations operating over a TSN aware DetNet service running over an MPLS network. DetNet Edge Nodes sit at the boundary of a DetNet domain. They are responsible for mapping non-DetNet aware L2 traffic to DetNet services. They also support the imposition and disposition of the required DetNet encapsulation. These are functionally similar to pseudowire (PW) Terminating Provider Edge (T-PE) nodes which use MPLS-TE LSPs. In this example they understand and support IEEE 802.1 TSN and are able to map TSN flows into DetNet flows. The specific of this operation are discussed in [TBD-TSN-OVER-DETNET]. Native TSN flow and DetNet MPLS flow differ not only by the additional MPLS specific encapsulation, but DetNet MPLS flows have on each DetNet node an associated DetNet specific data structure, what defines flow related characteristics and required forwarding functions. In this example, edge Nodes provide a service proxy function that "associates" the DetNet flows and native flows at the edge of the DetNet domain. This ensures that the DN Flow is properly served at the Edge node (and inside the domain). Figure 9 illustrates how DetNet can provide services for IEEE 802.1TSN end systems, CE1 and CE2, over a DetNet enabled MPLS network. Similar to Figure 6, the edge nodes, E1 and E2, insert and remove required DetNet data plane encapsulation. The 'X' in the edge nodes and relay node, R1, represent a potential DetNet compound flow packet replication and elimination point. This conceptually parallels L2VPN services, and could leverage existing related solutions as discussed below. TSN |<------- End to End DetNet Service ------>| TSN Service | Transit Transit | Service TSN (AC) | |<-Tnl->| |<-Tnl->| | (AC) TSN End | V V 1 V V 2 V V | End System | +--------+ +--------+ +--------+ | System +---+ | | E1 |=======| R1 |=======| E2 | | +---+ | |--|----|._X_....|..DF1..|.._ _...|..DF3..|...._X_.|---|---| | |CE1| | | \ | | X | | / | | |CE2| | | | \_.|..DF2..|._/ \_..|..DF4..|._/ | | | +---+ | |=======| |=======| | +---+ ^ +--------+ +--------+ +--------+ ^ | Edge Node Relay Node Edge Node | | (T-PE) (S-PE) (T-PE) | | | |<- TSN -> <------- TSN Over DetNet MPLS -------> <- TSN ->| | | |<--- Emulated Time Sensitive Networking (TSN) Service --->| X = Service protection DFx = DetNet member flow x over a TE LSP Figure 9: IEEE 802.1TSN Over DetNet 4.3. Packet Flow Example with Service Protection An example DetNet MPLS network fragment and packet flow is illustrated in Figure 10. 1 1.1 1.1 1.2.1 1.2.1 1.2.2 CE1----EN1--------R1-------R2-------R3--------EN2-----CE2 \ 1.2.1 / / \1.2 /-----+ / +------R4------------------------+ 1.2.2 Figure 10: Example Packet Flow in DetNet Enabled MPLS Network In Figure 10 the numbers are used to identify the instance of a packet. Packet 1 is the original packet, and packets 1.1, and 1.2 are two first generation copies of packet 1. Packet 1.2.1 is a second generation copy of packet 1.2 etc. Note that these numbers never appear in the packet, and are not to be confused with sequence numbers, labels or any other identifier that appears in the packet. They simply indicate the generation number of the original packet so that its passage through the network fragment can be identified to the reader. Customer Equipment CE1 sends a packet into the DetNet enabled MPLS network. This is packet (1). Edge Node EN1 encapsulates the packet as a DetNet Packet and sends it to Relay node R1 (packet 1.1). EN1 makes a copy of the packet (1.2), encapsulates it and sends this copy to Relay node R4. Note that along the MPLS path from EN1 to R1 there may be zero or more LSRs which, for clarity, are not shown. The same is true for any other path between two DetNet entities shown in Figure 10. Relay node R4 has been configured to send one copy of the packet to Relay Node R2 (packet 1.2.1) and one copy to Edge Node EN2 (packet 1.2.2). R2 receives packet copy 1.2.1 before packet copy 1.1 arrives, and, having been configured to perform packet elimination on this DetNet flow, forwards packet 1.2.1 to Relay Node R3. Packet copy 1.1 is of no further use and so is discarded by R2. Edge Node EN2 receives packet copy 1.2.2 from R4 before it receives packet copy 1.2.1 from R2 via relay Node R3. EN2 therefore strips any DetNet encapsulation from packet copy 1.2.2 and forwards the packet to CE2. When EN2 receives the later packet copy 1.2.1 this is discarded. The above is of course illustrative of many network scenarios that can be configured. Between a pair of relay nodes there may be one or more transit nodes that simply forward the DetNet traffic, but these are omitted for clarity. 5. DetNet MPLS Data Plane Considerations This section provides informative considerations related toprovidingproviding DetNet service to flows which are identified based on their header information. At a high level, the following are provided on a per flow basis: Eliminating contention loss and jitter reduction: Use of allocated resources (queuing, policing, shaping) to ensure that the congestion-related loss and latency/jitter requirements of a DetNet flow are met. Explicit routes: Use of a specific path for a flow. This limits misordering and bounds latency. Service protection: Which in the case of this document primarily relates to replication and elimination. Changing the explicit path after a failure is detected in order to restore delivery of the required DetNet service characteristics is also possible. Path changes, even in the case of failure recovery, can lead to the out of order delivery of data. Load sharing: Generally, distributing packets of the same DetNet flow over multiple paths is not recommended. Such load sharing, e.g., via ECMP or UCMP, impacts ordering and possibly jitter. Troubleshooting: For example, to support identification of misbehaving flows. Recognize flow(s) for analytics: For example, increase counters. Correlate events with flows: For example, unexpected loss. The DetNet data plane also allows for the aggregation of DetNet flows, e.g., via MPLS hierarchical LSPs, to improved scaling. When DetNet flows are aggregated, transit nodes provide service to the aggregate and not on a per-DetNet flow basis. In this case, nodes performing aggregation will ensure that per-flow service requirements are achieved. 5.1. End-System Specific Considerations Data-flows requiring DetNet service are generated and terminated on end-systems. Encapsulation depends on application and its preferences. In a DetNet MPLS domain the DN functions use the d-CWs, S-Labels and F-Labels to provide DetNet services. However, an application may exchange further flow related parameters (e.g., time- stamp), which are not provided by DN functions. Specifics related to non-MPLS DetNet end station behavior are out side the scope of this document. For example, details on support for DetNet IP data flows can be found in [I-D.ietf-detnet-dp-sol-ip]. This document is also useful for end stations that map IP flows to DetNet flows. As a general rule, DetNet MPLS domains are capable of forwarding any DetNet MPLS flows and the DetNet domain does not mandate the end- system or edge system encapsulation format. Unless there is a proxy of some form present, end-systems peer with similar end-systems using the same application encapsulation format. For example, as shown in Figure 11, IP applications peer with IP applications and Ethernet L2VPN applications peer with Ethernet L2VPN applications. +-----+ | X | +-----+ +-----+ | X | | Eth | ________ +-----+ +-----+ _____ / \ | Eth | \ / \__/ \___ +-----+ \ / \ / 0======== tunnel-1 ========0_ | \ \ | 0========= tunnel-2 =========0 / \ __/ \ +-----+ \__ DetNet MPLS domain / \ | X | \ __ / +-----+ +-----+ \_______/ \_____/ | X | | IP | +-----+ +-----+ | IP | +-----+ Figure 11: End-Systems and The DetNet MPLS Domain 5.2. Sub-Network Considerations As shown in Figure 2, MPLS nodes are interconnected by different sub- network technologies, which may include point-to-point links. Each of these need to provide appropriate service to DetNet flows. In some cases, e.g., on dedicated point-to-point links or TDM technologies, all that is required is for a DetNet node to appropriately queue its output traffic. In other cases, DetNet nodes will need to map DetNet flows to the flow semantics (i.e., identifiers) and mechanisms used by an underlying sub-network technology. Figure 12 shows several examples of header formats that can be used to carry DetNet MPLS flows over different sub-network technologies. L2 represent a generic layer-2 encapsulation that might be used on a point-to-point link. TSN represents the encapsulation used on an IEEE 802.1 TSN network, as described in Section 8. UDP/IP represents the encapsulation used on a DetNet IP PSN, as described in Section 9 . +------+ +------+ +------+ App-Flow | X | | X | | X | +-----+======+--+======+--+======+-----+ DetNet-MPLS | d-CW | | d-CW | | d-CW | +------+ +------+ +------+ |Labels| |Labels| |Labels| +-----+======+--+======+--+======+-----+ Sub-Network | L2 | | TSN | | UDP | +------+ +------+ +------+ | IP | +------+ | L2 | +------+ Figure 12: Example DetNet MPLS Sub-Network Formats 6. MPLS-Based DetNet Data Plane Solution 6.1. DetNet Over MPLS Encapsulation Components To carry DetNet over MPLS the following is required: 1. A method of identifying the MPLS payload type. 2. A method of identifying the DetNet flow group to the processing element. 3. A method of distinguishing DetNet OAM packets from DetNet data packets. 4. A method of carrying the DetNet sequence number. 5. A suitable LSP to deliver the packet to the egress PE. 6. A method of carrying queuing and forwarding indication. In this design an MPLS service label (the S-Label), similar toflowsa pseudowire (PW) label [RFC3985], is used to identify both the DetNet flow identity and the payload MPLS payload type satisfying (1) and (2) in the list above. OAM traffic discrimination happens through the use of the Associated Channel method described in [RFC4385]. The DetNet sequence number is carried in the DetNet Control word which carries the Data/OAM discriminator. To simplify implementation and to maximize interoperability two sequence number sizes areidentified based on their header information. Atsupported: a 16 bit sequence number and a 28 bit sequence number. The 16 bit sequence number is needed to support some types of legacy clients. The 28 bit sequence number is used in situations where it is necessary ensure that in highlevel,speed networks thefollowingsequence number space does not wrap whilst packets areprovided on a per flow basis: Congestion protection and latency control: Usagein flight. The LSP used to forward the DetNet packet may be ofallocated resources (queuing, policing, shaping)any type (MPLS- LDP, MPLS-TE, MPLS-TP [RFC5921], or MPLS-SR [I-D.ietf-spring-segment-routing-mpls]). The LSP (F-Label) label and/or the S-Label may be used toensureindicate the queue processing as well as the forwarding parameters. Note that thecongestion-related loss and latency/jitter requirementspossible use of Penultimate Hop Popping (PHP) means that the only label in a received label stack may be the S-Label. 6.2. MPLS Data Plane Encapsulation Figure 13 illustrates a DetNetflow are met. Explicit routes: Usedata plane MPLS encapsulation. The MPLS-based encapsulation of the DetNet flows is aspecific pathgood fit fora flow. This limits miss-ordering and latency. Service protection: Which inthecase of this document primarily relatesscenarios described in Section 4.2.1 and Section 4.2.2. Furthermore, end to end DetNet service i.e., native DetNet deployment (see Section 4.2) is also possible if DetNet end systems are capable of initiating and termination MPLS encapsulated packets. The MPLS-based DetNet data plane encapsulation consists of: o DetNet control word (d-CW) containing sequencing information for packet replication andelimination. Changingduplicate elimination purposes, and theexplicit path afterOAM indicator. o DetNet service Label (S-Label) that identifies afailure is detected in order to restore delivery ofDetNet flow at therequiredreceiving DetNet servicecharacteristicssub-layer processing node. o Zero or more Detnet MPLS Forwarding label(s) (F-Label) used to direct the packet along the label switched path (LSP) to the next service sub-layer processing node along the path. When Penultimate Hop Popping isalso possible. Path changes, evenin use there may be no label F-Label in thecase of failure recovery, can lead to the out of order delivery of data. Load sharing: Generally, distributing packets ofprotocol stack on thesame DetNet flow over multiple pathsfinal hop. o The necessary data-link encapsulation isnot recommended. Such load sharing, e.g., via ECMP or UCMP, impacts ordering and end-to-end jitter. Troubleshooting: For example,then applied prior tosupport identification of misbehaving flows. Recognize flow(s) for analytics: For example, increase counters. Correlate events with flows: For example, unexpected loss. Thetransmission over the physical media. DetNet MPLS-based encapsulation +---------------------------------+ | | | DetNet App-Flow | | Payload Packet | | | +---------------------------------+ <--\ | DetNet Control Word | | +---------------------------------+ +--> DetNet data planealso allows for the aggregation| S-Label | | MPLS encapsulation +---------------------------------+ | | F-Label(s) | | +---------------------------------+ <--/ | Data-Link | +---------------------------------+ | Physical | +---------------------------------+ Figure 13: Encapsulation of a DetNetflows, e.g., via MPLS hierarchical LSPs,App-Flow in an MPLS(-TP) PSN 6.2.1. DetNet Control Word and the DetNet Sequence Number A DetNet control word (d-CW) conforms toimproved scaling. Whenthe Generic PW MPLS Control Word (PWMCW) defined in [RFC4385]. The d-CW formatted as shown in Figure 14 MUST be present in all DetNetflows are aggregated, transit nodes may have limited abilitypackets containing app-flow data. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0 0 0 0| Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 14: DetNet Control Word (bits 0 toprovide service on per-flow3) Per [RFC4385], MUST be set to zero (0). Sequence Number (bits 4 to 31) An unsigned value implementing the DetNetidentifiers. Therefore, identifyingsequence number. A separate sequence number space MUST be maintained by the the node that adds the d-CW for eachindividualDetNetflow on a transit node may notapp-flow. The following sequence number field lengths MUST beachieved in some network scenarios, but DetNet service can stillsupported: 0 bits 16 bits 28 bits The sequence number length MUST beassured in these scenarios through resource allocation and control. 5.1. End-system specific considerations Data-flows requiring DetNet service are generated and terminated on end-systems. Encapsulation dependsprovisioned (configured) onapplication and its preferences. In a DetNet (or evenaTSN) domainper app-flow basis via configuration, e.g., theDN (TSN) functions use at most two flow parameters, namely Flow-ID and Sequence Number. However, an application may exchange further flow related parameters (e.g., time-stamp), which are not considered by DN functions. Two types of end-systems are distinguished: o L2 (Ethernet) end-system: application directly over L2. o L3 (IP) end-system: application over L3. Note: An MPLS DetNet end system (as showncontroller plane described inFigure 5) can be treated as a combination of an L3 (IP) end-system and an MPLS DetNet edge node. In case of Ethernet end-systems the application dataSection 7. A 0 bit sequence number field length indicates that there isencapsulated directly in L2. From the DN domain perspectivenoupper layer protocols are visible. The Data-flow uses only Ethernet tag(s) and further flow specific parameters (if needed) are hidden insideDetNet sequence number used for theprotocol data unit (PDU). The IP end-system scenario is different. In this case, data-flows are encapsulated directly in IP and, typically, other higher layer protocols such as UDP and Real-time Transport Protocol (RTP). Many valid combinations exist and it is up to applications to select specific headers toflow. When the length is zero, the sequence number field MUST beused. Detailsset to zero (0) onsupportall packets sent for the flow. When the sequence number field length is 16 or 28 bits forDetNet IP data flows can be found in [I-D.ietf-detnet-dp-sol-ip]. Asageneral rule, DetNet domainsflow, the sequence number MUST becapable of forwarding any Data-flows andincremented by one for each new app-flow packet sent. When theDetNet domainfield length is 16 bits, d-CW bits 4 to 15 MUSTNOT mandate the end-system encapsulation format. Furthermore, no application-level-proxy functionbe set to zero (0). This values carried in this field can wrap and it isenvisioned inside the DetNet domain, so end-systems peer with end-systems usingimportant to note that zero (0) is a valid field value. For example, were thesame application encapsulation format (see figure below): o L2 end-systems peer with L2 end-systems and o L3 end-systems peer with L3 end-systems. +-----+ | X | +-----+ +-----+ | X | | Eth | ________ +-----+ +-----+ _____ / \ | Eth | \ / \__/ \___ +-----+ \ / \ / 0======== tunnel-1 ========0_ | \ \ | 0========= tunnel-2 =========0 / \ __/ \ +-----+ \__ DetNet domain / \ | X | \ __ / +-----+ +-----+ \_______/ \_____/ | X | | IP | +-----+ +-----+ | IP | +-----+ Figure 9: End-systems andsequence number size is 16 bits, theDetNet domain 5.2. DetNet domain specific considerations From a connection type perspective, three scenarios are distinguished:sequence will contain: 65535, 0, 1.Directly attached: end-systemIn this case, zero (0) is an ordinary sequence number. This differs from [RFC4448] where a sequence number of zero (0) does not indicate that no sequence number field value isdirectly connected to an edge node. 2. Indirectly attached: end-systemin use. The sequence number isbehindoptionally used during receive processing as described below in Section 6.2.2.1 and Section 6.2.2.2. 6.2.2. S-Labels App-flow identification at a(L2-TSN / L3-DetNet) sub-network. 3. DN integrated: end-systemDetNet service sub-layer ispart ofrealized by an S-Label. Each app-flow MUST be sent by the node that adds a d-CW with a single specific S-Label value. MPLS-aware DetNetdomain. L3 end-systems may use any of these connection types, however L2 end-end systemsmay use only the first two (directly or indirectly attached). DetNet domainand edge nodes, which are by definition MPLS ingress and egress nodes, MUSTallow communication between any end-systems of the same type (L2-L2, L3-L3), independent of their connection typeadd andDetNet capability. However directly attachedremove the d-CW andindirectly attached end-systems have no knowledge aboutS-Label. Relay nodes MAY swap S-Label values when processing an app-flow. The S-Label value MUST be provisioned per app-flow via configuration, e.g., via the controller plane described in Section 7. Note that S-Labels provide app-flow identification at the downstream DetNetdomainservice sub-layer receiver, not the sender. As such, S-Labels MUST be allocated by the entity that controls the service sub-layer receiving node's label space, andits encapsulation formatMAY be allocated from the platform label space [RFC3031]. The S-Label will normally be atall. See Figure 10 for L3 end-system scenarios. ____+----+ +----+ _____ / | ES3| | ES1|____ / \__/ +----+___ +----+ \ / \ + | ____ \ _/ +----+ __/ \ +__ DetNet domain / | ES2|____/ L2/L3 |___/ \ __ __/ +----+ \_______/ \_______/ \___/ Figure 10: Connection typesthe bottom ofL3 end-systems 5.2.1. DetNet Layer Two Service The simplest DetNet service is to provide tunneling for layer two, wheretheconnected hosts arelabel stack, immediately preceding the d-CW. To support service sub-layer level OAM, an OAM Associated Channel Header (ACH) [RFC4385] together with a Generic Associated Channel Label (GAL) [RFC5586] MAY be used in place of a d-CW. Similarly, an Entropy Label Indicator/Entropy Label (ELI/EL) [RFC6790] MAY be carried below thesame broadcast (BC) domain. Forwarding overS-Label in the label stack in networks where DetNetdomain is based on L2 (MAC) addresses (i.e. dst-MAC), or onflows would otherwise receivedinterface [RFC3985]. In both casesECMP treatment. When ELs are used, theL2 headers MUST either be kept, or provision mustsame EL value SHOULD bemadeused fortheir reconstruction at egress fromall of theDetNet domain. +------+ | X | +------+ +------+ | X | | IP | +------+ +------+ End-system | L2 | | L2 | +-----+======+-+======+--+------+ DetNet tunnel | Shim | +------+ | MPLS | +------+ | L2 | +------+ Examples: +------+ +------+ | X | | X | +------+ +------+ +------+ | X | | IP | | IP | +------+ +------+ +------+ | L2 | | L2 | | L2 | +-----+======+--+======+--+======+-----+ | d-CW | | d-CW | | d-CW | +------+ +------+ +------+ | MPLS | | MPLS | | MPLS | +------+ +------+ +------+ | L2 | | L2 | | UDP | +------+ +------+ +------+ | IP | +------+ | L2 | +------+ Figure 11: Encapsulation format for DetNet Layer Two Service As shown in Figure 11 both L2 and L3 end-systems can be served by suchpackets sent using a specific S-Label to force the flow to follow the same path. However, as previously stated in Section 5, the use of ECMP for DetNetL2 encapsulation service. This encapsulation service mayflows is NOT RECOMMENDED. ECMP MAY becarried over MPLS natively Section 6.2, or over MPLS over IP Section 11. 5.2.2.used for non- DetNetRouting Service (IP over MPLS) IP traffic and IPflows within a DetNetflows, see [I-D.ietf-detnet-dp-sol-ip], can be carried overdomain. When receiving a DetNet MPLSdomain.flow, an implementation MUST identify the app-flow associated with the incoming packet based on the S-Label. When a node is using platform labels for S-Labels, no additional information is needed as the S-label uniquely identifies the app-flow. Insuch cases,theIP headerscase where platform labels aremodified per standard router behavior, e.g., TTL handling. Figure 12 showsnot used, one or more F-Labels proceeding theencapsulation ofS-Label MUST be used together with the S-Label to uniquely identify the incoming app-flows. When PHP is used, the incoming interface MAY also be used to together with any present F-Label(s) and the S-Label to uniquely identify anIP flow over MPLS as well as when MPLSincoming app-flows. Note that choice to use platform label space for S-Label or S-Label plus one or more F-Labels to identify app flows iscarried overa local implementation choice, with one caveat. When one or more F-labels, or incoming interface, is needed together with anIP PSN, see Section 11. +------+ | X | +------+ End-system | IP | +-----+======+--+------+S-Label to uniquely identify, the controller plane MUST ensure that incoming DetNettunnel | Shim | +------+ | MPLS | +------+ | L2 | +------+ Examples: +------+ +------+ | X | | X | +------+ +------+ | IP | | IP | +-----+======+--+======+-----+ | d-CW | | d-CW | +------+ +------+ |MPLS| | MPLS | +------+ +------+ | L2 | | UDP | +------+ +------+ | IP | +------+ | L2 | +------+ Figure 12: Encapsulation formatpackets arrive with the needed information (F-label(s) and/or incoming interface); the details of such are outside the scope of this document. While NOT REQUIRED, the use of platform labels forDetNet RoutingS-Labels matches other pseudowire encapsulations. This implementation choice also impacts PEF and POF processing as described inMPLS PSN for L3 end-systems 5.3. DetNet Inter-Workingthe next section. 6.2.2.1. Packet Elimination Function(DN-IWF) 5.3.1. Networks with multiple technology segments There are networking scenarios, whereProcessing Implementations MAY support the Packet Elimination Function (PEF) for received DetNetdomain contains multiple technology segments (IP, MPLS, ..) and all those segments are underMPLS flows. When supported, use of thesame administrative control (see Figure 13). Furthermore, DetNet nodes mayPEF for a particular app-flow MUST beinterconnectedprovisioned viaTSN segments. An important aspect ofconfiguration, e.g., via the controller plane described in Section 7. After an app-flow is identified for a received DetNetnetwork designMPLS packet, as described above, an implementation MUST check if PEF is configured for that app-flow. When configured theplacement of DetNet functions acrossimplementation MUST track thedomain. Designs based on segment-by- segment optimization can provide only sub-optimal solutions. In order to achieve global optimized Inter-Working Functions (DN-IWF) can be placed at segment edge nodes, which stitch together DetNet flows across connected segments. DN-IWF maysequence number contained in received d-CWs and MUST ensure thatflow attributes are correlated across segment edges. For example, there are two DetNet functions which require Sequence Numbers: (1) PEF: removes duplications from flows and (2) POF: ensures in-order-deliveryduplicate (replicated) instances ofpacket inaflow. Stitching flows together and correlating attributes meansparticular sequence number are discarded. The specific mechanisms used forexample that replication ofan implementation to identify which received packetscan happen in one segment and elimination ofare duplicatesin a different one. ______ ____ / \__ ____ / \__/ \___ ______ +----+ __/ +======+ +==+ \ +----+ |src |__/ Seg1 ) | | \ Seg3 \____| dst| +----+ \_______+ \ Segment-2 | \+_____/ +----+ \======+_ _+===/ \ __ __/ \_______/ \___/ +------------+ +---------------E----+ | | +----+ | | +----R---+ | +----+ |src |-------R +---+ | E----------+ dst| +----+ | | E--------+ +----+ +-----------R | +-----------------+ Figure 13: Optimal replicationandelimination placement across technology segments example 5.3.2. DN-IWF related considerations The goal of DN-IWFwhich are new isto (1) matchan implementation choice. Note that per Section 6.2.1 the sequence number field length may be 16 or 28 bits, and(2) translate segment specific flow attributes. The DN-IWF ensures that segment specific attributes comprise per domain unique attributes forthewhole DetNet domain. This characteristicfield value canensurewrap. Note thatDetNet functions can be basedan implementation MAY wish to constrain the maximum number sequence numbers that are tracked, on platform-wide or perdomain attributes and not per segment attributes. The two DetNet specific attributes haveflow basis. Some implementations MAY support thefollowing characteristics: o Flow-ID: it is same in all packetsprovisioning of the maximum number sequence numbers that are tracked number on either a platform-wide or per flowo Sequence Number: itbasis. 6.2.2.2. Packet Ordering Function Processing A function that is related to PEF isdifferent packet-by-packet FortheFlow-IDPacket Ordering Function (POF). Implementations MAY support POF. When supported, use of theDN-IWF can implementPOF for astatic mapping. The situationparticular app-flow MUST be provisioned via configuration, e.g., via the controller plane described by Section 7. Implementations MAY required that PEF and POF be used in combination. There ismore complicatedno requirement related to the order of execution of the Packet Elimination and Ordering Functions in an implementation. After an app-flow is identified forSequence Numbera received DetNet MPLS packet, asit is different packet-by-packet, so it may need more sophisticated translation unless its formatdescribed above, an implementation MUST check if POF isexactly the same in the two technology segments. In this later caseconfigured for that app-flow. When configured theDN-IWF can simple copyimplementation MUST track theSequence Number field betweensequence number contained in received d-CWs and MUST ensure that packets are processed in thetunneling encapsulation oforder indicated in thetwo technology segments. In case of three technology segments (IP, MPLS and TSN) three DN-IWF functions canreceived d-CW sequence number field, which may not bespecified. Inin therest of this sectionorder the packets are received. As defined in Section 6.2.1 thefocussequence number field length may be 16 or 28 bits, ison theincremened by one (1)IP - MPLS network scenario. Note:for each new app-flow packet sent, and theuse-cases are out- of-scopefield value can wrap. The specific mechanisms used for(2) TSN - IP, (3) TSN - MPLS. Simplestan implementationof DN-IWF is provided if the flow attributes haveto identify thesame format. Such a common denominatororder ofthe tunnel encapsulation format is the pseudowire encapsulation over both IP and MPLS. +--------+ | | | X X | | ____ | | / \ | | | |/\/\/\/\| Oops! 404 Not Found Figure 14: FIGURE Placeholder PW over X [Editor's note: Wherereceived packets isthe text describing how 802.1 TSN Streams are mappingan implementation choice. Note that an implementation MAY wish toDetNet services/flows. i.e., EVPN+] 6. MPLS-based DetNet data plane solution 6.1. DetNet over MPLS Encapsulation Components To carry DetNet over MPLSconstrain thefollowing is required: 1. A methodmaximum number ofidentifying the MPLS payload type. 2. A methodout ofidentifying the DetNetorder packets that can be processed, on platform-wide or per flowgroup tobasis. Some implementations MAY support theprocessing element. 3. A methodprovisioning ofdistinguishing DetNet OAMthis number on either a platform-wide or per flow basis. The number of out of order packetsfrom DetNet data packets. 4. A methodthat can be processed also impacts the latency ofcarryinga flow. 6.2.3. F-Labels F-Labels are support the DetNetsequence number. 5. A suitable LSP to deliver the packet to the egress PE. 6. A method of carrying queuing andforwardingindication. In this design an MPLS service label (the S-Label), similar to a pseudowire (PW) label [RFC3985], issub-layer. F-Labels are used toidentify both theprovide LSP-based connectivity between DetNetflow identityservice sub- layer processing nodes. 6.2.3.1. Service Sub-Layer andthe payloadPacket Replication Function Processing DetNet MPLSpayload type satisfying (1)end systems, edge nodes and(2) in the list above. OAM traffic discrimination happens throughrelay nodes may operate at theuseDetNet service sub-layer with understand of app-flows and their requirements. As mentioned above, when operating at this layer such nodes can push, pop or swap (pop then push) S-Labels. In all cases, theAssociated Channel method described in [RFC4385]. The sequence number is carried inF-Labels used for the app-flow are always replaced and this section applies. When sending a DetNetControl word which carriesflow, Zero or more F-Labels MAY be added on top of an S-Label by theData/OAM discriminator.node pushing an S-Label. TheLSP usedF-Labels totransport the DetNet packet maybe pushed when sending a particular app-flow MUST be provisioned per app-flow via configuration, e.g., via the controller plane discussed in Section 7. To allow for the omission ofany type (MPLS-LDP, MPLS-TE, MPLS-TP [RFC5921], or MPLS-SR [I-D.ietf-spring-segment-routing-mpls]).F-Labels, an implementation SHOULD also allow an outgoing interface to be provisioned. TheLSP (T-Label) label and/or the S-Label mayPacket Replication Function (PRF) function MAY beused to indicate the queue processing as well assupported by an implementation for outgoing DetNet flows. When supported, the same app-flow data will be sent over multiple outgoing forwardingparameters.sub- layer LSPs. Tosimplifysupport PRF an implementationandMUST support the setting of different sets of F-Labels. Hereto, tomaximize interoperability two sequence number sizes are supported:allow for the omission of F-Labels, an implementation SHOULD also allow multiple outgoing interfaces to be provisioned. PRF MUST NOT be used with app-flows configured with a16 bitd-CW sequence numberandfield length of 0 bits. When a28 bit sequence number. The 16 bit sequence numbersingle set of F-Labels isneeded to support some typesprovisioned for a particular outgoing app-flow, that set oflegacy clients.F-labels MUST be pushed after the S-Label is pushed. The28 bit sequence numberoutgoing packet isusedthen forwarded as described below insituations where itSection 6.2.3.2. When a single outgoing interface isnecessary ensure that in high speed networksprovisioned, thesequence number space does not wrap whilst packets areoutgoing packet is then forwarded as described below inflight. In addition it must be possible to sendSection 6.2.3.2. When multiple sets of F-Labels or interfaces are provisioned for apacket withparticular outgoing app-flow, azero length sequence number, to supportcopy of thecase where sequence numbersoutgoing packet, including the pushed S-Label, MUST be made per F-label set and outgoing interface. Each set of provisioned F-Labels arenot required bythen pushed onto a copy of the packet. Each copy is then forwarded as described below in Section 6.2.3.2. As described in the previous section, when receiving aparticularDetNetflow. Note thatMPLS flow, an implementation identifies theconcept ofapp-flow associated with the incoming packet based on the S-Label. When azero length sequence numbernode isnot tousing platform labels for S-Labels, any F-Labels can beconfused with a sequence number of zero. For example, werepopped and thesequence number size is 16 bits,S-label uniquely identifies thesequence will contain: 65535, 0, 1.app-flow. Inthisthe casezero is an ordinary sequence number. Unlike [RFC4448] a sequence number of zero does not indicate that no sequence number is in use. Where sequence numberswhere platform labels are notin use, and thus a zero length sequence numberused, F-Label(s) MUST also be provisioned for incoming app- flows. When PHP is used, incoming interface MUST be provisioned. The provisioned information MUST then be used to identify incoming app-flows based on the combination of S-Label and F-Label(s) or incoming interface. 6.2.3.2. Common F-Label Processing All DetNet aware MPLS nodes process F-Labels as needed to meet the service requirements of the DetNet flow or flows carried inused,thesequence number field inLSPs represented by thepacketF-Labels. This includes normal push, pop and swap operations. Such processing issent as zero. The DetNet packet forwarder knows whichessentially the same type ofthese cases applies through configuration parameters associated with eachprocessing enabled for TE LSPs, although the specificDetNet flow. Note that whenservice parameters, or traffic specification, can differ. When thenetwork consists only ofDetNetenabled nodes with no aggregation, Penultimate Hop Popping (PHP) means thatservice parameters of theonly labelapp-flow or flows carried inthe label stack mayan LSP represented by an F-Label can be met by an exiting TE mechanism, theS-label. 6.2. MPLS data plane encapsulation Figure 15 illustratesforwarding sub-layer processing node MAY be a DetNetdata planeunaware, i.e., standard, MPLSencapsulation. The MPLS-based encapsulation ofLSR. Such TE LSPs may provide LSP forwarding service as defined in, but not limited to, [RFC3209], [RFC3270], [RFC3272], [RFC3473], [RFC4875], [RFC5440], and [RFC6006]. More specifically, as mentioned above, the DetNetflowsforwarding sub- layer provides explicit routes and allocated resources, and F-Labels are used to map to each. Explicit routes are supported based on the topmost (outermost) F-Label that is pushed or swapped and the LSP that corresponds to this label. This topmost (outgoing) label MUST be associated with agood fitprovisioned outgoing interface and, for non- point-to-point outgoing interfaces, a next hop LSR. Note that this information MUST be provisioned via configuration or theLayer-2 interconnect deployment cases (see Figure 4). Furthermore, end to end DetNet service i.e., native DetNet deployment (see Figure 5)controller plane. In the previously mentioned special case where there isalso possible if DetNet end systems are capable of initiating and termination MPLS encapsulated packets. The MPLS-based DetNet data plane encapsulation consists of: o DetNet control word (d-CW) containing sequencing information for packet replication and duplicate elimination purposes,no added F-labels and theOAM indicator. Thereoutgoing interface is not a point-to-point interface, the outgoing interface MUST also be associated with aseparate sequence number space fornext hop LSR. Resources may be allocated in a hierarchical fashion per LSP that is represented by eachDetNet flow. o DetNetF-Label. Each LSP MAY be provisioned with a serviceLabel (S-label)parameters thatidentifies a DetNet flow todictates thepeer node that isspecific traffic treatment toprocess it. The S-Label is allocated frombe received by theplatform label space [RFC3031]. o Zero or more MPLS transporttraffic carried over that LSP. Implementations of this document MUST ensure that traffic carried over each LSPlabel(s) (T-label)represented by an F-Label receives the traffic treatment provisioned for that LSP. Typical mechanisms used todirect the packet along the label switched path (LSP)provide different treatment to different flows includes thenext peer node along the path. When Penultimate Hop Popping is in use there mayallocation of system resources (such as queues and buffers) and provisioning or related parameters (such as shaping, and policing). Support can also beno label T-labelprovided via an underlying network technology such IEEE802.1 TSN Section 8. Other than in theprotocol stack on the final hop. o The necessary data-link encapsulation is then applied prior to transmission overTSN case, thephysical media. DetNet MPLS-based encapsulation +---------------------------------+ | | |specific mechanisms used by a DetNetFlow | | Payload Packet | | | +---------------------------------+ <--\ |node to ensure DetNetControl Word | | +---------------------------------+ +-->service delivery requirements are met for supported DetNetdata plane | S-Label | | MPLS encapsulation +---------------------------------+ <--/ | T-Label(s) | +---------------------------------+ | Data-Link | +---------------------------------+ | Physical | +---------------------------------+ Figure 15: Encapsulationflows is outside the scope of this document. Packets that are marked in a way that does not correspond to allocated resources, e.g., lack a provisioned F-Label, can disrupt the QoS offered to properly reserved DetNet flows by using resources allocated to the reserved flows. Therefore, the network nodes of a DetNetflow in an MPLS(-TP) PSN 6.3.network: o MUST defend the DetNetcontrol word AQoS by discarding or remarking (to an allocated DetNetcontrol word (d-CW) conforms toflow or non-competing non-DetNet flow) packets received that are not theGeneric PW MPLS Control Word (PWMCW) defined in [RFC4385] and is illustrated in Figure 16. The upper nibblesubject ofthe d-CWa completed resource allocation. o MUSTbe set to zero (0). Two sequence number sizes are supported: 16 bits and 28 bits. The sequence number size inNOT usefor the d-CW associated witha DetNetflow (S-Label) is configured either byallocated resource, e.g. acontrol planequeue ormanuallyshaper reserved foreachDetNetflow. The sequence number is aligned toflows, for any packet that does match theright (least significant bits) and unused bits MUST be set to zero (0). Eachcorresponding DetNetflowflow. o MUSThaveensure a QoS flow does not exceed itsown sequence number counter. The sequence numberallocated resources or provisioned service level, o MUST ensure a CoS flow or service class does not impact the service delivered to other flows. This requirement isincremented by onesimilar to requirement foreach new packet. As discussed in Section 6, zero is an ordinary sequence number with no special meaning. Also as discussed therein, where no sequence number is used by a particular DetNet flow,MPLS LSRs to that CoS LSPs do not impact the resources allocated to TE LSPs, e.g., via [RFC3473]. Subsequent sections provide additional considerations related to CoS (Section 6.6.1), QoS (Section 6.6.2) and aggregation (Section 6.6.3). 6.3. OAM Indication OAM follows thesequence number fieldprocedures set out in [RFC5085] with thed-CWrestriction that only Virtual Circuit Connectivity Verification (VCCV) type 1 isset to zero. The d-CW MUST always be presentsupported. As shown ina packet. In a case whereFigure 3 of [RFC5085] when thesequence number is not used (e.g., for DetNet-t-flows) a zero length sequence numberfirst nibble of the d-CW isused and0x0 thesequence number MUST be set to zero (0). 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0 0 0 0| Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 16: DetNet Control Word 6.4. Flow Identification DetNet flow identification at a DetNet service layerpayload following the d-CW isrealized by an S-label. The S-labelnormal user data. However, when the first nibble of the d-CW isallocated from0X1, theplatform label space [RFC3031] which meanspayload that follows theDetNet flowd-DW iscorrectly identified and matched toan OAM payload with theflow parameters, includingOAM type indicated by theflow history, regardless of which input interfacevalue in thepacket arrives on.d-CW Channel Type field. TheS-label MUST be at the bottom label of the label stackreader is referred to [RFC5085] for aDetNet- s- or DetNet-st-flowmore detailed description of the Associated Channel mechanism, andMUST precedeto thed-CW. The S-labelDetNet work on OAM fora specificmore information DetNetflow is uniqueOAM. 6.4. Flow Aggregation [Author's note: need to revisit this section and ensure that we cover (and fully specify) desired types of aggregation.] 1. Aggregate at the LSP (Forwarding) 2. Aggregating DetNetflow onflows as aspecific node, but is not required to be identical with the S-label for thatnew DetNet flowin any other node within3. Simple Aggregation at the DetNetdomain. Thuslayer The resource control and management aspects of aggregation (including theS-label can onlyqueuing/shaping/ policing implications) will beusedcovered in other documents. The ability toidentifyaggregate individual flows, and their associated resource control, into a larger aggregate is an important technique for improving scaling of control in the data, management and control planes. The DetNetflow atdata plane allows for theintended receiving node. 6.5. Indicationaggregation oftheDetNetPayload Type The only nodes that needsflows, toknow the payload typeimproved scaling. There are three methods ofaintroducing floware the DetNet ingress node and the DetNet egress nodes.aggregation: [Editor's note:] Theingress node hasfollowing review comments were received when this section was committed toknow howgithub. General comment: We should points toprocessthepacket it receives frommajor issue of aggregation, namely theingress AC or IP flow,Seq.Num related problem. The aggregated flows have their own Seq.Num andthe egress edge node has to know how to prepare the packet for transmissionthose are independent. We should consider to group thenext hop. On ingressaggregation techniques as per their impact on what DetNet functions they allow on a DetNetedge node has to classify the packets into those that are for transmission as Detnet packetsflow. (E.g., aggregation without new Aggregate.Seq.Num would prohibit usage of FR, EF andthose that are for transmissionin-order- delivery function on the aggregate flow). SR based aggregation can be treated as"normal" packets at onea form ofmore lower priorities. The packet type is indicated toH-LSP aggregation. Should we differentiate them? What are theegress edge node throughdifferences? What are thevalueissues when aggregating of different payload types? Should we add an editor note on this? Simple-aggregation-at-the-detnet-layer: is this not theS-label. Thus, whensame as H-LSP? The A-label can be treated just as an additional F-Label. [Editor's note: End of review comment.] 6.4.1. Aggregation at theegress edge node looks upLSP DetNet flows forwarded via MPLS can leverage MPLS-TE's existing support for hierarchical LSPs (H-LSPs), see [RFC4206]. H-LSPs are typically used to aggregate control and resources, they may also be used to provide OAM or protection for theS-label oneaggregated LSPs. Arbitrary levels of aggregation naturally falls out of theparameters returned isdefinition for hierarchy and thepacket typeMPLS label stack [RFC3032]. DetNet nodes whichin turn tells the egress edge node how to preparesupport aggregation (LSP hierarchy) map one or more LSPs (labels) into and from an H-LSP. Both carried LSPs and H-LSPs may or may not use thepacket for transmissionTC field, i.e., L-LSPs or E-LSPs. Such nodes will need toa next hop. The consequence of this approach isensure thatif multiple packet encapsulationstraffic from aggregated LSPs areprocessed on a node pair, each encapsulation will need its own S-Label. That is not generallyplaced (shaped/policed/ enqueued) onto the H-LSPs in aproblems, since it is anticipatedfashion thatonly one encapsulation type will be present for eachensures the required DetNetflow. Of course, if for some reasonservice is preserved. Additional details of themultiple encapsulations aretraffic control capabilities neededto supportat asingle DetNet service, multiple S-labels willDetNet-aware node may berequired for that service. Note thatcovered in theunlikely case that Ipv4new service descriptions mentioned above or in separate future documents. Management andIPv6control plane mechanisms willmapalso need to ensure that thesame DetNet flow, different S-labels will be needed to differentiate betweenservice required on theversions of IP. 6.6. OAM Indication OAM followsaggregate flow (H-LSP or DSCP) are provided, which may include theprocedures set outdiscarding or remarking mentioned in[RFC5085] withtherestriction that only Virtual Circuit Connectivity Verification (VCCV) type 1 is supported. Asprevious sections. 6.4.2. Aggregating DetNet Flows as a new DetNet flow An aggregate can be built by layering DetNet flows as shownin Figure 3 of [RFC5085] whenbelow: +---------------------------------+ | | | DetNet Flow | | Payload Packet | | | +---------------------------------+ <--\ | DetNet Control Word | | +=================================+ | | S-Label | | +---------------------------------+ +----DetNet data plane | DetNet Control Word | | MPLS encapsulation +=================================+ | | A-Label | | +---------------------------------+ | | F-Label(s) | <--/ +---------------------------------+ | Data-Link | +---------------------------------+ | Physical | +---------------------------------+ Both thefirst nibbleAggregation (A) label and the S-Label have their MPLS S bit set indicating bottom of stack, and the d-CWis 0x0 the payload followingallows thed-CWPREOF to work. It isnormal user data. However, when the first nibblea property of thed-CW is 0X1, the payloadA-label that what followsthe d-DWisan OAM payload with the OAM type indicatedd-CW followed by an S-Label. A relay node processing thevalue inA-label would not know thed-CW Channel Type field. The reader is referredunderlying payload type. This would only be known to[RFC5085] foramore detailed descriptionnode that was a peer of theAssociated Channel mechanism, and tonode imposing theDetNet work on OAMS-Label. However there is no real need formore information DetNet OAM. 6.7. Flow Aggregation 1. Aggregate atit to know theLSP (Transport) 2. Aggregating DetNet flows as a new DetNet flow 3.payload type during aggregation processing. 6.4.3. Simple Aggregation at the DetNetlayer A further method of using SRLayer Another approach would be not toperform aggregation isinclude a d-CW forfurther study. The resource control and management aspectsthe aggregated flow. This would be functionally similar to aggregation at the forwarding sub-layer using H-LSPs, but would confine knowledge of the aggregation(includingto thequeuing/shaping/ policing implications) willDetNet layer. Such an approach shares the disadvantage that PREOF operations would not becoveredpossible. OAM operation inother documents. The ability to aggregate individual flows, and their associated resource control, into a larger aggregatethis mode isan important techniqueforimproving scaling of control in the data, management and control planes. Thefurther study. +---------------------------------+ | | | DetNet Flow | | Payload Packet | | | +---------------------------------+ <--\ | DetNet Control Word | | +=================================+ | | S-Label | +----DetNet data planeallows for the aggregation of DetNet flows, to improved scaling. There are three methods of introducing flow aggregation: The following review comments were received when this section was committed to github. General comment: We should points to the major issue of aggregation, namely the Seq.Num related problem.+---------------------------------+ | MPLS encapsulation | A-Label | | +---------------------------------+ | | F-Label(s) | <--/ +---------------------------------+ | Data-Link | +---------------------------------+ | Physical | +---------------------------------+ 6.5. Service Sub-Layer Considerations Theaggregated flows have their own Seq.Numedge andthose are independent. We should considerrelay node internal procedures related togroup the aggregation techniques as per their impact on what DetNet functions they allow onPREOF are implementation specific. The order of aDetNet flow. (E.g., aggregation without new Aggregate.Seq.Num would prohibit usagepacket elimination or replication is out ofFR, EF and in-order- delivery function on the aggregate flow). SR based aggregation canscope in this specification. However, care should betreatedtaken that the replication function does not actually loopback packets asa form of H-LSP aggregation. Should we differentiate them? What are"replicas". Looped back packets include artificial delay when thedifferences? What arenode that originally initiated theissues when aggregating of different payload types? Should we add an editor note on this? Simple-aggregation-at-the-detnet-layer:packet receives it again. Also, looped back packets may make the network condition to look healthier than it actually isthis(in some cases link failures are not reflected properly because looped back packets make thesame as H-LSP? The A-label can be treated just as an additional T-label. End of review comment. 6.7.1. Aggregation atsituation appear better than it actually is). It is important that theLSPDetNetflows transported via MPLS can leverage MPLS-TE?s existing support for hierarchical LSPs (H-LSPs), see [RFC4206]. H-LSPs are typically used to aggregate control and resources, they may also be used to provide OAM or protection for the aggregated LSPs. Arbitrary levels of aggregation naturally falls out oflayer is configured such that a DetNet node never receives its own replicated packets. If it were to receive such packets thedefinition for hierarchy andreplication function would make theMPLS label stack [RFC3032]. DetNet nodes which support aggregation (LSP hierarchy) map one orloop moreLSPs (labels) into and from an H-LSP. Both carried LSPs and H-LSPs may or may not usedestructive of bandwidth than a conventional unicast loop. Ultimately theTC field, i.e., L-LSPs or E-LSPs. Such nodesTTL in the S-Label willneed to ensure that traffic from aggregated LSPs are placed (shaped/policed/ enqueued) ontocause theH-LSPs inpacket to die during afashion that ensurestransient, but given the sensitivity of applications to packet latency the impact on therequiredDetNetserviceapplication would be severe. 6.5.1. Edge Node Processing An edge node ispreserved. Additional details ofresposable for matching ingress packets to thetraffic control capabilities needed at a DetNet-awareservice they require and encapsulating them accordingly.An edge node maybe coveredparticipate in thenew service descriptions mentioned above or in separate future documents. Managementpacket replication andcontrol plane mechanisms will also need to ensure thatduplication elimination. The DetNet-aware forwarder selects theservice requiredegress DetNet member flow segment based on theaggregateflow(H-LSP or DSCP) are provided, which may include the discarding or remarking mentioned in the previous sections. 6.7.2. Aggregating DetNet flows as a newidentification. The mapping of ingress DetNet member flowAn aggregate can be built by layering DetNet flows as shown below: +---------------------------------+ | | | DetNet Flow | | Payload Packet | | | +---------------------------------+ <--\ | DetNet Control Word | | +=================================+ | | S-Label | | +---------------------------------+ +----DetNet data plane |segment to egress DetNetControl Word | | MPLS encapsulation +=================================+ | | A-Label | | +---------------------------------+ <--/ | T-Label(s) | +---------------------------------+ | Data-Link | +---------------------------------+ | Physical | +---------------------------------+ Bothmember flow segment may be statically or dynamically configured. Additionally theAggregation (A) labelDetNet- aware forwarder does duplicate frame elimination based on the flow identification and theS-label have their MPLS S bit set indicating bottom of stack,sequence number combination. The packet replication is also done within the DetNet-aware forwarder. During elimination and thed-CW allowsreplication process thePREOFsequence number of the DetNet member flow MUST be preserved and copied towork. It isthe egress DetNet member flow. The internal design of apropertyrelay node is out of scope of this document. However theA-label that what followsreader's attention isd-CW followed by an S-label. A relay node processingdrawn to theA-label would not knowneed to make any PREOF state available to theunderlying payload type. This would onlypacket processor(s) dealing with packets to which the PREOF functions must beknownapplied, and toa nodemaintain thatwas a peer of the node imposing the S-label. However therestate isno real need forsuch as way that it is available toknowthepayload type during aggregation processing. 6.7.3. Simple Aggregation atpacket processor operation on the next packet in the DetNetlayer Another approach wouldflow (which may benot to includead-CW forduplicate, a late packet, or theaggregated flow. This would be functionally similar to aggregation atnext packet in sequence. [Editor's note: I think thetransport layer using H-LSPs, but would confine knowledgerest ofthe aggregation tothis section belongs in a new "802.1 TSN (island Interconnect) over DetNet MPLS" section.] This may be done in the DetNetlayer. Such an approach shareslayer, or where thedisadvantage that PREOF operations would notnative service processing (NSP) [RFC3985] is IEEE 802.1CB [IEEE8021CB] capable, the packet replication and duplicate elimination MAY entirely bepossible. OAM operationdone inthis mode is for further study. +---------------------------------+ | | | DetNet Flow | | Payload Packet | | | +---------------------------------+ <--\ |the NSP, bypassing the DetNetControl Word | | +=================================+ | | S-Label | +----DetNet data plane +---------------------------------+ | MPLSflow encapsulation| A-Label | | +---------------------------------+ <--/ | T-Label(s) | +---------------------------------+ | Data-Link | +---------------------------------+ | Physical | +---------------------------------+ 6.8. Service Layer Considerationsand logic entirely. This enables operating over unmodified implementations and deployments. The NSP approach works only between edge nodes and cannot make use of relaynode internal procedures relatednodes. The NSP approach is useful end to end tunnel and for for "island interconnect" scenarios. However, when there is a need to do PREOFare implementation specific. The order ofin a middle of the network, such plain edge to edge operation is not sufficient. The extended forwarder MAY copy the sequencing information from the native DetNet packetelimination or replicationinto the DetNet sequence number field and vice versa. If there isout of scopeno existing sequencing information available inthis specification. However, care should be taken thatthereplication function doesnative packet or the forwarder chose notactually loopback packets as "replicas". Looped back packets include artificial delay whento copy it from the native packet, then the extended forwarder MUST maintain a sequence number counter for each DetNet flow (indexed by the DetNet flow identification). 6.5.2. Relay Node Processing A DetNet Relay nodethat originally initiatedoperates in thepacketDetNet forwarding sub-layer . This processing is done within an extended forwarder function. Whether an ingress DetNet member flow receivesit again. Also, looped back packets may makeDetNet specific processing depends on how thenetwork conditionforwarding is programmed. Some relay nodes may be DetNet service aware, while others may be unmodified LSRs that only understand how tolook healthier than it actuallyswicth MPLS-TE LSPs. It is(in some cases link failures are not reflected properly because looped back packets makealso possible to treat thesituation appear better than it actually is). Itrelay node as a transit node, see Section 6.6.3. Again, this isimportantentirely up to how the forwarding has been programmed. 6.6. Forwarding Sub-Layer Considerations 6.6.1. Class of Service Class and quality of service, i.e., CoS and QoS, are terms that are often used interchangeably and confused with each other. In theDetNet layercontext of DetNet, CoS isconfigured suchused to refer to mechanisms that provide traffic forwarding treatment based on aggregate group basis and QoS is used to refer to mechanisms that provide traffic forwarding treatment based on a specific DetNetnode never receives its own replicated packets. If it were to receive such packetsflow basis. Examples of existing network level CoS mechanisms include DiffServ which is enabled by IP header differentiated services code point (DSCP) field [RFC2474] and MPLS label traffic class field [RFC5462], and at Layer- 2, by IEEE 802.1p priority code point (PCP). CoS for DetNet flows carried in PWs and MPLS is provided using thereplication function would makeexisting MPLS Differentiated Services (DiffServ) architecture [RFC3270]. Both E-LSP and L-LSP MPLS DiffServ modes MAY be used to support DetNet flows. The Traffic Class field (formerly theloop more destructiveEXP field) ofbandwidth than a conventional unicast loop. Ultimatelyan MPLS label follows the definition of [RFC5462] and [RFC3270]. The Uniform, Pipe, and Short Pipe DiffServ tunneling and TTL processing models are described inthe S-Label will cause the packet to die during a transient, but given the sensitivity[RFC3270] and [RFC3443] and MAY be used for MPLS LSPs supporting DetNet flows. MPLS ECN MAY also be used as defined in ECN [RFC5129] and updated by [RFC5462]. 6.6.2. Quality ofapplicationsService In addition to explicit routes, and packetlatency the impact on thereplication and elimination, described in Section 6 above, DetNetapplication would be severe. 6.8.1. Edge node processing An edge node is resposable for matching ingress packetsprovides zero congestion loss and bounded latency and jitter. As described in [I-D.ietf-detnet-architecture], there are different mechanisms that maybe used separately or in combination to deliver a zero congestion loss service. This includes Quality of Service (QoS) mechanisms at theservice they require and encapsulating them accordingly.An edge nodeMPLS layer, that mayparticipate inbe combined with thepacket replication and duplication elimination. The DetNet-aware forwarder selectsmechanisms defined by theegress DetNet memberunderlying network layer such as 802.1TSN. Quality of Service (QoS) mechanisms for flowsegment based onspecific traffic treatment typically includes a guarantee/agreement for theflow identification. The mappingservice, and allocation ofingress DetNet member flow segmentresources toegress DetNet member flow segment may be statically or dynamically configured. Additionally the DetNet- aware forwarder does duplicate frame elimination based onsupport the service. Example QoS mechanisms include discrete resource allocation, admission control, flow identification andthe sequence number combination.isolation, and sometimes path control, traffic protection, shaping, policing and remarking. Example protocols that support QoS control include Resource ReSerVation Protocol (RSVP) [RFC2205] (RSVP) and RSVP-TE [RFC3209] and [RFC3473]. Thepacket replication isexisting MPLS mechanisms defined to support CoS [RFC3270] can alsodone within the DetNet-aware forwarder. During elimination and the replication process the sequence numberbe used to reserve resources for specific traffic classes. A baseline set oftheQoS capabilities for DetNetmember flow MUST be preservedflows carried in PWs andcopied toMPLS can provided by MPLS with Traffic Engineering (MPLS-TE) [RFC3209] and [RFC3473]. TE LSPs can also support explicit routes (path pinning). Current service definitions for packet TE LSPs can be found in "Specification of theegress DetNet member flow. The internal designControlled Load Quality ofa relay node is outService", [RFC2211], "Specification ofscopeGuaranteed Quality ofthis document. However the reader's attention is drawn to the need to make any PREOF state availableService", [RFC2212], and "Ethernet Traffic Parameters", [RFC6003]. Additional service definitions are expected in future documents to support thepacket processor(s) dealing with packets to whichfull range of DetNet services. In all cases, thePREOF functions must be applied,existing label-based marking mechanisms defined for TE-LSPs and even E-LSPs are use tomaintain that state is such as way that it is available to the packet processor operation on the next packet insupport the identification of flows requiring DetNetflow (which may be a duplicate, a late packet, or the next packet in sequence.QoS. 6.6.3. Cross-DetNet Flow Resource Aggregation [Editor'snote: I think the rest ofNOTE: Isn't this sectionbelongs in a new "802.1 TSN (island Interconnect) over MPLS DetNet" section.] This may be done in the DetNet layer, or wherethenative service processing (NSP) [RFC3985] is IEEE 802.1CB [IEEE8021CB] capable,same as "Aggregation at thepacket replicationLSP". -- Address as part of aggregation section cleanup.] The ability to aggregate individual flows, andduplicate elimination MAY entirely be donetheir associated resource control, into a larger aggregate is an important technique for improving scaling of control in theNSP, bypassing the DetNet flow encapsulationdata, management andlogic entirely.control planes. Thisenables operating over unmodified implementations and deployments.document identifies the traffic identification related aspects of aggregation of DetNet flows. TheNSP approach works only between edge nodesresource control andcannot make usemanagement aspects ofrelay nodes.aggregation (including the queuing/shaping/ policing implications) will be covered in other documents. TheNSP approach is useful end to end tunnel anddata plane implications of aggregation are independent for PW/MPLS and IP encapsulated DetNet flows. DetNet flows forwarded via MPLS can leverage MPLS-TE's existing support for"island interconnect" scenarios. However, when there is a needhierarchical LSPs (H-LSPs), see [RFC4206]. H-LSPs are typically used todo PREOF in a middle of the network, such plain edgeaggregate control and resources, they may also be used toedge operation is not sufficient. The extended forwarder MAY copyprovide OAM or protection for thesequencing information fromaggregated LSPs. Arbitrary levels of aggregation naturally falls out of thenative DetNet packet intodefinition for hierarchy and the MPLS label stack [RFC3032]. DetNetsequence number fieldnodes which support aggregation (LSP hierarchy) map one or more LSPs (labels) into andvice versa. If there is no existing sequencing information available in the native packetfrom an H-LSP. Both carried LSPs and H-LSPs may orthe forwarder chosemay not use the TC field, i.e., L-LSPs or E-LSPs. Such nodes will need tocopy itensure that traffic from aggregated LSPs are placed (shaped/policed/ enqueued) onto thenative packet, then the extended forwarder MUST maintain a sequence number counter for each DetNet flow (indexed by the DetNet flow identification). 6.8.2. Relay node processing A DetNet Relay node operatesH-LSPs in a fashion that ensures the required DetNettransport layer . This processingservice isdone within an extended forwarder function. Whether an ingress DetNet member flow receives DetNet specific processing depends on howpreserved. [NOTE: This needs to be revised:] Additional details of theforwarding is programmed. Some relay nodestraffic control capabilities needed at a DetNet-aware node may beDetNetcovered in the new serviceaware, while others may be unmodified LSRs that only understand how to swicth MPLS-TE LSPs. It isdescriptions mentioned above or in separate future documents. Management and control plane mechanisms will alsopossibleneed totreatensure that therelay node as a transit node, see Section 6.9.3. Again, this is entirely up to howservice required on theforwarding has been programmed. 6.9. Other DetNet data plane considerations 6.9.1. Class of Serviceaggregate flow (H-LSP or DSCP) are provided, which may include the discarding or remarking mentioned in the previous sections. 6.6.4. Layer 2 Addressing and QoS Considerations [Editor'snote: this section needs to updated to discuss how DetNet service is mapped to E-NOTE: review andL-LSPs. Perhapssimplify thisgets merged withsection. Doesn't this belong in theaggregation section or dropped?] Class and qualityTSN section? Alternatively, describe in generic/non sub-network technology specific terms.] The Time-Sensitive Networking (TSN) Task Group ofservice, i.e., CoS and QoS, are terms that are often used interchangeably and confused with each other. InthecontextIEEE 802.1 Working Group have defined (and are defining) a number ofDetNet, CoS is used to referamendments tomechanismsIEEE 802.1Q [IEEE8021Q] that providetraffic forwarding treatment based on aggregate group basiszero congestion loss andQoS is used to refer to mechanismsbounded latency in bridged networks. IEEE 802.1CB [IEEE8021CB] defines packet replication and elimination functions thatprovide traffic forwarding treatment based onshould prove both compatible with and useful to, DetNet networks. As is the case for DetNet, a Layer 2 network node such as a bridge may need to identify the specific DetNet flowbasis. Examples of existing network level CoS mechanisms include DiffServto whichis enabled by IP header differentiated services code point (DSCP) field [RFC2474] and MPLS label traffic class field [RFC5462], and at Layer- 2, by IEEE 802.1p priority code point (PCP). CoS for DetNet flows carrieda packet belongs inPWs and MPLS is provided using the existing MPLS Differentiated Services (DiffServ) architecture [RFC3270]. Both E-LSP and L-LSP MPLS DiffServ modes MAY be usedorder tosupport DetNet flows. The Traffic Class field (formerlyprovide the TSN/DetNet QoS for that packet. It also will likely need a CoS marking, such as theEXP field)priority field of anMPLS label followsIEEE Std 802.1Q VLAN tag, to give thedefinition of [RFC5462] and [RFC3270]. The Uniform, Pipe, and Short Pipe DiffServ tunneling and TTL processing models arepacket proper service. Although the flow identification methods described in[RFC3270] and [RFC3443]IEEE 802.1CB [IEEE8021CB] are flexible, andMAY be used for MPLS LSPs supporting DetNet flows. MPLS ECN MAY also be used as definedinECN [RFC5129] and updated by [RFC5462]. CoS forfact, include IP 5-tuple identification methods, the baseline TSN standards assume that every Ethernet frame belonging to a TSN stream (i.e. DetNetflows carried in IPv6flow) carries a multicast destination MAC address that isprovided usingunique to that flow within thestandard differentiated services code point (DSCP) field [RFC2474] and related mechanisms. The 2-bit explicit congestion notification (ECN) [RFC3168] field MAY also be used. One additional consideration for DetNet nodesbridged network over whichsupport CoS servicesit is carried. Furthermore, IEEE 802.1CB [IEEE8021CB] describes three methods by which a packet sequence number can be encoded in an Ethernet frame. Ensuring thatthey MUST ensure that the CoS service classes do not impactthecongestion protectionproper Ethernet VLAN tag priority andlatency control mechanismsdestination MAC address are used on a DetNet/TSN packet may require further clarification of the customary L2/L3 transformations carried out by routers and edge label switches. Edge nodes may also have toprovide DetNet QoS. This requirement is similar to requirement for MPLS LSRs to that CoS LSPs do not impactmove sequence number fields among Layer 2, PW, and IP encapsulations. 6.6.5. Time Synchronization [Editor's Note: A detailed discussion of time synchronization is outside theresources allocated to TE LSPs via [RFC3473]. 6.9.2. Qualityscope ofService Qualitythis document, and the production ofService (QoS) mechanisms for flow specific traffic treatment typically includesaguarantee/agreement forspecialist text discussing this topic is encouraged. This section will be updated/removed if such a document is available before publication of this text.] Time synchronization is important both from theservice, and allocationperspective ofresources to supportoperating theservice. Example QoS mechanisms include discrete resource allocation, admission control, flow identification and isolation, and sometimes path control, traffic protection, shaping, policing and remarking. Example protocols that support QoS control include Resource ReSerVation Protocol (RSVP) [RFC2205] (RSVP) and RSVP-TE [RFC3209]DetNet network itself and[RFC3473]. The existing MPLS mechanisms defined to support CoS [RFC3270] can alsofrom the perspective of transferring time across the network between client applications. Some clients may beused to reserve resources for specific traffic classes. In additionable toexplicit routes, and packet replication and elimination, described in Section 6 above,use the DetNetprovides zero congestion loss and bounded latencyas their provider of time andjitter. As described in [I-D.ietf-detnet-architecture], there are different mechanisms that maybe used separately or in combinationfrequency, others may require the DetNet todelivertransfer time between azero congestion loss service. These mechanisms are provided by the eitherclient clock source and a client clock user. For example, [RFC8169] describes a method of recording the packet queuing time in an MPLSor IP layers,LSR on a packet by per packet basis andmayforwarding this information to the egress edge system. This allows compensation for any variable packet queuing delay to becombined withapplied at the packet receiver. Other mechanisms for IP/MPLS networks are definedby the underlying network layerbased on IEEE Standard 1588 [IEEE1588], such as802.1TSN.ITU-T [G.8275.1] and [G.8275.2]. Abaseline setmore detailed discussion ofQoS capabilities for DetNet flows carried in PWstime synchronization is outside the scope of this document. 7. Controller Plane (Management andMPLS canControl) Considerations While management plane and control planes are traditionally considered separately, from the Data Plane perspective there is no practical difference based on the origin of flow provisioning information, and the DetNet architecture [I-D.ietf-detnet-architecture] refers to these collectively as the 'Controller Plane'. This document therefore does not distinguish between information provided byMPLS with Traffic Engineering (MPLS-TE)distributed control plane protocols, e.g., RSVP-TE [RFC3209] and[RFC3473]. TE LSPs can also support explicit routes (path pinning). Current service definitions for packet TE LSPs can be found in "Specification of[RFC3473], or by centralized network management mechanisms, e.g., RestConf [RFC8040], YANG [RFC7950], and theControlled Load Quality of Service", [RFC2211], "Specification of Guaranteed Quality of Service", [RFC2212],Path Computation Element Communication Protocol (PCEP) [I-D.ietf-pce-pcep-extension-for-pce-controller] or any combination thereof. Specific considerations and"Ethernet Traffic Parameters", [RFC6003]. Additional service definitionsrequirements for the DetNet Controller Plane areexpecteddiscussed infuture documentsSection 7.6. 7.1. S-Label and F-Label Assignment and Distribution [Editor's note - we may need additional text on resource allocation in this section.] DetNet S-Labels (see Section 6.2.2 for their definition) are similar tosupportother MPLS service labels that denote thefull rangecontents ofDetNet services. In all cases,theexisting label-based marking mechanisms defined for TE-LSPs and even E-LSPsMPLS packet payload such as a layer 2 pseudowire, an IP packet that is routed in a VPN context with a private address, or an Ethernet virtual private network (EVPN) service. S-Labels areuseexpected tosupportbe allocated in theidentification of flows requiring DetNet QoS. Packets that are marked withsame manner as any other service labels. S-Labels uniquely identify a particular DetNetClass of Service value, but that have not beenflow, and are local to thesubject of a completed reservation, can disruptnode on which the label is allocated. In theQoS offered to properly reservedDetNetflows by using resources allocated toservice sub-layer thereserved flows. Therefore,explicit route consists of thenetwork nodesset ofaRelay Nodes that the DetNetnetwork: o MUST defendflow must traverse. They can be used to identify the DetNetQoS by discarding or remarking (toflow that a packet belongs to as it traverses a particular node in anon-DetNetCoS) packets received thatdomain. Because labels arenot the subject oflocal to each node rather than being acompleted reservation. o MUST NOT useglobal identifier within a domain, they must be advertised to their upstream DetNetreserved resource, e.g.service-aware peer nodes (e.g., aqueue or shaper reserved forDetNetflows, for any packet that does not carryMPLS End System as shown in Figure 3, or a DetNetClass of Service marker. 6.9.3. Cross-DetNet flow resource aggregation [Editor's NOTE: keep and extend this section.] The ability to aggregate individual flows, and their associated resource control, into a larger aggregate is an important technique for improving scaling of controlRelay or Edge Node as shown inthe data, managementFigure 7) andcontrol planes. This document identifiesinterpreted in thetraffic identification related aspects of aggregationcontext of their received F-Label. As discussed in Section 4, the forwarding sub-layer uses one or more F-Labels to forward DetNetflows. The resource control and management aspects of aggregation (includingpackets between DetNet service-aware nodes along explicitly defined routes at thequeuing/shaping/ policing implications) will be coveredDetNet forwarding sub-layer, which inother documents. The data plane implicationsthe context ofaggregation are independent for PW/MPLS and IP encapsulated DetNet flows. DetNet flows transported viathis document is the MPLS layer. F-Labels canleverage MPLS-TE's existing support for hierarchical LSPs (H-LSPs), see [RFC4206]. H-LSPs are typically used to aggregate control and resources, they mayalsobe used toprovideOAM or protectioncontext for an S-Label. In theaggregated LSPs. Arbitrary levels of aggregation naturally falls out ofDetNet Forwarding (MPLS) sub-layer thedefinition for hierarchy andexplicit route consists of theMPLS label stack [RFC3032].set of DetNet nodes whichsupport aggregation (LSP hierarchy) map one or more LSPs (labels) intoare LSRs, links, andfrom an H-LSP. Both carried LSPspossibly link bundle members andH-LSPs may or may not use the TC field, i.e., L-LSPs or E-LSPs. Such nodes will need to ensure that traffic from aggregated LSPs are placed (shaped/policed/ enqueued) onto the H-LSPs in a fashionqueues thatensurestherequiredDetNetservice is preserved. DetNet flows transported via IP have more limited aggregation options, due to the available traffic flow identification fieldspackets ofthe IP solution. One available approach is to manage the resources associated withaDSCP identified traffic class and to map (remark) individually controlled DetNet flows onto that traffic class. This approach also requires thatflow must traverse between nodessupport aggregation ensure that traffic from aggregated LSPs are placed (shaped/policed/enqueued)ina fashion that ensurestherequiredDetNet serviceis preserved. In both the MPLSsub-layer (i.e. between a specific Edge Node andIP cases, additional details ofthetraffic control capabilities needed atnext hop Relay Node, between specific Relay Nodes, and between aDetNet-awarespecific Relay nodemay be covered in the new service descriptions mentioned above or in separate future documents. Managementandcontrol plane mechanisms will also needthe egress Edge Node. Resource allocation corresponding toensure thattheservice required onset of Services supported over theaggregate flow (H-LSP or DSCP) are provided,forwarding sub-layer, which mayinclude the discardingorremarking mentioned in the previous sections. 6.9.4. Layer 2 addressing and QoS Considerations [Editor's NOTE: review and simplifymay not include aggregation, is required at thissection.] The Time-Sensitive Networking (TSN) Task Group of the IEEE 802.1 Working Group have defined (andsub-layer. Explicit routes aredefining) a number of amendmentsused toIEEE 802.1Q [IEEE8021Q] that provide zero congestion loss and bounded latency in bridged networks. IEEE 802.1CB [IEEE8021CB] defines packet replication and elimination functionsensure thatshould prove both compatible with and useful to, DetNet networks. As ispackets are routed through thecaseresources that have been reserved forDetNet, a Layer 2 network node such as a bridge may need to identify the specific DetNet flow to which a packet belongs in order tothem, and hence provide theTSN/DetNet QoS for that packet. It also will likely need a CoS marking, such asDetNet application with thepriority field ofrequired service. Multiple F-Labels may be pushed after anIEEE Std 802.1Q VLAN tag,S-Label and there is no requirement for all F-Labels togive the packet proper service. Althoughbe controlled via theflow identification methods describedsame controller mechanisms. For example inIEEE 802.1CB [IEEE8021CB]EVPN, some labels areflexible,distributed using BGP while others are distributed using LDP or RSVP. Whether configuring, calculating and instantiating these routes is a single-stage or multi-stage process, or infact, include IP 5-tuple identification methods, the baseline TSN standards assume that every Ethernet frame belonging toaTSN stream (i.e. DetNet flow) carriescentralized or distributed manner, is out of scope of this document. There are amulticast destination MAC addressnumber of approaches thatis uniquecould be used tothat flow withinprovide explicit routes and resource allocation in thebridged network over which it is carried. Furthermore, IEEE 802.1CB [IEEE8021CB] describes three methodsMPLS layer: o The path could be explicitly set up by a controller which calculates the path and explicitly configures each node along that path with the appropriate forwarding and resource allocation information. o The path could be set up using RSVP-TE signaling. o The path could be implemented using MPLS-based segment routing when extended to support resource allocation. See Section 7.6 for further discussion of these alternatives. Much like other MPLS labels, there are apacket sequencenumbercan be encoded inof alternatives available for DetNet S-Label and F-Label advertisement to anEthernet frame. Ensuringupstream peer node. These include distributed signaling protocols such as RSVP-TE, centralized label distribution via a controller that manages both theproper Ethernet VLAN tag prioritysender anddestination MAC address are used on a DetNet/TSN packet may require further clarificationthe receiver using NETCONF/YANG, BGP, PCEP, etc., and hybrid combinations of thecustomary L2/L3 transformations carried out by routerstwo. The details of the controller plane solution required for the label distribution andedgethe management of the labelswitches. Edge nodes may also have to move sequencenumberfields among Layer 2, PW, and IPv6 encapsulations. 6.9.5. Time Synchronization [Editor's Note: A detailed discussionspace are out oftime synchronizationscope of this document, but as mentioned above, there are particular DetNet considerations and requirements that are discussed in Section 7.6. 7.2. Packet Replication, Elimination, and Ordering (PREOF) The controller plane protocol solution required for managing the PREOF processing is outside the scope of thisdocument, and the production of a specialist text discussing this topic is encouraged. This section willdocument. That said, it should beupdated/removed if such a document is available before publication of this text.] Time synchronization is important both fromnoted that theperspective of operatingability to determine, for a particular flow, optimal packet replication and elimination points in the DetNetnetwork itselfdomain requires explicit support. There are be capabilities that can be used, or extended, for example GMPLS end-to-end recovery [RFC4872] andfrom the perspective of transferring time acrossGMPLS segment recovery [RFC4873]. 7.3. Contention Loss and Jitter Reduction As discussed in Section 1, this document does not specify thenetwork between client applications. Some clients may be ablemechanisms needed touse theeliminate contention loss or reduce jitter for DetNetas their provider of time and frequency, others may requireflows at the DetNet forwarding sub-layer. The ability totransfer time between a client clock source and a client clock user. For example, [RFC8169] describes a method of recording the packet queuing time in an MPLS LSR on a packet by per packet basismanage node andforwarding this informationlink resources tothe egress edge system. This allows compensation for any variable packet queuing delaybe able to provide these functions will beapplied at the packet receiver. Other mechanisms for IP/MPLS networks are defined based on IEEE Standard 1588 [IEEE1588], such as ITU-T [G.8275.1] and [G.8275.2]. A more detailed discussiona necessary part oftime synchronization is outsidethescope of this document. 7. Management and control considerations [Editor's note: This section needsDetNet controller plane. It will also be necessary to bedifferentable to control the required queuing mechanisms used to provide these functions along a flow's path through the network. See Section 7.6 for further discussion of these requirements. 7.4. Bidirectional Traffic Some DetNet applications generate bidirectional traffic. Using MPLSand IP solutions. Most solutionsdefinitions [RFC5654] there aretechnology dependant. Currently most text in this section is just a draftassociated bidirectional flows, andmay have bits that are already movedco-routed bidirectional flows. MPLS defines a point-to-point associated bidirectional LSP as consisting of two unidirectional point-to-point LSPs, one from A toother places/documents.] While management planeB andcontrol planes are traditionally considered separately, from the Data Plane perspective there is no practical difference based ontheorigin of flow provisioning information. This document therefore does not distinguish between information provided byother from B to A, which are regarded as providing acontrol plane protocol, e.g., RSVP-TE [RFC3209] and [RFC3473],single logical bidirectional forwarding path. This would be analogous of standard IP routing, orbyPWs running over two reciprocal unidirection LSPs. MPLS defines anetwork management mechanisms, e.g., RestConf [RFC8040]point-to-point co-routed bidirectional LSP as an associated bidirectional LSP which satisfies the additional constraint that its two unidirectional component LSPs follow the same path (in terms of both nodes andYANG [RFC7950]. [Editor's note: This section is a worklinks) inprogress. discuss here what kindboth directions. An important property ofenhancements are needed for DetNet and specifically for PREOF and DetNet zero congest loss and latency control. Need to coverco-routed bidirectional LSPs is that their unidirectional component LSPs share fate. In bothtraffic control (queuing)types of bidirectional LSPs, resource reservations may differ in each direction. The concepts of associated bidirectional flows andconnection control (control plane).] 7.1. MPLS-basedco-routed bidirectional flows can be applied to DetNet flows. While the MPLS data plane7.1.1. S-Label assignment and distribution [Editor's note: Outdated and needs more work.] Themust support bidirectional DetNetS-Label distribution follows the same mechanisms specified for XYZ . The details offlows, there are no special bidirectional features with respect to thecontroldata planeprotocol solution required for the label distribution andother than themanagement ofneed for thelabel number space are out of scopetwo directions ofthis document. 7.1.2. Explicit routes It is necessarya co-routed bidirectional flow toconsider explicit routes both attake theDetNet layersame path. Fate sharing andin the MPLS layer. In the DetNet layer the explicit route consists ofassociated vs co-routed bidirectional flows can be managed at theset of Relay Nodescontrol level. Note thatthethere is no stated requirement for bidirectional DetNetflow must traverse. Inflows to be supported using the same MPLSlayer the explicit route consistsLabels in each direction. DetNet's use of PREOF may increase thesetcomplexity ofLSRs, links, and possibly link bundle members and queues thatusing co-routing bidirectional flows, since if PREOF is used, then theDetNet packets of a flow must traverse between nodesreplication points in one direction would have to match theDetNet layer (i.e. between a specific Edge Node andelimination points in thenext hop Relay Node, between specific Relay Nodes,other direction, andbetween a specific Relay nodevice versa, and theegress Edge Node. This detailed steering is needed to ensure that packets are routed through the resources that have been reservedoptimal points forthem, and hence providethese functions in one direction may not match theDetNet application withoptimal points in therequired performance. Whether configuring, calculatingother. Control andinstantiating this is a multi- stage process, or a single stage process ismanagement mechanisms will need to support bidirectional flows, but the specification of such mechanisms are out of scope of this document.The one method of explicitly setting up the explicit path at the DetNet layerRelated control plan mechanisms have been defined in [RFC3473], [RFC6387] and [RFC7551]. This isthroughfurther discussed in Section 7.6. 7.5. Flow Aggregation Control Section 6.4 discusses the use of flow aggregation in DetNet. It includes flow aggregation accomplished through themanagement controller. [Editor's note: a methoduse ofsetting uphierarchical LSPs, aggregating multiple DetNet flows into agraph throughsingle new DetNet flow, and simple aggregation at the DetNetNodes usinglayer. It will be theIGP has been proposed. A reference is needed to e.g., RFC 7813 IS-IS Path Control and Reservation.] There are a numberresponsibility ofapproaches that canthe DetNet controller plane to betakenable toprovide explicit routes/pathsproperly provision the use of these mechanisms. These requirements are included in theMPLS layer: o The path can be explicitly set up bynext section. 7.6. DetNet Controller (Control and Management) Plane Requirements While themanagementdefinition of controllercalculatingplane for DetNet is out of thepathscope of this document, there are particular considerations andexplicitly configuring each node alongrequirements for such thatpath. o The LSP can be set up using RSVP-TE. Such an approach confinesresult from thepacket tounique characteristics of theexplicit path. oDetNet architecture [I-D.ietf-detnet-architecture] and data plane as defined herein. Thepath can be implemented using segment routing. Whereprimary requirements of the DetNettraffic is carried over IP Section 11controller plane are that it must be able to: o Instantiate DetNet flows in a DetNet domain (which may include some or all of explicitpaths may need to be provided in the IP layer. This is for further study. 7.2. Packetpath and PREOF replication and elimination[Editor's note: Outdatednode determination, link bandwidth reservations, node buffer andat the functional level technology independent.. but needs more work.] The control plane protocol solutionother resource reservations, specification of requiredfor managing the PREOF processing is outsidequeuing disciplines along thescope of this document. 7.3. Congestion protection and latency control [Editor's note: TBD] 7.4. Bidirectional traffic [Editor's NOTE: this section needs to be updated to have its scope limitedpath, ability tomanagement and control.] Some DetNet applications generate bidirectional traffic. Using MPLS definitions [RFC5654] there are associatedmanage bidirectional flows,and co-routed bidirectional flows. MPLS defines a point-to-point associated bidirectional LSPetc.) asconsisting of two unidirectional point-to-point LSPs, one from A to Bneeded for a flow. o Manage DetNet S-Label and F-Label allocation and distribution, when theother from BDetNet MPLS encapsulation is in use o The ability toA, which are regardedsupport DetNet flow aggregation o Advertise static and dynamic node and link resources such asproviding a single logical bidirectional transport path. This would be analogous of standard IP routing,capabilities and adjacencies to other network nodes (for dynamic signaling approaches) orPWs running over two reciprocal unidirection LSPs. MPLS defines a point-to-point co-routed bidirectional LSP as an associated bidirectional LSP which satisfies the additional constraint that its two unidirectional component LSPs followto network controllers (for centralized approaches) o Scale to handle thesame path (in termsnumber ofboth nodes and links)DetNet flows expected inboth directions. An important property of co-routed bidirectional LSPs is that their unidirectional component LSPs share fate. In both typesa domain (which may require per-flow signaling or provisioning) o Provision flow identification information at each ofbidirectional LSPs, resource allocationsthe nodes along the path, and it may differ depending on the location ineach direction. The concepts of associated bidirectional flowsthe network andco-routed bidirectional flows can be applied tothe DetNetflowsfunctionality (e.g. transit node vs. relay node). These requirements, aswell whether IPv6stated earlier, could be satisfied using distributed control protocol signaling (such as RSVP-TE), centralized network management provisioning mechanisms (such as BGP, PCEP, YANG [I-D.ietf-detnet-flow-information-model], etc.) orMPLS is used. Whilehybrid combinations of theIPv6two, andMPLS data planes must support bidirectional DetNet flows, there are no special bidirectional features with respect tocould also make use of MPLS-based segment routing. In the abstract, the results of either distributed signaling or centralized provisioning are equivalent from a DetNet data planeother than need for the two directions take the same paths. Fate sharing and associated vs co-routed bidirectionalperspective - flowscan be managed atare instantiated, explicit routes are determined, resources are reserved, and packets are forwarded through thecontrol level. Note, that there is no stated requirement for bidirectional DetNet flows to be supporteddomain using thesame IPv6 Flow Labels orMPLSLabels in each direction. Control mechanisms will need to support such bidirectional flows for both IPv6data plane. However, from a practical andMPLS, but such mechanismsimplementation standpoint, they areout of scope of this document. An example control plane solution for MPLS can be foundnot equivalent at all. Some approaches are more scalable than others in[RFC7551]. 7.5. Flow aggregation control [TBD] 8. DetNet IP Operation over DetNet MPLS Service [Editor's note: this is a place holder section. A standalone sectionterms of signaling load onoperationthe network. Some can take advantage ofIP flows over DetNet MPLS data plane. Includes RFC2119 Language.] 9. IEEE 802.1 TSN Interconnection overglobal tracking of resources in the DetNetMPLS Service [Editor's note: thisdomain for better overall network resource optimization. Some are more resilient than others if link, node, or management equipment failures occur. While a detailed analysis of the control plane alternatives is out of the scope of this document, the requirements from this document can be used as the basis of aplace holder section. A standalone section on TSN "island" interconnect over DetNet". Includes RFC2119 Language.] 10.later analysis of the alternatives. 8. DetNet MPLSTransport LayerOperationoverOver IEEE 802.1 TSNSub- NetworksSub-Networks [Editor's note: this is a place holder section. A standalone section on MPLS over IEEE 802.1 TSN. Includes RFC2119 Language.] This section covers howMPLSDetNet MPLS flows operate over an IEEE 802.1 TSN sub-network. Figure1715 illustrates such a scenario, where two MPLS (DetNet) nodes are interconnected by a TSN sub-network. Node-1 is single homed and Node-2 is dual-homed. MPLS nodes can be (1)MPLSDetNet MPLS End System, (2)MPLSDetNet MPLS Edge or Relay node or (3) MPLS Transit node. Note: in case of MPLS Transit node there is no DetNet Service sub-layer.layer processing. MPLS (DetNet) MPLS (DetNet) Node-1 Node-2+---------+ +---------++----------+ +----------+ <--|Service*|--Service* |-- DetNet flow ---|Service*|--> +---------+ +---------+ |Transport| |Transport| +-------.-+Service* |--> +----------+ +----------+ |Forwarding| |Forwarding| +--------.-+ <-TSN Str->+-.-----.-++-.-----.--+ \ ,-------. / / +----[ TSN-Sub ]---+ / [ Network ]--------+ `-------' <---------------- DetNet MPLS ---------------> Note: * no service sub-layer required for transit nodes Figure17:15: DetNet Enabled MPLS NetworkoverOver a TSNsub-networkSub-Network The Time-Sensitive Networking (TSN) Task Group of the IEEE 802.1 Working Group have defined (and are defining) a number of amendments to IEEE 802.1Q [IEEE8021Q] that provide zero congestion loss and bounded latency in bridged networks. Furthermore IEEE 802.1CB [IEEE8021CB] defines frame replication and elimination functions for reliability that should prove both compatible with and useful to, DetNet networks. All these functions have to identify flows those require TSN treatment. As is the case for DetNet, a Layer 2 network node such as a bridge may need to identify the specific DetNet flow to which a packet belongs in order to provide the TSN/DetNet QoS for that packet. It also may need a CoS marking, such as the priority field of an IEEE Std 802.1Q VLAN tag, to give the packet proper service. The challange for MPLS DeNet flows is that the protocol interworking function defined in IEEE 802.1CB [IEEE8021CB] works only for IP flows. The aim of the protocol interworking function is to convert an ingress flow to use a specific multicast destination MAC address and VLAN, for example to direct the packets through a specific path inside the bridged network. A similar interworking pair at the other end of the TSN sub-network would restore the packet to its original destination MAC address and VLAN. As protocol interworking function defined in [IEEE8021CB] does not work for MPLS labeled flows, theMPLSDetNet MPLS nodes MUST ensure proper TSN sub-network specific Ethernet encapsulation of theMPLSDetNet MPLS packets. For a given TSN Stream (i.e., DetNet flow) an MPLS (DetNet) node MUST behave as a TSN-aware Talker or a Listener inside the TSN sub-network.10.1.8.1. Mapping of TSN Stream ID and Sequence Number TSN capable MPLS (DetNet) nodes are TSN-aware Talker/Listener as shown in Figure18.16. MPLS (DetNet) node MUST provide the TSN sub- network specific Ethernet encapsulation over the link(s) towards the sub-network. An TSN-aware MPLS (DetNet) node MUST support the following TSN components: 1. For recognizing flows: * Stream Identification (MPLS-flow-aware) 2. For FRER used inside the TSN domain, additonaly: * Sequencing function (MPLS-flow-aware) * Sequence encode/decode function 3. For FRER when the node is a TSN replication or elimination point, additionally: * Stream splitting function * Individual recovery function [Editor's note: Should we added here requirements regarding IEEE 802.1Q C-VLAN component?] The Stream Identification and The Sequencing functions are slightly modified for frames passed down the protocol stack from the upper layers. Stream Identification MUST pair MPLS flows and TSN Streams and encode that in data plane formats as well. The packet's stream_handle subparameter (see IEEE 802.1CB [IEEE8021CB]) inside the Talker/ Listener is defined based on the Flow-ID used in the upperMPLSDetNet MPLS layer. Stream Identification function MUST encode Ethernet header fields namely (1) the destination MAC-address, (2) the VLAN-ID and (3) priority parameters with TSN sub-network specific values. Encoding is provided for the frame passed down the stack from the upper layers. The sequence generation function resides in the Sequencing function. It generates a sequence_number subparameter for each packet of a Stream passed down to the lower layers. Sequencing function MUST copy sequence information from the MPLS d-CW of the packet to the sequence_number subparameter for the frame passed down the stack from the upper layers. MPLS (DetNet) Node-1<---------> +---------+<----------> +----------+ <--| Service |-- DetNet flow ------------------+---------+ |Transport| +---------++----------+ |Forwarding| +----------+ +---------------+ | L2 with |<---| L2 Relay with |---- TSN ---- | TSN | | TSN function | Stream+----.----++-----.----+ +--.---------.--+ \__________/ \______ TSN-aware Talker / TSN-Bridge Listener Relay <--------- TSN sub-network ------------ Figure18:16: MPLS (DetNet)nodeNode with TSNfunctionsFunctions The Sequence encode/decode function MUST support the Redundancy tag (R-TAG) format as per Clause 7.8 of IEEE 802.1CB [IEEE8021CB].10.2.8.2. TSN Usage of FRER TSN Streams supporting DetNet flows may use Frame Replication and Elimination for Redundancy (FRER) [802.1CB] based on the loss service requirements of the TSN Stream, which is derived from the DetNet service requirements of the DetNet mapped flow. The specific operation of FRER is not modified by the use of DetNet and follows IEEE 802.1CB [IEEE8021CB]. FRER function and the provided service recovery is available only within the TSN sub-network however as the Stream-ID and the TSN sequence number are paired with the MPLS flow parameters they can be combined with PREOF functions.10.3.8.3. Management and Control Implications [Editor's note: This section is TBD Covers Creation, mapping, removal of TSN Stream IDs, related parameters and,when needed, configuration of FRER. Supported by management/control plane.]11.9. DetNet MPLSTransport LayerOperation overIPDetNet IP PSNs This section specifies the DetNet encapsulation over an IPtransportnetwork. The approach is modeled on the operation of MPLS and PseudoWires (PW) over an IP Packet Switched Network (PSN) [RFC3985][RFC4385][RFC7510]. Itis also based onmaps the MPLS data plane encapsulation described in Section6.2.6.2 to the DetNet IP data plane define in [I-D.ietf-detnet-dp-sol-ip]. To carry DetNet with full functionality at the DetNet layer over an IPtransportnetwork, the following components are required (these are a subset of the requirements for MPLS encapsulation listed in Section 6.1): 1. A method of identifying the DetNet flow group to the processing element. 2. A method of carrying the DetNet sequence number. 3. A method of distinguishing DetNet OAM packets from DetNet data packets. 4. A method of carrying queuing and forwarding indication. These requirements are satisfied by the DetNet over MPLS Encapsulation described in Section 6.2.To simplify operations and implementations, rather than inventing a new encapsulation, the IP encapsulation takes advantage ofThis document builds on theMPLS encapsulation. By usingthe specification of MPLS over UDP andIP in [RFC7510], the T-Label(s) shown in Figure 15 in Section 6.2 can be replaced by UDP and IP, resulting in the following encapsulation: +---------------------------------+ | | | DetNet Flow | | Payload Packet | | | +---------------------------------+ <--\ | DetNet Control Word | | +---------------------------------+ +--> DetNet data plane | S-Label | | MPLS encapsulation +---------------------------------+ <--/ | UDP Header | +---------------------------------+ | IP Header | +---------------------------------+ | Data-Link | +---------------------------------+ | Physical | +---------------------------------+ Figure 19: IP Encapsulation of DetNet Where the UDP header is used as defined in Section 3 of [RFC7510]. As in Section 6.2, the S-Label is used to identify a DetNet flow to the peer node that processes it,IP defined inthis case[RFC7510]. It replaces thenode addressed bytheIP HeaderF-Label(s) used inFigure 19.Section 6.2 with UDP and IP headers. TheS-LabelUDP and IP header information isallocated from the receiving node?s platform label space [RFC3031]. In ingress Edge Nodes, the encapsulation in Figure 19 will be imposed on Detnet Flow Payload Packets as received fromused to identify DetNetEnd Systems, and theflows, including member flows, per [I-D.ietf-detnet-dp-sol-ip]. The resulting encapsulationwill be removedis shown inegress Edge Nodes as they transmit the Payload Packets to the End Systems.Figure 17. Note that this encapsulation works equally well with IPv4 and IPv6.This encapsulation can also be used in conjunction with segment routing as specified in [I-D.ietf-spring-segment-routing-mpls]. In this case, the T-Label(s) in Figure 19 should be retained, and at each hop, the top T-label is popped and mapped to a corresponding UDP/IP tunnel, resulting in the following encapsulation:+---------------------------------+ | | | DetNetFlowApp-Flow | | Payload Packet | | | +---------------------------------+ <--\ | DetNet Control Word | | +---------------------------------+ +--> DetNet data plane | S-Label | | MPLS encapsulation +---------------------------------+<--/ | T-Label(s) | +---------------------------------+<--X. | UDP Header | | +---------------------------------+ +--> DetNet data plane | IP Header | | IP encapsulation +---------------------------------+ <--/ | Data-Link | +---------------------------------+ | Physical | +---------------------------------+ Figure20:17: IP Encapsulation of DetNetwith MPLS-SR Again, the UDP header isMPLS d-CW and and S-Labels are used as defined in Section36.2 and are not modified by this section. To support outgoing DetNet MPLS over IP, an implementation MUST support the provisioning of[RFC7510].IP/UDP header information in place of sets of F-Labels. Note thatif required in both the casemultiple sets ofIP EncapsulationF-Labels can be provisioned to support PRF on transmitted DetNet flows and therefore, when PRF is supported, multiple IP/UDP headers MAY be provisioned. When multiple IP/UDP headers are provisioned for a particular outgoing app-flow, a copy of the outgoing packet, including the pushed S-Label, MUST be made for each. The headers for each outgoing packet MUST be based on the configuration information and as defined in [RFC7510], with one exception. The one exceptions is that the UDP Source Port value MUST be set to uniquely identify the DetNet (forwarding sub-layer) flow. The packet MUST then be handed as a DetNetFigure 19, and ofIPEncapsulationpacket, per [I-D.ietf-detnet-dp-sol-ip]. To support receive processing an implementation MUST also support the provisioning ofDetNet with MPLS-SR Figure 20, itreceived IP/UDP header information. When S-Labels are taken from platform label space, all that is required ispossibletoomitprovision that receiving IP/UDP encapsulated DetNet MPLS packets is permitted. Once theUDPIP/UDP headerif required. Operation of MPLS directly over IPisdescribed in [RFC4023]. In this case DetNet Service canstripped, the S-label uniquely identifies the app-flow. When S-Labels are not taken from platform label space, IP/UDP header information MUST beprovidedprovisioned. The provisioned information MUST then be used to identify incoming app- flows based ona per IP flow basis as described in [I-D.ietf-detnet-dp-sol-ip]. 12.the combination of S-Label and incoming IP/UDP header. Normal receive processing, including PEOF can then take place. 10. SecurityconsiderationsConsiderations The security considerations of DetNet in general are discussed in [I-D.ietf-detnet-architecture] and [I-D.sdt-detnet-security]. Other security considerations will be added in a future version of this draft.13.11. IANAconsiderationsConsiderations This document makes no IANA requests.14.12. Contributors RFC7322 limits the number of authors listed on the front page of a draft to a maximum of 5, far fewer than the20many individuals below who made important contributions to this draft. The editor wishes to thank and acknowledge each of the following authors for contributing text to this draft. See also Section15.13. Loa Andersson Huawei Email: loa@pi.nu Yuanlong Jiang Huawei Email: jiangyuanlong@huawei.com Norman Finn Huawei 3101 Rio Way Spring Valley, CA 91977 USA Email: norman.finn@mail01.huawei.com Janos Farkas Ericsson Magyar Tudosok krt. 11. Budapest 1117 Hungary Email: janos.farkas@ericsson.com Carlos J. Bernardos Universidad Carlos III de Madrid Av. Universidad, 30 Leganes, Madrid 28911 Spain Email: cjbc@it.uc3m.es Tal Mizrahi Marvell 6 Hamada st. Yokneam Israel Email: talmi@marvell.com Lou Berger LabN Consulting, L.L.C. Email: lberger@labn.net Stewart Bryant Huawei Technologies Email: stewart.bryant@gmail.com Mach Chen Huawei Technologies Email: mach.chen@huawei.com15.Andrew G. Malis Huawei Technologies Email: agmalis@gmail.com 13. Acknowledgements The author(s) ACK and NACK. The following people were part of the DetNet Data Plane Solution Design Team: Jouni Korhonen Janos Farkas Norman Finn Balazs Varga Loa Andersson Tal Mizrahi David Mozes Yuanlong Jiang Andrew Malis Carlos J. Bernardos The DetNet chairs serving during the DetNet Data Plane Solution Design Team: Lou Berger Pat Thaler Thanks for Stewart Bryant for his extensive review of the previous versions of the document.16.14. References16.1.14.1. Normativereferences [I-D.ietf-spring-segment-routing-mpls] Bashandy, A., Filsfils, C., Previdi, S., Decraene, B., Litkowski, S., and R. Shakir, "Segment Routing with MPLS data plane", draft-ietf-spring-segment-routing-mpls-14 (work in progress), June 2018.References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>. [RFC2211] Wroclawski, J., "Specification of the Controlled-Load Network Element Service", RFC 2211, DOI 10.17487/RFC2211, September 1997, <https://www.rfc-editor.org/info/rfc2211>. [RFC2212] Shenker, S., Partridge, C., and R. Guerin, "Specification of Guaranteed Quality of Service", RFC 2212, DOI 10.17487/RFC2212, September 1997, <https://www.rfc-editor.org/info/rfc2212>.[RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, DOI 10.17487/RFC2474, December 1998, <https://www.rfc-editor.org/info/rfc2474>.[RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol Label Switching Architecture", RFC 3031, DOI 10.17487/RFC3031, January 2001, <https://www.rfc-editor.org/info/rfc3031>. [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack Encoding", RFC 3032, DOI 10.17487/RFC3032, January 2001, <https://www.rfc-editor.org/info/rfc3032>.[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, DOI 10.17487/RFC3168, September 2001, <https://www.rfc-editor.org/info/rfc3168>.[RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, <https://www.rfc-editor.org/info/rfc3209>. [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- Protocol Label Switching (MPLS) Support of Differentiated Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, <https://www.rfc-editor.org/info/rfc3270>. [RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing in Multi-Protocol Label Switching (MPLS) Networks", RFC 3443, DOI 10.17487/RFC3443, January 2003, <https://www.rfc-editor.org/info/rfc3443>. [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label Switching (GMPLS) Signaling Resource ReserVation Protocol- Traffic Engineering (RSVP-TE) Extensions", RFC 3473, DOI 10.17487/RFC3473, January 2003, <https://www.rfc-editor.org/info/rfc3473>.[RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed., "Encapsulating MPLS in IP or Generic Routing Encapsulation (GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005, <https://www.rfc-editor.org/info/rfc4023>.[RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP) Hierarchy with Generalized Multi-Protocol Label Switching (GMPLS) Traffic Engineering (TE)", RFC 4206, DOI 10.17487/RFC4206, October 2005, <https://www.rfc-editor.org/info/rfc4206>. [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385, February 2006, <https://www.rfc-editor.org/info/rfc4385>. [RFC5085] Nadeau, T., Ed. and C. Pignataro, Ed., "Pseudowire Virtual Circuit Connectivity Verification (VCCV): A Control Channel for Pseudowires", RFC 5085, DOI 10.17487/RFC5085, December 2007, <https://www.rfc-editor.org/info/rfc5085>. [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 2008, <https://www.rfc-editor.org/info/rfc5129>. [RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic Class" Field", RFC 5462, DOI 10.17487/RFC5462, February 2009, <https://www.rfc-editor.org/info/rfc5462>.[RFC6003] Papadimitriou, D., "Ethernet Traffic Parameters", RFC 6003, DOI 10.17487/RFC6003, October 2010, <https://www.rfc-editor.org/info/rfc6003>.[RFC7510] Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black, "Encapsulating MPLS in UDP", RFC 7510, DOI 10.17487/RFC7510, April 2015, <https://www.rfc-editor.org/info/rfc7510>. [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.16.2.14.2. InformativereferencesReferences [G.8275.1] International Telecommunication Union, "Precision time protocol telecom profile for phase/time synchronization with full timing support from the network", ITU-T G.8275.1/Y.1369.1 G.8275.1, June 2016, <https://www.itu.int/rec/T-REC-G.8275.1/en>. [G.8275.2] International Telecommunication Union, "Precision time protocol telecom profile for phase/time synchronization with partial timing support from the network", ITU-T G.8275.2/Y.1369.2 G.8275.2, June 2016, <https://www.itu.int/rec/T-REC-G.8275.2/en>. [I-D.ietf-detnet-architecture] Finn, N., Thubert, P., Varga, B., and J. Farkas, "Deterministic Networking Architecture", draft-ietf-detnet-architecture-08detnet-architecture-11 (work in progress),September 2018. [I-D.ietf-detnet-dp-alt]February 2019. [I-D.ietf-detnet-dp-sol-ip] Korhonen, J., Varga, B., "DetNet IP Data Plane Encapsulation", 2018. [I-D.ietf-detnet-flow-information-model] Farkas, J.,Mirsky, G., Thubert, P., Zhuangyan,Varga, B., Cummings, R., and Y. Jiang, "DetNet Flow Information Model", draft-ietf-detnet-flow- information-model-03 (work in progress), March 2019. [I-D.ietf-pce-pcep-extension-for-pce-controller] Zhao, Q., Li, Z., Negi, M., andL. Berger, "DetNet Data Plane ProtocolC. Zhou, "PCEP Procedures andSolution Alternatives", draft-ietf-detnet-dp-alt-00Protocol Extensions for Using PCE as a Central Controller (PCECC) of LSPs", draft-ietf-pce-pcep- extension-for-pce-controller-01 (work in progress),October 2016. [I-D.ietf-detnet-dp-sol-ip] Korhonen, J., Varga,February 2019. [I-D.ietf-spring-segment-routing-mpls] Bashandy, A., Filsfils, C., Previdi, S., Decraene, B.,"DetNet IP Data Plane Encapsulation",Litkowski, S., and R. Shakir, "Segment Routing with MPLS data plane", draft-ietf-spring-segment-routing-mpls-18 (work in progress), December 2018. [I-D.sdt-detnet-security] Mizrahi, T., Grossman, E., Hacker, A., Das, S., "Deterministic Networking (DetNet) Security Considerations, draft-sdt-detnet-security, work in progress", 2017. [IEEE1588] IEEE, "IEEE 1588 Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems Version 2", 2008. [IEEE8021CB] Finn, N., "Draft Standard for Local and metropolitan area networks - Seamless Redundancy", IEEE P802.1CB /D2.1 P802.1CB, December 2015, <http://www.ieee802.org/1/files/private/cb-drafts/ d2/802-1CB-d2-1.pdf>. [IEEE8021Q] IEEE 802.1, "Standard for Local and metropolitan area networks--Bridges and Bridged Networks (IEEE Std 802.1Q- 2014)", 2014, <http://standards.ieee.org/about/get/>. [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, September 1997, <https://www.rfc-editor.org/info/rfc2205>. [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, DOI 10.17487/RFC2474, December 1998, <https://www.rfc-editor.org/info/rfc2474>. [RFC3272] Awduche, D., Chiu, A., Elwalid, A., Widjaja, I., and X. Xiao, "Overview and Principles of Internet Traffic Engineering", RFC 3272, DOI 10.17487/RFC3272, May 2002, <https://www.rfc-editor.org/info/rfc3272>. [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation Edge-to-Edge (PWE3) Architecture", RFC 3985, DOI 10.17487/RFC3985, March 2005, <https://www.rfc-editor.org/info/rfc3985>. [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, "Encapsulation Methods for Transport of Ethernet over MPLS Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, <https://www.rfc-editor.org/info/rfc4448>. [RFC4872] Lang, J., Ed., Rekhter, Y., Ed., and D. Papadimitriou, Ed., "RSVP-TE Extensions in Support of End-to-End Generalized Multi-Protocol Label Switching (GMPLS) Recovery", RFC 4872, DOI 10.17487/RFC4872, May 2007, <https://www.rfc-editor.org/info/rfc4872>. [RFC4873] Berger, L., Bryskin, I., Papadimitriou, D., and A. Farrel, "GMPLS Segment Recovery", RFC 4873, DOI 10.17487/RFC4873, May 2007, <https://www.rfc-editor.org/info/rfc4873>. [RFC4875] Aggarwal, R., Ed., Papadimitriou, D., Ed., and S. Yasukawa, Ed., "Extensions to Resource Reservation Protocol - Traffic Engineering (RSVP-TE) for Point-to- Multipoint TE Label Switched Paths (LSPs)", RFC 4875, DOI 10.17487/RFC4875, May 2007, <https://www.rfc-editor.org/info/rfc4875>. [RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation Element (PCE) Communication Protocol (PCEP)", RFC 5440, DOI 10.17487/RFC5440, March 2009, <https://www.rfc-editor.org/info/rfc5440>. [RFC5586] Bocci, M., Ed., Vigoureux, M., Ed., and S. Bryant, Ed., "MPLS Generic Associated Channel", RFC 5586, DOI 10.17487/RFC5586, June 2009, <https://www.rfc-editor.org/info/rfc5586>. [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., Sprecher, N., and S. Ueno, "Requirements of an MPLS Transport Profile", RFC 5654, DOI 10.17487/RFC5654, September 2009, <https://www.rfc-editor.org/info/rfc5654>. [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, L., and L. Berger, "A Framework for MPLS in Transport Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, <https://www.rfc-editor.org/info/rfc5921>. [RFC6003] Papadimitriou, D., "Ethernet Traffic Parameters", RFC 6003, DOI 10.17487/RFC6003, October 2010, <https://www.rfc-editor.org/info/rfc6003>. [RFC6006] Zhao, Q., Ed., King, D., Ed., Verhaeghe, F., Takeda, T., Ali, Z., and J. Meuric, "Extensions to the Path Computation Element Communication Protocol (PCEP) for Point-to-Multipoint Traffic Engineering Label Switched Paths", RFC 6006, DOI 10.17487/RFC6006, September 2010, <https://www.rfc-editor.org/info/rfc6006>. [RFC6073] Martini, L., Metz, C., Nadeau, T., Bocci, M., and M. Aissaoui, "Segmented Pseudowire", RFC 6073, DOI 10.17487/RFC6073, January 2011, <https://www.rfc-editor.org/info/rfc6073>. [RFC6387] Takacs, A., Berger, L., Caviglia, D., Fedyk, D., and J. Meuric, "GMPLS Asymmetric Bandwidth Bidirectional Label Switched Paths (LSPs)", RFC 6387, DOI 10.17487/RFC6387, September 2011, <https://www.rfc-editor.org/info/rfc6387>. [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and L. Yong, "The Use of Entropy Labels in MPLS Forwarding", RFC 6790, DOI 10.17487/RFC6790, November 2012, <https://www.rfc-editor.org/info/rfc6790>. [RFC7551] Zhang, F., Ed., Jing, R., and R. Gandhi, Ed., "RSVP-TE Extensions for Associated Bidirectional Label Switched Paths (LSPs)", RFC 7551, DOI 10.17487/RFC7551, May 2015, <https://www.rfc-editor.org/info/rfc7551>. [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", RFC 7950, DOI 10.17487/RFC7950, August 2016, <https://www.rfc-editor.org/info/rfc7950>. [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, <https://www.rfc-editor.org/info/rfc8040>. [RFC8169] Mirsky, G., Ruffini, S., Gray, E., Drake, J., Bryant, S., and A. Vainshtein, "Residence Time Measurement in MPLS Networks", RFC 8169, DOI 10.17487/RFC8169, May 2017, <https://www.rfc-editor.org/info/rfc8169>. Appendix A. Example of DetNetdata plane operationData Plane Operation [Editor's note: Add a simplified example of DetNet data plane and how labels etc work in the case of MPLS-based PSN and utilizing PREOF. The figure is subject to change depending on the further DT decisions on the label handling..] Authors' Addresses Jouni Korhonen (editor) Email: jouni.nospam@gmail.com Balazs Varga (editor) Ericsson Magyar Tudosok krt. 11. Budapest 1117 Hungary Email: balazs.a.varga@ericsson.com