--- 1/draft-ietf-detnet-use-cases-05.txt 2016-03-04 17:16:17.162331769 -0800 +++ 2/draft-ietf-detnet-use-cases-06.txt 2016-03-04 17:16:17.302335328 -0800 @@ -1,38 +1,38 @@ Internet Engineering Task Force E. Grossman, Ed. Internet-Draft DOLBY Intended status: Informational C. Gunther -Expires: August 26, 2016 HARMAN +Expires: September 5, 2016 HARMAN P. Thubert P. Wetterwald CISCO J. Raymond HYDRO-QUEBEC J. Korhonen BROADCOM Y. Kaneko Toshiba S. Das Applied Communication Sciences Y. Zha HUAWEI B. Varga J. Farkas Ericsson F. Goetz J. Schmitt Siemens - February 23, 2016 + March 4, 2016 Deterministic Networking Use Cases - draft-ietf-detnet-use-cases-05 + draft-ietf-detnet-use-cases-06 Abstract This draft documents requirements in several diverse industries to establish multi-hop paths for characterized flows with deterministic properties. In this context deterministic implies that streams can be established which provide guaranteed bandwidth and latency which can be established from either a Layer 2 or Layer 3 (IP) interface, and which can co-exist on an IP network with best-effort traffic. @@ -54,21 +54,21 @@ Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." - This Internet-Draft will expire on August 26, 2016. + This Internet-Draft will expire on September 5, 2016. Copyright Notice Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents @@ -77,125 +77,132 @@ include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Pro Audio Use Cases . . . . . . . . . . . . . . . . . . . . . 5 2.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 5 2.2. Fundamental Stream Requirements . . . . . . . . . . . . . 6 - 2.2.1. Guaranteed Bandwidth . . . . . . . . . . . . . . . . 6 + 2.2.1. Guaranteed Bandwidth . . . . . . . . . . . . . . . . 7 2.2.2. Bounded and Consistent Latency . . . . . . . . . . . 7 2.2.2.1. Optimizations . . . . . . . . . . . . . . . . . . 8 2.3. Additional Stream Requirements . . . . . . . . . . . . . 9 2.3.1. Deterministic Time to Establish Streaming . . . . . . 9 2.3.2. Use of Unused Reservations by Best-Effort Traffic . . 9 2.3.3. Layer 3 Interconnecting Layer 2 Islands . . . . . . . 10 2.3.4. Secure Transmission . . . . . . . . . . . . . . . . . 10 2.3.5. Redundant Paths . . . . . . . . . . . . . . . . . . . 10 - 2.3.6. Link Aggregation . . . . . . . . . . . . . . . . . . 10 + 2.3.6. Link Aggregation . . . . . . . . . . . . . . . . . . 11 2.3.7. Traffic Segregation . . . . . . . . . . . . . . . . . 11 2.3.7.1. Packet Forwarding Rules, VLANs and Subnets . . . 11 2.3.7.2. Multicast Addressing (IPv4 and IPv6) . . . . . . 11 2.4. Integration of Reserved Streams into IT Networks . . . . 12 2.5. Security Considerations . . . . . . . . . . . . . . . . . 12 2.5.1. Denial of Service . . . . . . . . . . . . . . . . . . 12 2.5.2. Control Protocols . . . . . . . . . . . . . . . . . . 12 2.6. A State-of-the-Art Broadcast Installation Hits Technology Limits . . . . . . . . . . . . . . . . . . . . . . . . . 13 - 3. Utility Telecom Use Cases . . . . . . . . . . . . . . . . . . 13 - 3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 13 - 3.2. Telecommunications Trends and General telecommunications - Requirements . . . . . . . . . . . . . . . . . . . . . . 14 - 3.2.1. General Telecommunications Requirements . . . . . . . 14 - 3.2.1.1. Migration to Packet-Switched Network . . . . . . 15 - 3.2.2. Applications, Use cases and traffic patterns . . . . 16 - 3.2.2.1. Transmission use cases . . . . . . . . . . . . . 16 - 3.2.2.2. Distribution use case . . . . . . . . . . . . . . 26 - 3.2.2.3. Generation use case . . . . . . . . . . . . . . . 29 - 3.2.3. Specific Network topologies of Smart Grid - Applications . . . . . . . . . . . . . . . . . . . . 30 - 3.2.4. Precision Time Protocol . . . . . . . . . . . . . . . 31 - 3.3. IANA Considerations . . . . . . . . . . . . . . . . . . . 32 - 3.4. Security Considerations . . . . . . . . . . . . . . . . . 32 - 3.4.1. Current Practices and Their Limitations . . . . . . . 32 - 3.4.2. Security Trends in Utility Networks . . . . . . . . . 34 - 4. Building Automation Systems . . . . . . . . . . . . . . . . . 35 - 4.1. Use Case Description . . . . . . . . . . . . . . . . . . 35 - 4.2. Building Automation Systems Today . . . . . . . . . . . . 36 - 4.2.1. BAS Architecture . . . . . . . . . . . . . . . . . . 36 - 4.2.2. BAS Deployment Model . . . . . . . . . . . . . . . . 37 - 4.2.3. Use Cases for Field Networks . . . . . . . . . . . . 39 - 4.2.3.1. Environmental Monitoring . . . . . . . . . . . . 39 - 4.2.3.2. Fire Detection . . . . . . . . . . . . . . . . . 39 - 4.2.3.3. Feedback Control . . . . . . . . . . . . . . . . 40 - 4.2.4. Security Considerations . . . . . . . . . . . . . . . 40 - 4.3. BAS Future . . . . . . . . . . . . . . . . . . . . . . . 40 - 4.4. BAS Asks . . . . . . . . . . . . . . . . . . . . . . . . 41 - 5. Wireless for Industrial . . . . . . . . . . . . . . . . . . . 41 - 5.1. Use Case Description . . . . . . . . . . . . . . . . . . 41 - 5.1.1. Network Convergence using 6TiSCH . . . . . . . . . . 42 - 5.1.2. Common Protocol Development for 6TiSCH . . . . . . . 42 - 5.2. Wireless Industrial Today . . . . . . . . . . . . . . . . 43 - 5.3. Wireless Industrial Future . . . . . . . . . . . . . . . 43 - 5.3.1. Unified Wireless Network and Management . . . . . . . 43 - 5.3.1.1. PCE and 6TiSCH ARQ Retries . . . . . . . . . . . 45 - 5.3.2. Schedule Management by a PCE . . . . . . . . . . . . 46 - 5.3.2.1. PCE Commands and 6TiSCH CoAP Requests . . . . . . 46 - 5.3.2.2. 6TiSCH IP Interface . . . . . . . . . . . . . . . 47 - 5.3.3. 6TiSCH Security Considerations . . . . . . . . . . . 47 - 5.4. Wireless Industrial Asks . . . . . . . . . . . . . . . . 48 - 6. Cellular Radio Use Cases . . . . . . . . . . . . . . . . . . 48 - 6.1. Use Case Description . . . . . . . . . . . . . . . . . . 48 - 6.1.1. Network Architecture . . . . . . . . . . . . . . . . 48 - 6.1.2. Time Synchronization Requirements . . . . . . . . . . 49 - 6.1.3. Time-Sensitive Stream Requirements . . . . . . . . . 51 - 6.1.4. Security Considerations . . . . . . . . . . . . . . . 51 - 6.2. Cellular Radio Networks Today . . . . . . . . . . . . . . 52 - 6.3. Cellular Radio Networks Future . . . . . . . . . . . . . 52 - 6.4. Cellular Radio Networks Asks . . . . . . . . . . . . . . 54 - 7. Cellular Coordinated Multipoint Processing (CoMP) . . . . . . 54 - 7.1. Use Case Description . . . . . . . . . . . . . . . . . . 54 - 7.1.1. CoMP Architecture . . . . . . . . . . . . . . . . . . 55 - 7.1.2. Delay Sensitivity in CoMP . . . . . . . . . . . . . . 56 - 7.2. CoMP Today . . . . . . . . . . . . . . . . . . . . . . . 56 - 7.3. CoMP Future . . . . . . . . . . . . . . . . . . . . . . . 56 - 7.3.1. Mobile Industry Overall Goals . . . . . . . . . . . . 56 - 7.3.2. CoMP Infrastructure Goals . . . . . . . . . . . . . . 57 - 7.4. CoMP Asks . . . . . . . . . . . . . . . . . . . . . . . . 57 - 8. Industrial M2M . . . . . . . . . . . . . . . . . . . . . . . 58 - 8.1. Use Case Description . . . . . . . . . . . . . . . . . . 58 - 8.2. Industrial M2M Communication Today . . . . . . . . . . . 59 - 8.2.1. Transport Parameters . . . . . . . . . . . . . . . . 59 - 8.2.2. Stream Creation and Destruction . . . . . . . . . . . 60 - 8.3. Industrial M2M Future . . . . . . . . . . . . . . . . . . 60 - 8.4. Industrial M2M Asks . . . . . . . . . . . . . . . . . . . 61 - 9. Internet-based Applications . . . . . . . . . . . . . . . . . 61 - 9.1. Use Case Description . . . . . . . . . . . . . . . . . . 61 - 9.1.1. Media Content Delivery . . . . . . . . . . . . . . . 61 - 9.1.2. Online Gaming . . . . . . . . . . . . . . . . . . . . 61 - 9.1.3. Virtual Reality . . . . . . . . . . . . . . . . . . . 61 - 9.2. Internet-Based Applications Today . . . . . . . . . . . . 62 - 9.3. Internet-Based Applications Future . . . . . . . . . . . 62 - 9.4. Internet-Based Applications Asks . . . . . . . . . . . . 62 - 10. Use Case Common Elements . . . . . . . . . . . . . . . . . . 62 - 11. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 63 - 11.1. Pro Audio . . . . . . . . . . . . . . . . . . . . . . . 63 - 11.2. Utility Telecom . . . . . . . . . . . . . . . . . . . . 64 - 11.3. Building Automation Systems . . . . . . . . . . . . . . 64 - 11.4. Wireless for Industrial . . . . . . . . . . . . . . . . 64 - 11.5. Cellular Radio . . . . . . . . . . . . . . . . . . . . . 64 - 11.6. Industrial M2M . . . . . . . . . . . . . . . . . . . . . 64 - 11.7. Internet Applications and CoMP . . . . . . . . . . . . . 64 - 12. Informative References . . . . . . . . . . . . . . . . . . . 65 - Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 73 + 3. Electrical Utilities . . . . . . . . . . . . . . . . . . . . 13 + 3.1. Use Case Description . . . . . . . . . . . . . . . . . . 13 + 3.1.1. Transmission Use Cases . . . . . . . . . . . . . . . 13 + 3.1.1.1. Protection . . . . . . . . . . . . . . . . . . . 13 + 3.1.1.2. Intra-Substation Process Bus Communications . . . 19 + 3.1.1.3. Wide Area Monitoring and Control Systems . . . . 20 + 3.1.1.4. IEC 61850 WAN engineering guidelines requirement + classification . . . . . . . . . . . . . . . . . 21 + 3.1.2. Generation Use Case . . . . . . . . . . . . . . . . . 22 + 3.1.3. Distribution use case . . . . . . . . . . . . . . . . 23 + 3.1.3.1. Fault Location Isolation and Service Restoration + (FLISR) . . . . . . . . . . . . . . . . . . . . . 23 + 3.2. Electrical Utilities Today . . . . . . . . . . . . . . . 24 + 3.2.1. Security Current Practices and Limitations . . . . . 24 + 3.3. Electrical Utilities Future . . . . . . . . . . . . . . . 26 + 3.3.1. Migration to Packet-Switched Network . . . . . . . . 26 + 3.3.2. Telecommunications Trends . . . . . . . . . . . . . . 27 + 3.3.2.1. General Telecommunications Requirements . . . . . 27 + 3.3.2.2. Specific Network topologies of Smart Grid + Applications . . . . . . . . . . . . . . . . . . 28 + 3.3.2.3. Precision Time Protocol . . . . . . . . . . . . . 29 + 3.3.3. Security Trends in Utility Networks . . . . . . . . . 30 + 3.4. Electrical Utilities Asks . . . . . . . . . . . . . . . . 32 + 4. Building Automation Systems . . . . . . . . . . . . . . . . . 32 + 4.1. Use Case Description . . . . . . . . . . . . . . . . . . 32 + 4.2. Building Automation Systems Today . . . . . . . . . . . . 32 + 4.2.1. BAS Architecture . . . . . . . . . . . . . . . . . . 33 + 4.2.2. BAS Deployment Model . . . . . . . . . . . . . . . . 34 + 4.2.3. Use Cases for Field Networks . . . . . . . . . . . . 36 + 4.2.3.1. Environmental Monitoring . . . . . . . . . . . . 36 + 4.2.3.2. Fire Detection . . . . . . . . . . . . . . . . . 36 + 4.2.3.3. Feedback Control . . . . . . . . . . . . . . . . 37 + 4.2.4. Security Considerations . . . . . . . . . . . . . . . 37 + 4.3. BAS Future . . . . . . . . . . . . . . . . . . . . . . . 37 + 4.4. BAS Asks . . . . . . . . . . . . . . . . . . . . . . . . 38 + 5. Wireless for Industrial . . . . . . . . . . . . . . . . . . . 38 + 5.1. Use Case Description . . . . . . . . . . . . . . . . . . 38 + 5.1.1. Network Convergence using 6TiSCH . . . . . . . . . . 39 + 5.1.2. Common Protocol Development for 6TiSCH . . . . . . . 39 + + 5.2. Wireless Industrial Today . . . . . . . . . . . . . . . . 40 + 5.3. Wireless Industrial Future . . . . . . . . . . . . . . . 40 + 5.3.1. Unified Wireless Network and Management . . . . . . . 40 + 5.3.1.1. PCE and 6TiSCH ARQ Retries . . . . . . . . . . . 42 + 5.3.2. Schedule Management by a PCE . . . . . . . . . . . . 43 + 5.3.2.1. PCE Commands and 6TiSCH CoAP Requests . . . . . . 43 + 5.3.2.2. 6TiSCH IP Interface . . . . . . . . . . . . . . . 44 + 5.3.3. 6TiSCH Security Considerations . . . . . . . . . . . 44 + 5.4. Wireless Industrial Asks . . . . . . . . . . . . . . . . 45 + 6. Cellular Radio Use Cases . . . . . . . . . . . . . . . . . . 45 + 6.1. Use Case Description . . . . . . . . . . . . . . . . . . 45 + 6.1.1. Network Architecture . . . . . . . . . . . . . . . . 45 + 6.1.2. Time Synchronization Requirements . . . . . . . . . . 46 + 6.1.3. Time-Sensitive Stream Requirements . . . . . . . . . 48 + 6.1.4. Security Considerations . . . . . . . . . . . . . . . 48 + 6.2. Cellular Radio Networks Today . . . . . . . . . . . . . . 49 + 6.3. Cellular Radio Networks Future . . . . . . . . . . . . . 49 + 6.4. Cellular Radio Networks Asks . . . . . . . . . . . . . . 51 + 7. Cellular Coordinated Multipoint Processing (CoMP) . . . . . . 51 + 7.1. Use Case Description . . . . . . . . . . . . . . . . . . 51 + 7.1.1. CoMP Architecture . . . . . . . . . . . . . . . . . . 52 + 7.1.2. Delay Sensitivity in CoMP . . . . . . . . . . . . . . 53 + 7.2. CoMP Today . . . . . . . . . . . . . . . . . . . . . . . 53 + 7.3. CoMP Future . . . . . . . . . . . . . . . . . . . . . . . 53 + 7.3.1. Mobile Industry Overall Goals . . . . . . . . . . . . 53 + 7.3.2. CoMP Infrastructure Goals . . . . . . . . . . . . . . 54 + 7.4. CoMP Asks . . . . . . . . . . . . . . . . . . . . . . . . 54 + 8. Industrial M2M . . . . . . . . . . . . . . . . . . . . . . . 55 + 8.1. Use Case Description . . . . . . . . . . . . . . . . . . 55 + 8.2. Industrial M2M Communication Today . . . . . . . . . . . 56 + 8.2.1. Transport Parameters . . . . . . . . . . . . . . . . 56 + 8.2.2. Stream Creation and Destruction . . . . . . . . . . . 57 + 8.3. Industrial M2M Future . . . . . . . . . . . . . . . . . . 57 + 8.4. Industrial M2M Asks . . . . . . . . . . . . . . . . . . . 58 + 9. Internet-based Applications . . . . . . . . . . . . . . . . . 58 + 9.1. Use Case Description . . . . . . . . . . . . . . . . . . 58 + 9.1.1. Media Content Delivery . . . . . . . . . . . . . . . 58 + 9.1.2. Online Gaming . . . . . . . . . . . . . . . . . . . . 58 + 9.1.3. Virtual Reality . . . . . . . . . . . . . . . . . . . 58 + 9.2. Internet-Based Applications Today . . . . . . . . . . . . 59 + 9.3. Internet-Based Applications Future . . . . . . . . . . . 59 + 9.4. Internet-Based Applications Asks . . . . . . . . . . . . 59 + 10. Use Case Common Elements . . . . . . . . . . . . . . . . . . 59 + 11. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 60 + 11.1. Pro Audio . . . . . . . . . . . . . . . . . . . . . . . 60 + 11.2. Utility Telecom . . . . . . . . . . . . . . . . . . . . 61 + 11.3. Building Automation Systems . . . . . . . . . . . . . . 61 + 11.4. Wireless for Industrial . . . . . . . . . . . . . . . . 61 + 11.5. Cellular Radio . . . . . . . . . . . . . . . . . . . . . 61 + 11.6. Industrial M2M . . . . . . . . . . . . . . . . . . . . . 61 + 11.7. Internet Applications and CoMP . . . . . . . . . . . . . 61 + 12. Informative References . . . . . . . . . . . . . . . . . . . 62 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 70 1. Introduction This draft presents use cases from diverse industries which have in common a need for deterministic streams, but which also differ notably in their network topologies and specific desired behavior. Together, they provide broad industry context for DetNet and a yardstick against which proposed DetNet designs can be measured (to what extent does a proposed design satisfy these various use cases?) @@ -583,291 +590,130 @@ they possibly could with packet-based technology. They constructed seven individual studios using layer 2 LANS (using IEEE 802.1 AVB) that were entirely effective at routing audio within the LANs, and they were very happy with the results, however to interconnect these layer 2 LAN islands together they ended up using dedicated links because there is no standards-based routing solution available. This is the kind of motivation we have to develop these standards because customers are ready and able to use them. -3. Utility Telecom Use Cases - -3.1. Overview - - [I-D.finn-detnet-problem-statement] defines the characteristics of a - deterministic flow as a data communication flow with a bounded - latency, extraordinarily low frame loss, and a very narrow jitter. - This document intends to define the utility requirements for - deterministic networking. - - Utility Telecom Networks - - The business and technology trends that are sweeping the utility - industry will drastically transform the utility business from the way - it has been for many decades. At the core of many of these changes - is a drive to modernize the electrical grid with an integrated - telecommunications infrastructure. However, interoperability, - concerns, legacy networks, disparate tools, and stringent security - requirements all add complexity to the grid transformation. Given - the range and diversity of the requirements that should be addressed - by the next generation telecommunications infrastructure, utilities - need to adopt a holistic architectural approach to integrate the - electrical grid with digital telecommunications across the entire - power delivery chain. - - Many utilities still rely on complex environments formed of multiple - application-specific, proprietary networks. Information is siloed - between operational areas. This prevents utility operations from - realizing the operational efficiency benefits, visibility, and - functional integration of operational information across grid - applications and data networks. The key to modernizing grid - telecommunications is to provide a common, adaptable, multi-service - network infrastructure for the entire utility organization. Such a - network serves as the platform for current capabilities while - enabling future expansion of the network to accommodate new - applications and services. - - To meet this diverse set of requirements, both today and in the - future, the next generation utility telecommunnications network will - be based on open-standards-based IP architecture. An end-to-end IP - architecture takes advantage of nearly three decades of IP technology - development, facilitating interoperability across disparate networks - and devices, as it has been already demonstrated in many mission- - critical and highly secure networks. - - IEC (International Electrotechnical Commission) and different - National Committees have mandated a specific adhoc group (AHG8) to - define the migration strategy to IPv6 for all the IEC TC57 power - automation standards. IPv6 is seen as the obvious future - telecommunications technology for the Smart Grid. The Adhoc Group - has disclosed, to the IEC coordination group, their conclusions at - the end of 2014. - - It is imperative that utilities participate in standards development - bodies to influence the development of future solutions and to - benefit from shared experiences of other utilities and vendors. - -3.2. Telecommunications Trends and General telecommunications - Requirements - - These general telecommunications requirements are over and above the - specific requirements of the use cases that have been addressed so - far. These include both current and future telecommunications - related requirements that should be factored into the network - architecture and design. - -3.2.1. General Telecommunications Requirements - - o IP Connectivity everywhere - - o Monitoring services everywhere and from different remote centers - - o Move services to a virtual data center - - o Unify access to applications / information from the corporate - network - - o Unify services - - o Unified Communications Solutions - - o Mix of fiber and microwave technologies - obsolescence of SONET/ - SDH or TDM - - o Standardize grid telecommunications protocol to opened standard to - ensure interoperability - - o Reliable Telecommunications for Transmission and Distribution - Substations - - o IEEE 1588 time synchronization Client / Server Capabilities - - o Integration of Multicast Design - - o QoS Requirements Mapping - - o Enable Future Network Expansion - - o Substation Network Resilience - - o Fast Convergence Design - - o Scalable Headend Design - - o Define Service Level Agreements (SLA) and Enable SLA Monitoring - - o Integration of 3G/4G Technologies and future technologies - - o Ethernet Connectivity for Station Bus Architecture - - o Ethernet Connectivity for Process Bus Architecture - - o Protection, teleprotection and PMU (Phaser Measurement Unit) on IP - -3.2.1.1. Migration to Packet-Switched Network - - Throughout the world, utilities are increasingly planning for a - future based on smart grid applications requiring advanced - telecommunications systems. Many of these applications utilize - packet connectivity for communicating information and control signals - across the utility's Wide Area Network (WAN), made possible by - technologies such as multiprotocol label switching (MPLS). The data - that traverses the utility WAN includes: - - o Grid monitoring, control, and protection data - o Non-control grid data (e.g. asset data for condition-based - monitoring) - - o Physical safety and security data (e.g. voice and video) - - o Remote worker access to corporate applications (voice, maps, - schematics, etc.) - - o Field area network backhaul for smart metering, and distribution - grid management +3. Electrical Utilities - o Enterprise traffic (email, collaboration tools, business - applications) +3.1. Use Case Description - WANs support this wide variety of traffic to and from substations, - the transmission and distribution grid, generation sites, between - control centers, and between work locations and data centers. To - maintain this rapidly expanding set of applications, many utilities - are taking steps to evolve present time-division multiplexing (TDM) - based and frame relay infrastructures to packet systems. Packet- - based networks are designed to provide greater functionalities and - higher levels of service for applications, while continuing to - deliver reliability and deterministic (real-time) traffic support. + Many systems that an electrical utility deploys today rely on high + availability and deterministic behavior of the underlying networks. + Here we present use cases in Transmission, Generation and + Distribution, including key timing and reliability metrics. We also + discuss security issues and industry trends which affect the + architecture of next generation utility networks -3.2.2. Applications, Use cases and traffic patterns +3.1.1. Transmission Use Cases - Among the numerous applications and use cases that a utility deploys - today, many rely on high availability and deterministic behaviour of - the telecommunications networks. Protection use cases and generation - control are the most demanding and can't rely on a best effort - approach. +3.1.1.1. Protection -3.2.2.1. Transmission use cases + Protection means not only the protection of human operators but also + the protection of the electrical equipment and the preservation of + the stability and frequency of the grid. If a fault occurs in the + transmission or distribution of electricity then severe damage can + occur to human operators, electrical equipment and the grid itself, + leading to blackouts. - Protection means not only the protection of the human operator but - also the protection of the electric equipments and the preservation - of the stability and frequency of the grid. If a default occurs on - the transmission or the distribution of the electricity, important - damages could occured to the human operator but also to very costly - electrical equipments and perturb the grid leading to blackouts. The - time and reliability requirements are very strong to avoid dramatic - impacts to the electrical infrastructure. + Communication links in conjunction with protection relays are used to + selectively isolate faults on high voltage lines, transformers, + reactors and other important electrical equipment. The role of the + teleprotection system is to selectively disconnect a faulty part by + transferring command signals within the shortest possible time. -3.2.2.1.1. Tele Protection +3.1.1.1.1. Key Criteria - The key criteria for measuring Teleprotection performance are command + The key criteria for measuring teleprotection performance are command transmission time, dependability and security. These criteria are defined by the IEC standard 60834 as follows: o Transmission time (Speed): The time between the moment where state changes at the transmitter input and the moment of the corresponding change at the receiver output, including propagation - delay. Overall operating time for a Teleprotection system + delay. Overall operating time for a teleprotection system includes the time for initiating the command at the transmitting end, the propagation delay over the network (including equipments) and the selection and decision time at the receiving end, including any additional delay due to a noisy environment. o Dependability: The ability to issue and receive valid commands in the presence of interference and/or noise, by minimizing the probability of missing command (PMC). Dependability targets are typically set for a specific bit error rate (BER) level. o Security: The ability to prevent false tripping due to a noisy environment, by minimizing the probability of unwanted commands (PUC). Security targets are also set for a specific bit error rate (BER) level. - Additional key elements that may impact Teleprotection performance - include bandwidth rate of the Teleprotection system and its - resiliency or failure recovery capacity. Transmission time, - bandwidth utilization and resiliency are directly linked to the - telecommunications equipments and the connections that are used to - transfer the commands between relays. + Additional elements of the the teleprotection system that impact its + performance include: -3.2.2.1.1.1. Latency Budget Consideration + o Network bandwidth + + o Failure recovery capacity (aka resiliency) + +3.1.1.1.2. Fault Detection and Clearance Timing + + Most power line equipment can tolerate short circuits or faults for + up to approximately five power cycles before sustaining irreversible + damage or affecting other segments in the network. This translates + to total fault clearance time of 100ms. As a safety precaution, + however, actual operation time of protection systems is limited to + 70- 80 percent of this period, including fault recognition time, + command transmission time and line breaker switching time. - Delay requirements for utility networks may vary depending upon a - number of parameters, such as the specific protection equipments - used. Most power line equipment can tolerate short circuits or - faults for up to approximately five power cycles before sustaining - irreversible damage or affecting other segments in the network. This - translates to total fault clearance time of 100ms. As a safety - precaution, however, actual operation time of protection systems is - limited to 70- 80 percent of this period, including fault recognition - time, command transmission time and line breaker switching time. Some system components, such as large electromechanical switches, require particularly long time to operate and take up the majority of the total clearance time, leaving only a 10ms window for the telecommunications part of the protection scheme, independent of the distance to travel. Given the sensitivity of the issue, new networks impose requirements that are even more stringent: IEC standard 61850 limits the transfer time for protection messages to 1/4 - 1/2 cycle or 4 - 8ms (for 60Hz lines) for the most critical messages. -3.2.2.1.1.2. Asymetric delay +3.1.1.1.3. Symmetric Channel Delay - In addition to minimal transmission delay, a differential protection - telecommunications channel must be synchronous, i.e., experiencing - symmetrical channel delay in transmit and receive paths. This - requires special attention in jitter-prone packet networks. While - optimally Teleprotection systems should support zero asymmetric - delay, typical legacy relays can tolerate discrepancies of up to - 750us. + Teleprotection channels which are differential must be synchronous, + which means that any delays on the transmit and receive paths must + match each other. Teleprotection systems ideally support zero + asymmetric delay; typical legacy relays can tolerate delay + discrepancies of up to 750us. - The main tools available for lowering delay variation below this + Some tools available for lowering delay variation below this threshold are: - o A jitter buffer at the multiplexers on each end of the line can be - used to offset delay variation by queuing sent and received - packets. The length of the queues must balance the need to - regulate the rate of transmission with the need to limit overall - delay, as larger buffers result in increased latency. This is the - old TDM traditional way to fulfill this requirement. + o For legacy systems using Time Division Multiplexing (TDM), jitter + buffers at the multiplexers on each end of the line can be used to + offset delay variation by queuing sent and received packets. The + length of the queues must balance the need to regulate the rate of + transmission with the need to limit overall delay, as larger + buffers result in increased latency. - o Traffic management tools ensure that the Teleprotection signals - receive the highest transmission priority and minimize the number - of jitter addition during the path. This is one way to meet the - requirement in IP networks. + o For jitter-prone IP packet networks, traffic management tools can + ensure that the teleprotection signals receive the highest + transmission priority to minimize jitter. - o Standard Packet-Based synchronization technologies, such as + o Standard packet-based synchronization technologies, such as 1588-2008 Precision Time Protocol (PTP) and Synchronous Ethernet - (Sync-E), can help maintain stable networks by keeping a highly - accurate clock source on the different network devices involved. - -3.2.2.1.1.2.1. Other traffic characteristics - - o Redundancy: The existence in a system of more than one means of - accomplishing a given function. - - o Recovery time : The duration of time within which a business - process must be restored after any type of disruption in order to - avoid unacceptable consequences associated with a break in - business continuity. - - o performance management : In networking, a management function - defined for controlling and analyzing different parameters/metrics - such as the throughput, error rate. - - o packet loss : One or more packets of data travelling across - network fail to reach their destination. + (Sync-E), can help keep networks stable by maintaining a highly + accurate clock source on the various network devices. -3.2.2.1.1.2.2. Teleprotection network requirements +3.1.1.1.4. Teleprotection Network Requirements (IEC 61850) - The following table captures the main network requirements (this is - based on IEC 61850 standard) + The following table captures the main network metrics as based on the + IEC 61850 standard. +-----------------------------+-------------------------------------+ | Teleprotection Requirement | Attribute | +-----------------------------+-------------------------------------+ | One way maximum delay | 4-10 ms | | Asymetric delay required | Yes | | Maximum jitter | less than 250 us (750 us for legacy | | | IED) | | Topology | Point to point, point to Multi- | | | point | @@ -875,29 +721,25 @@ | precise timing required | Yes | | Recovery time on node | less than 50ms - hitless | | failure | | | performance management | Yes, Mandatory | | Redundancy | Yes | | Packet loss | 0.1% to 1% | +-----------------------------+-------------------------------------+ Table 1: Teleprotection network requirements -3.2.2.1.2. Inter-Trip Protection scheme +3.1.1.1.5. Inter-Trip Protection scheme - Inter-tripping is the controlled tripping of a circuit breaker to - complete the isolation of a circuit or piece of apparatus in concert - with the tripping of other circuit breakers. The main use of such - schemes is to ensure that protection at both ends of a faulted - circuit will operate to isolate the equipment concerned. Inter- - tripping schemes use signaling to convey a trip command to remote - circuit breakers to isolate circuits. + "Inter-tripping" is the signal-controlled tripping of a circuit + breaker to complete the isolation of a circuit or piece of apparatus + in concert with the tripping of other circuit breakers. +--------------------------------+----------------------------------+ | Inter-Trip protection | Attribute | | Requirement | | +--------------------------------+----------------------------------+ | One way maximum delay | 5 ms | | Asymetric delay required | No | | Maximum jitter | Not critical | | Topology | Point to point, point to Multi- | | | point | @@ -905,82 +747,62 @@ | Availability | 99.9999 | | precise timing required | Yes | | Recovery time on node failure | less than 50ms - hitless | | performance management | Yes, Mandatory | | Redundancy | Yes | | Packet loss | 0.1% | +--------------------------------+----------------------------------+ Table 2: Inter-Trip protection network requirements -3.2.2.1.3. Current Differential Protection Scheme +3.1.1.1.6. Current Differential Protection Scheme Current differential protection is commonly used for line protection, - and is typical for protecting parallel circuits. A main advantage - for differential protection is that, compared to overcurrent - protection, it allows only the faulted circuit to be de-energized in - case of a fault. At both end of the lines, the current is measured - by the differential relays, and based on Kirchhoff's law, both relays - will trip the circuit breaker if the current going into the line does - not equal the current going out of the line. This type of protection - scheme assumes some form of communications being present between the - relays at both end of the line, to allow both relays to compare - measured current values. A fault in line 1 will cause overcurrent to - be flowing in both lines, but because the current in line 2 is a - through following current, this current is measured equal at both - ends of the line, therefore the differential relays on line 2 will - not trip line 2. Line 1 will be tripped, as the relays will not - measure the same currents at both ends of the line. Line - differential protection schemes assume a very low telecommunications - delay between both relays, often as low as 5ms. Moreover, as those - systems are often not time-synchronized, they also assume symmetric - telecommunications paths with constant delay, which allows comparing - current measurement values taken at the exact same time. + and is typical for protecting parallel circuits. At both end of the + lines the current is measured by the differential relays, and both + relays will trip the circuit breaker if the current going into the + line does not equal the current going out of the line. This type of + protection scheme assumes some form of communications being present + between the relays at both end of the line, to allow both relays to + compare measured current values. Line differential protection + schemes assume a very low telecommunications delay between both + relays, often as low as 5ms. Moreover, as those systems are often + not time-synchronized, they also assume symmetric telecommunications + paths with constant delay, which allows comparing current measurement + values taken at the exact same time. +----------------------------------+--------------------------------+ | Current Differential protection | Attribute | | Requirement | | +----------------------------------+--------------------------------+ | One way maximum delay | 5 ms | | Asymetric delay Required | Yes | | Maximum jitter | less than 250 us (750us for | | | legacy IED) | | Topology | Point to point, point to | | | Multi-point | | Bandwidth | 64 Kbps | | Availability | 99.9999 | | precise timing required | Yes | | Recovery time on node failure | less than 50ms - hitless | | performance management | Yes, Mandatory | | Redundancy | Yes | | Packet loss | 0.1% | +----------------------------------+--------------------------------+ - Table 3: Current Differential Protection requirements + Table 3: Current Differential Protection metrics -3.2.2.1.4. Distance Protection Scheme +3.1.1.1.7. Distance Protection Scheme Distance (Impedance Relay) protection scheme is based on voltage and - current measurements. A fault on a circuit will generally create a - sag in the voltage level. If the ratio of voltage to current - measured at the protection relay terminals, which equates to an - impedance element, falls within a set threshold the circuit breaker - will operate. The operating characteristics of this protection are - based on the line characteristics. This means that when a fault - appears on the line, the impedance setting in the relay is compared - to the apparent impedance of the line from the relay terminals to the - fault. If the relay setting is determined to be below the apparent - impedance it is determined that the fault is within the zone of - protection. When the transmission line length is under a minimum - length, distance protection becomes more difficult to coordinate. In - these instances the best choice of protection is current differential - protection. + current measurements. The network metrics are similar (but not + identical to) Current Differential protection. +-------------------------------+-----------------------------------+ | Distance protection | Attribute | | Requirement | | +-------------------------------+-----------------------------------+ | One way maximum delay | 5 ms | | Asymetric delay Required | No | | Maximum jitter | Not critical | | Topology | Point to point, point to Multi- | | | point | @@ -988,39 +810,39 @@ | Availability | 99.9999 | | precise timing required | Yes | | Recovery time on node failure | less than 50ms - hitless | | performance management | Yes, Mandatory | | Redundancy | Yes | | Packet loss | 0.1% | +-------------------------------+-----------------------------------+ Table 4: Distance Protection requirements -3.2.2.1.5. Inter-Substation Protection Signaling +3.1.1.1.8. Inter-Substation Protection Signaling This use case describes the exchange of Sampled Value and/or GOOSE (Generic Object Oriented Substation Events) message between Intelligent Electronic Devices (IED) in two substations for protection and tripping coordination. The two IEDs are in a master- slave mode. The Current Transformer or Voltage Transformer (CT/VT) in one substation sends the sampled analog voltage or current value to the - Merging Unit (MU) over hard wire. The merging unit sends the time- - synchronized 61850-9-2 sampled values to the slave IED. The slave - IED forwards the information to the Master IED in the other - substation. The master IED makes the determination (for example - based on sampled value differentials) to send a trip command to the - originating IED. Once the slave IED/Relay receives the GOOSE trip - for breaker tripping, it opens the breaker. It then sends a - confirmation message back to the master. All data exchanges between - IEDs are either through Sampled Value and/or GOOSE messages. + Merging Unit (MU) over hard wire. The MU sends the time-synchronized + 61850-9-2 sampled values to the slave IED. The slave IED forwards + the information to the Master IED in the other substation. The + master IED makes the determination (for example based on sampled + value differentials) to send a trip command to the originating IED. + Once the slave IED/Relay receives the GOOSE trip for breaker + tripping, it opens the breaker. It then sends a confirmation message + back to the master. All data exchanges between IEDs are either + through Sampled Value and/or GOOSE messages. +----------------------------------+--------------------------------+ | Inter-Substation protection | Attribute | | Requirement | | +----------------------------------+--------------------------------+ | One way maximum delay | 5 ms | | Asymetric delay Required | No | | Maximum jitter | Not critical | | Topology | Point to point, point to | | | Multi-point | @@ -1028,35 +850,34 @@ | Availability | 99.9999 | | precise timing required | Yes | | Recovery time on node failure | less than 50ms - hitless | | performance management | Yes, Mandatory | | Redundancy | Yes | | Packet loss | 1% | +----------------------------------+--------------------------------+ Table 5: Inter-Substation Protection requirements -3.2.2.1.6. Intra-Substation Process Bus Communications +3.1.1.2. Intra-Substation Process Bus Communications This use case describes the data flow from the CT/VT to the IEDs in - the substation via the merging unit (MU). The CT/VT in the - substation send the sampled value (analog voltage or current) to the - Merging Unit (MU) over hard wire. The merging unit sends the time- - synchronized 61850-9-2 sampled values to the IEDs in the substation - in GOOSE message format. The GPS Master Clock can send 1PPS or - IRIG-B format to MU through serial port, or IEEE 1588 protocol via - network. Process bus communication using 61850 simplifies - connectivity within the substation and removes the requirement for - multiple serial connections and removes the slow serial bus - architectures that are typically used. This also ensures increased - flexibility and increased speed with the use of multicast messaging - between multiple devices. + the substation via the MU. The CT/VT in the substation send the + sampled value (analog voltage or current) to the MU over hard wire. + The MU sends the time-synchronized 61850-9-2 sampled values to the + IEDs in the substation in GOOSE message format. The GPS Master Clock + can send 1PPS or IRIG-B format to the MU through a serial port or + IEEE 1588 protocol via a network. Process bus communication using + 61850 simplifies connectivity within the substation and removes the + requirement for multiple serial connections and removes the slow + serial bus architectures that are typically used. This also ensures + increased flexibility and increased speed with the use of multicast + messaging between multiple devices. +----------------------------------+--------------------------------+ | Intra-Substation protection | Attribute | | Requirement | | +----------------------------------+--------------------------------+ | One way maximum delay | 5 ms | | Asymetric delay Required | No | | Maximum jitter | Not critical | | Topology | Point to point, point to | | | Multi-point | @@ -1064,21 +885,21 @@ | Availability | 99.9999 | | precise timing required | Yes | | Recovery time on Node failure | less than 50ms - hitless | | performance management | Yes, Mandatory | | Redundancy | Yes - No | | Packet loss | 0.1% | +----------------------------------+--------------------------------+ Table 6: Intra-Substation Protection requirements -3.2.2.1.7. Wide Area Monitoring and Control Systems +3.1.1.3. Wide Area Monitoring and Control Systems The application of synchrophasor measurement data from Phasor Measurement Units (PMU) to Wide Area Monitoring and Control Systems promises to provide important new capabilities for improving system stability. Access to PMU data enables more timely situational awareness over larger portions of the grid than what has been possible historically with normal SCADA (Supervisory Control and Data Acquisition) data. Handling the volume and real-time nature of synchrophasor data presents unique challenges for existing application architectures. Wide Area management System (WAMS) makes @@ -1107,30 +928,29 @@ | Recovery time on | less than 50ms - hitless | | Node failure | | | performance | Yes, Mandatory | | management | | | Redundancy | Yes | | Packet loss | 1% | +----------------------+--------------------------------------------+ Table 7: WAMS Special Communication Requirements -3.2.2.1.8. IEC 61850 WAN engineering guidelines requirement +3.1.1.4. IEC 61850 WAN engineering guidelines requirement classification The IEC (International Electrotechnical Commission) has recently published a Technical Report which offers guidelines on how to define and deploy Wide Area Networks for the interconnections of electric substations, generation plants and SCADA operation centers. The IEC 61850-90-12 is providing a classification of WAN communication - requirements into 4 classes. You will find herafter the table - summarizing these requirements: + requirements into 4 classes. Table 8 summarizes these requirements: +----------------+------------+------------+------------+-----------+ | WAN | Class WA | Class WB | Class WC | Class WD | | Requirement | | | | | +----------------+------------+------------+------------+-----------+ | Application | EHV (Extra | HV (High | MV (Medium | General | | field | High | Voltage) | Voltage) | purpose | | | Voltage) | | | | | Latency | 5 ms | 10 ms | 100 ms | > 100 ms | | Jitter | 10 us | 100 us | 1 ms | 10 ms | @@ -1142,116 +962,71 @@ | | 10-6 | 10-4 | | | | Unavailability | 10-7 to | 10-5 to | 10-3 | | | | 10-6 | 10-4 | | | | Recovery delay | Zero | 50 ms | 5 s | 50 s | | Cyber security | extremely | High | Medium | Medium | | | high | | | | +----------------+------------+------------+------------+-----------+ Table 8: 61850-90-12 Communication Requirements; Courtesy of IEC -3.2.2.2. Distribution use case - -3.2.2.2.1. Fault Location Isolation and Service Restoration (FLISR) - - As the name implies, Fault Location, Isolation, and Service - Restoration (FLISR) refers to the ability to automatically locate the - fault, isolate the fault, and restore service in the distribution - network. It is a self-healing feature whose purpose is to minimize - the impact of faults by serving portions of the loads on the affected - circuit by switching to other circuits. It reduces the number of - customers that experience a sustained power outage by reconfiguring - distribution circuits. This will likely be the first wide spread - application of distributed intelligence in the grid. Secondary - substations can be connected to multiple primary substations. - Normally, static power switch statuses (open/closed) in the network - dictate the power flow to secondary substations. Reconfiguring the - network in the event of a fault is typically done manually on site to - operate switchgear to energize/de-energize alternate paths. - Automating the operation of substation switchgear allows the utility - to have a more dynamic network where the flow of power can be altered - under fault conditions but also during times of peak load. It allows - the utility to shift peak loads around the network. Or, to be more - precise, alters the configuration of the network to move loads - between different primary substations. The FLISR capability can be - enabled in two modes: - - o Managed centrally from DMS (Distribution Management System), or - - o Executed locally through distributed control via intelligent - switches and fault sensors. - - There are 3 distinct sub-functions that are performed: - - 1. Fault Location Identification - - This sub-function is initiated by SCADA inputs, such as lockouts, - fault indications/location, and, also, by input from the Outage - Management System (OMS), and in the future by inputs from fault- - predicting devices. It determines the specific protective device, - which has cleared the sustained fault, identifies the de-energized - sections, and estimates the probable location of the actual or the - expected fault. It distinguishes faults cleared by controllable - protective devices from those cleared by fuses, and identifies - momentary outages and inrush/cold load pick-up currents. This step - is also referred to as Fault Detection Classification and Location - (FDCL). This step helps to expedite the restoration of faulted - sections through fast fault location identification and improved - diagnostic information available for crew dispatch. Also provides - visualization of fault information to design and implement a - switching plan to isolate the fault. - - 2. Fault Type Determination - - I. Indicates faults cleared by controllable protective devices by - distinguishing between: - - a. Faults cleared by fuses +3.1.2. Generation Use Case - b. Momentary outages + The electrical power generation frequency should be maintained within + a very narrow band. Deviations from the acceptable frequency range + are detected and the required signals are sent to the power plants + for frequency regulation. - c. Inrush/cold load current + Automatic generation control (AGC) is a system for adjusting the + power output of generators at different power plants, in response to + changes in the load. - II. Determines the faulted sections based on SCADA fault indications - and protection lockout signals + +---------------------------------------------------+---------------+ + | FCAG (Frequency Control Automatic Generation) | Attribute | + | Requirement | | + +---------------------------------------------------+---------------+ + | One way maximum delay | 500 ms | + | Asymetric delay Required | No | + | Maximum jitter | Not critical | + | Topology | Point to | + | | point | + | Bandwidth | 20 Kbps | + | Availability | 99.999 | + | precise timing required | Yes | + | Recovery time on Node failure | N/A | + | performance management | Yes, | + | | Mandatory | + | Redundancy | Yes | + | Packet loss | 1% | + +---------------------------------------------------+---------------+ - III. Increases the accuracy of the fault location estimation based - on SCADA fault current measurements and real-time fault analysis + Table 9: FCAG Communication Requirements - 3. Fault Isolation and Service Restoration - Once the location and type of the fault has been pinpointed, the - systems will attempt to isolate the fault and restore the non-faulted - section of the network. This can have three modes of operation: +3.1.3. Distribution use case - I. Closed-loop mode : This is initiated by the Fault location sub- - function. It generates a switching order (i.e., sequence of - switching) for the remotely controlled switching devices to isolate - the faulted section, and restore service to the non-faulted sections. - The switching order is automatically executed via SCADA. +3.1.3.1. Fault Location Isolation and Service Restoration (FLISR) - II. Advisory mode : This is initiated by the Fault location sub- - function. It generates a switching order for remotely and manually - controlled switching devices to isolate the faulted section, and - restore service to the non-faulted sections. The switching order is - presented to operator for approval and execution. + Fault Location, Isolation, and Service Restoration (FLISR) refers to + the ability to automatically locate the fault, isolate the fault, and + restore service in the distribution network. This will likely be the + first widespread application of distributed intelligence in the grid. - III. Study mode : the operator initiates this function. It analyzes - a saved case modified by the operator, and generates a switching - order under the operating conditions specified by the operator. + Static power switch status (open/closed) in the network dictates the + power flow to secondary substations. Reconfiguring the network in + the event of a fault is typically done manually on site to energize/ + de-energize alternate paths. Automating the operation of substation + switchgear allows the flow of power to be altered automatically under + fault conditions. - With the increasing volume of data that are collected through fault - sensors, utilities will use Big Data query and analysis tools to - study outage information to anticipate and prevent outages by - detecting failure patterns and their correlation with asset age, - type, load profiles, time of day, weather conditions, and other - conditions to discover conditions that lead to faults and take the - necessary preventive and corrective measures. + FLISR can be managed centrally from a Distribution Management System + (DMS) or executed locally through distributed control via intelligent + switches and fault sensors. +----------------------+--------------------------------------------+ | FLISR Requirement | Attribute | +----------------------+--------------------------------------------+ | One way maximum | 80 ms | | delay | | | Asymetric delay | No | | Required | | | Maximum jitter | 40 ms | | Topology | Point to point, point to Multi-point, | @@ -1261,245 +1036,301 @@ | precise timing | Yes | | required | | | Recovery time on | Depends on customer impact | | Node failure | | | performance | Yes, Mandatory | | management | | | Redundancy | Yes | | Packet loss | 0.1% | +----------------------+--------------------------------------------+ - Table 9: FLISR Communication Requirements + Table 10: FLISR Communication Requirements -3.2.2.3. Generation use case +3.2. Electrical Utilities Today -3.2.2.3.1. Frequency Control / Automatic Generation Control (AGC) + Many utilities still rely on complex environments formed of multiple + application-specific proprietary networks, including TDM networks. - The system frequency should be maintained within a very narrow band. - Deviations from the acceptable frequency range are detected and - forwarded to the Load Frequency Control (LFC) system so that required - up or down generation increase / decrease pulses can be sent to the - power plants for frequency regulation. The trend in system frequency - is a measure of mismatch between demand and generation, and is a - necessary parameter for load control in interconnected systems. + In this kind of environment there is no mixing of OT and IT + applications on the same network, and information is siloed between + operational areas. - Automatic generation control (AGC) is a system for adjusting the - power output of generators at different power plants, in response to - changes in the load. Since a power grid requires that generation and - load closely balance moment by moment, frequent adjustments to the - output of generators are necessary. The balance can be judged by - measuring the system frequency; if it is increasing, more power is - being generated than used, and all machines in the system are - accelerating. If the system frequency is decreasing, more demand is - on the system than the instantaneous generation can provide, and all - generators are slowing down. + Specific calibration of the full chain is required, which is costly. - Where the grid has tie lines to adjacent control areas, automatic - generation control helps maintain the power interchanges over the tie - lines at the scheduled levels. The AGC takes into account various - parameters including the most economical units to adjust, the - coordination of thermal, hydroelectric, and other generation types, - and even constraints related to the stability of the system and - capacity of interconnections to other power grids. + This kind of environment prevents utility operations from realizing + the operational efficiency benefits, visibility, and functional + integration of operational information across grid applications and + data networks. - For the purpose of AGC we use static frequency measurements and - averaging methods are used to get a more precise measure of system - frequency in steady-state conditions. + In addition, there are many security-related issues as discussed in + the following section. - During disturbances, more real-time dynamic measurements of system - frequency are taken using PMUs, especially when different areas of - the system exhibit different frequencies. But that is outside the - scope of this use case. +3.2.1. Security Current Practices and Limitations - +---------------------------------------------------+---------------+ - | FCAG (Frequency Control Automatic Generation) | Attribute | - | Requirement | | - +---------------------------------------------------+---------------+ - | One way maximum delay | 500 ms | - | Asymetric delay Required | No | - | Maximum jitter | Not critical | - | Topology | Point to | - | | point | - | Bandwidth | 20 Kbps | - | Availability | 99.999 | - | precise timing required | Yes | - | Recovery time on Node failure | N/A | - | performance management | Yes, | - | | Mandatory | - | Redundancy | Yes | - | Packet loss | 1% | - +---------------------------------------------------+---------------+ + Grid monitoring and control devices are already targets for cyber + attacks, and legacy telecommunications protocols have many intrinsic + network-related vulnerabilities. For example, DNP3, Modbus, + PROFIBUS/PROFINET, and other protocols are designed around a common + paradigm of request and respond. Each protocol is designed for a + master device such as an HMI (Human Machine Interface) system to send + commands to subordinate slave devices to retrieve data (reading + inputs) or control (writing to outputs). Because many of these + protocols lack authentication, encryption, or other basic security + measures, they are prone to network-based attacks, allowing a + malicious actor or attacker to utilize the request-and-respond system + as a mechanism for command-and-control like functionality. Specific + security concerns common to most industrial control, including + utility telecommunication protocols include the following: - Table 10: FCAG Communication Requirements + o Network or transport errors (e.g. malformed packets or excessive + latency) can cause protocol failure. -3.2.3. Specific Network topologies of Smart Grid Applications + o Protocol commands may be available that are capable of forcing + slave devices into inoperable states, including powering-off + devices, forcing them into a listen-only state, disabling + alarming. + + o Protocol commands may be available that are capable of restarting + communications and otherwise interrupting processes. + + o Protocol commands may be available that are capable of clearing, + erasing, or resetting diagnostic information such as counters and + diagnostic registers. + + o Protocol commands may be available that are capable of requesting + sensitive information about the controllers, their configurations, + or other need-to-know information. + + o Most protocols are application layer protocols transported over + TCP; therefore it is easy to transport commands over non-standard + ports or inject commands into authorized traffic flows. + + o Protocol commands may be available that are capable of + broadcasting messages to many devices at once (i.e. a potential + DoS). + + o Protocol commands may be available to query the device network to + obtain defined points and their values (i.e. a configuration + scan). + + o Protocol commands may be available that will list all available + function codes (i.e. a function scan). + + These inherent vulnerabilities, along with increasing connectivity + between IT an OT networks, make network-based attacks very feasible. + + Simple injection of malicious protocol commands provides control over + the target process. Altering legitimate protocol traffic can also + alter information about a process and disrupt the legitimate controls + that are in place over that process. A man-in-the-middle attack + could provide both control over a process and misrepresentation of + data back to operator consoles. + +3.3. Electrical Utilities Future + + The business and technology trends that are sweeping the utility + industry will drastically transform the utility business from the way + it has been for many decades. At the core of many of these changes + is a drive to modernize the electrical grid with an integrated + telecommunications infrastructure. However, interoperability + concerns, legacy networks, disparate tools, and stringent security + requirements all add complexity to the grid transformation. Given + the range and diversity of the requirements that should be addressed + by the next generation telecommunications infrastructure, utilities + need to adopt a holistic architectural approach to integrate the + electrical grid with digital telecommunications across the entire + power delivery chain. + + The key to modernizing grid telecommunications is to provide a + common, adaptable, multi-service network infrastructure for the + entire utility organization. Such a network serves as the platform + for current capabilities while enabling future expansion of the + network to accommodate new applications and services. + + To meet this diverse set of requirements, both today and in the + future, the next generation utility telecommunnications network will + be based on open-standards-based IP architecture. An end-to-end IP + architecture takes advantage of nearly three decades of IP technology + development, facilitating interoperability across disparate networks + and devices, as it has been already demonstrated in many mission- + critical and highly secure networks. + + IPv6 is seen as a future telecommunications technology for the Smart + Grid; the IEC (International Electrotechnical Commission) and + different National Committees have mandated a specific adhoc group + (AHG8) to define the migration strategy to IPv6 for all the IEC TC57 + power automation standards. + +3.3.1. Migration to Packet-Switched Network + + Throughout the world, utilities are increasingly planning for a + future based on smart grid applications requiring advanced + telecommunications systems. Many of these applications utilize + packet connectivity for communicating information and control signals + across the utility's Wide Area Network (WAN), made possible by + technologies such as multiprotocol label switching (MPLS). The data + that traverses the utility WAN includes: + + o Grid monitoring, control, and protection data + + o Non-control grid data (e.g. asset data for condition-based + monitoring) + + o Physical safety and security data (e.g. voice and video) + + o Remote worker access to corporate applications (voice, maps, + schematics, etc.) + + o Field area network backhaul for smart metering, and distribution + grid management + + o Enterprise traffic (email, collaboration tools, business + applications) + + WANs support this wide variety of traffic to and from substations, + the transmission and distribution grid, generation sites, between + control centers, and between work locations and data centers. To + maintain this rapidly expanding set of applications, many utilities + are taking steps to evolve present time-division multiplexing (TDM) + based and frame relay infrastructures to packet systems. Packet- + based networks are designed to provide greater functionalities and + higher levels of service for applications, while continuing to + deliver reliability and deterministic (real-time) traffic support. + +3.3.2. Telecommunications Trends + + These general telecommunications topics are in addition to the use + cases that have been addressed so far. These include both current + and future telecommunications related topics that should be factored + into the network architecture and design. + +3.3.2.1. General Telecommunications Requirements + + o IP Connectivity everywhere + + o Monitoring services everywhere and from different remote centers + + o Move services to a virtual data center + + o Unify access to applications / information from the corporate + network + + o Unify services + + o Unified Communications Solutions + + o Mix of fiber and microwave technologies - obsolescence of SONET/ + SDH or TDM + + o Standardize grid telecommunications protocol to opened standard to + ensure interoperability + + o Reliable Telecommunications for Transmission and Distribution + Substations + + o IEEE 1588 time synchronization Client / Server Capabilities + + o Integration of Multicast Design + + o QoS Requirements Mapping + + o Enable Future Network Expansion + + o Substation Network Resilience + + o Fast Convergence Design + + o Scalable Headend Design + + o Define Service Level Agreements (SLA) and Enable SLA Monitoring + + o Integration of 3G/4G Technologies and future technologies + + o Ethernet Connectivity for Station Bus Architecture + + o Ethernet Connectivity for Process Bus Architecture + + o Protection, teleprotection and PMU (Phaser Measurement Unit) on IP + +3.3.2.2. Specific Network topologies of Smart Grid Applications Utilities often have very large private telecommunications networks. It covers an entire territory / country. The main purpose of the network, until now, has been to support transmission network monitoring, control, and automation, remote control of generation - sites, and providing FCAPS (Fault. Configuration. Accounting. - Performance. Security) services from centralized network operation + sites, and providing FCAPS (Fault, Configuration, Accounting, + Performance, Security) services from centralized network operation centers. Going forward, one network will support operation and maintenance of electrical networks (generation, transmission, and distribution), voice and data services for ten of thousands of employees and for exchange with neighboring interconnections, and administrative services. To meet those requirements, utility may deploy several physical networks leveraging different technologies across the country: an optical network and a microwave network for instance. Each protection and automatism system between two points has two telecommunications circuits, one on each network. Path diversity between two substations is key. Regardless of the event type (hurricane, ice storm, etc.), one path shall stay available so the - SPS can still operate. + system can still operate. In the optical network, signals are transmitted over more than tens of thousands of circuits using fiber optic links, microwave and telephone cables. This network is the nervous system of the utility's power transmission operations. The optical network represents ten of thousands of km of cable deployed along the power - lines. - - Due to vast distances between transmission substations (for example - as far as 280km apart), the fiber signal can be amplified to reach a - distance of 280 km without attenuation. + lines, with individual runs as long as 280 km. -3.2.4. Precision Time Protocol +3.3.2.3. Precision Time Protocol Some utilities do not use GPS clocks in generation substations. One of the main reasons is that some of the generation plants are 30 to 50 meters deep under ground and the GPS signal can be weak and unreliable. Instead, atomic clocks are used. Clocks are synchronized amongst each other. Rubidium clocks provide clock and - 1ms timestamps for IRIG-B. Some companies plan to transition to the - Precision Time Protocol (IEEE 1588), distributing the synchronization - signal over the IP/MPLS network. + 1ms timestamps for IRIG-B. - The Precision Time Protocol (PTP) is defined in IEEE standard 1588. - PTP is applicable to distributed systems consisting of one or more - nodes, communicating over a network. Nodes are modeled as containing - a real-time clock that may be used by applications within the node - for various purposes such as generating time-stamps for data or - ordering events managed by the node. The protocol provides a - mechanism for synchronizing the clocks of participating nodes to a - high degree of accuracy and precision. + Some companies plan to transition to the Precision Time Protocol + (PTP, [IEEE1588]), distributing the synchronization signal over the + IP/MPLS network. PTP provides a mechanism for synchronizing the + clocks of participating nodes to a high degree of accuracy and + precision. PTP operates based on the following assumptions : It is assumed that the network eliminates cyclic forwarding of PTP - messages within each communication path (e.g., by using a spanning - tree protocol). PTP eliminates cyclic forwarding of PTP messages - between communication paths. + messages within each communication path (e.g. by using a spanning + tree protocol). PTP is tolerant of an occasional missed message, duplicated message, or message that arrived out of order. However, PTP assumes that such impairments are relatively rare. - PTP was designed assuming a multicast communication model. PTP - also supports a unicast communication model as long as the + PTP was designed assuming a multicast communication model, however + PTP also supports a unicast communication model as long as the behavior of the protocol is preserved. Like all message-based time transfer protocols, PTP time accuracy - is degraded by asymmetry in the paths taken by event messages. - Asymmetry is not detectable by PTP, however, if known, PTP - corrects for asymmetry. - - A time-stamp event is generated at the time of transmission and - reception of any event message. The time-stamp event occurs when the - message's timestamp point crosses the boundary between the node and - the network. + is degraded by delay asymmetry in the paths taken by event + messages. Asymmetry is not detectable by PTP, however, if such + delays are known a priori, PTP can correct for asymmetry. IEC 61850 will recommend the use of the IEEE PTP 1588 Utility Profile - (as defined in IEC 62439-3 Annex B) which offers the support of - redundant attachment of clocks to Paralell Redundancy Protcol (PRP) + (as defined in [IEC62439-3:2012] Annex B) which offers the support of + redundant attachment of clocks to Parallel Redundancy Protcol (PRP) and High-availability Seamless Redundancy (HSR) networks. -3.3. IANA Considerations - - This memo includes no request to IANA. - -3.4. Security Considerations - -3.4.1. Current Practices and Their Limitations - - Grid monitoring and control devices are already targets for cyber - attacks and legacy telecommunications protocols have many intrinsic - network related vulnerabilities. DNP3, Modbus, PROFIBUS/PROFINET, - and other protocols are designed around a common paradigm of request - and respond. Each protocol is designed for a master device such as - an HMI (Human Machine Interface) system to send commands to - subordinate slave devices to retrieve data (reading inputs) or - control (writing to outputs). Because many of these protocols lack - authentication, encryption, or other basic security measures, they - are prone to network-based attacks, allowing a malicious actor or - attacker to utilize the request-and-respond system as a mechanism for - command-and-control like functionality. Specific security concerns - common to most industrial control, including utility - telecommunication protocols include the following: - - o Network or transport errors (e.g. malformed packets or excessive - latency) can cause protocol failure. - - o Protocol commands may be available that are capable of forcing - slave devices into inoperable states, including powering-off - devices, forcing them into a listen-only state, disabling - alarming. - - o Protocol commands may be available that are capable of restarting - communications and otherwise interrupting processes. - - o Protocol commands may be available that are capable of clearing, - erasing, or resetting diagnostic information such as counters and - diagnostic registers. - - o Protocol commands may be available that are capable of requesting - sensitive information about the controllers, their configurations, - or other need-to-know information. - - o Most protocols are application layer protocols transported over - TCP; therefore it is easy to transport commands over non-standard - ports or inject commands into authorized traffic flows. - - o Protocol commands may be available that are capable of - broadcasting messages to many devices at once (i.e. a potential - DoS). - - o Protocol commands may be available to query the device network to - obtain defined points and their values (i.e. a configuration - scan). - - o Protocol commands may be available that will list all available - function codes (i.e. a function scan). - - o Bump in the wire (BITW) solutions : A hardware device is added to - provide IPSec services between two routers that are not capable of - IPSec functions. This special IPsec device will intercept then - intercept outgoing datagrams, add IPSec protection to them, and - strip it off incoming datagrams. BITW can all IPSec to legacy - hosts and can retrofit non-IPSec routers to provide security - benefits. The disadvantages are complexity and cost. - - These inherent vulnerabilities, along with increasing connectivity - between IT an OT networks, make network-based attacks very feasible. - Simple injection of malicious protocol commands provides control over - the target process. Altering legitimate protocol traffic can also - alter information about a process and disrupt the legitimate controls - that are in place over that process. A man- in-the-middle attack - could provide both control over a process and misrepresentation of - data back to operator consoles. - -3.4.2. Security Trends in Utility Networks +3.3.3. Security Trends in Utility Networks Although advanced telecommunications networks can assist in - transforming the energy industry, playing a critical role in + transforming the energy industry by playing a critical role in maintaining high levels of reliability, performance, and manageability, they also introduce the need for an integrated security infrastructure. Many of the technologies being deployed to support smart grid projects such as smart meters and sensors can increase the vulnerability of the grid to attack. Top security concerns for utilities migrating to an intelligent smart grid telecommunications platform center on the following trends: o Integration of distributed energy resources @@ -1514,69 +1345,89 @@ smart metering o Demand for new levels of customer service and energy management This development of a diverse set of networks to support the integration of microgrids, open-access energy competition, and the use of network-controlled devices is driving the need for a converged security infrastructure for all participants in the smart grid, including utilities, energy service providers, large commercial and industrial, as well as residential customers. Securing the assets of - electric power delivery systems, from the control center to the - substation, to the feeders and down to customer meters, requires an + electric power delivery systems (from the control center to the + substation, to the feeders and down to customer meters) requires an end-to-end security infrastructure that protects the myriad of telecommunications assets used to operate, monitor, and control power - flow and measurement. Cyber security refers to all the security - issues in automation and telecommunications that affect any functions - related to the operation of the electric power systems. - Specifically, it involves the concepts of: + flow and measurement. + + "Cyber security" refers to all the security issues in automation and + telecommunications that affect any functions related to the operation + of the electric power systems. Specifically, it involves the + concepts of: o Integrity : data cannot be altered undetectably o Authenticity : the telecommunications parties involved must be validated as genuine o Authorization : only requests and commands from the authorized users can be accepted by the system o Confidentiality : data must not be accessible to any unauthenticated users When designing and deploying new smart grid devices and - telecommunications systems, it's imperative to understand the various - impacts of these new components under a variety of attack situations - on the power grid. Consequences of a cyber attack on the grid - telecommunications network can be catastrophic. This is why security - for smart grid is not just an ad hoc feature or product, it's a - complete framework integrating both physical and Cyber security - requirements and covering the entire smart grid networks from - generation to distribution. Security has therefore become one of the - main foundations of the utility telecom network architecture and must - be considered at every layer with a defense-in-depth approach. - Migrating to IP based protocols is key to address these challenges - for two reasons: + telecommunications systems, it is imperative to understand the + various impacts of these new components under a variety of attack + situations on the power grid. Consequences of a cyber attack on the + grid telecommunications network can be catastrophic. This is why + security for smart grid is not just an ad hoc feature or product, + it's a complete framework integrating both physical and Cyber + security requirements and covering the entire smart grid networks + from generation to distribution. Security has therefore become one + of the main foundations of the utility telecom network architecture + and must be considered at every layer with a defense-in-depth + approach. Migrating to IP based protocols is key to address these + challenges for two reasons: - 1. IP enables a rich set of features and capabilities to enhance the + o IP enables a rich set of features and capabilities to enhance the security posture - 2. IP is based on open standards, which allows interoperability + o IP is based on open standards, which allows interoperability between different vendors and products, driving down the costs associated with implementing security solutions in OT networks. Securing OT (Operation technology) telecommunications over packet- switched IP networks follow the same principles that are foundational for securing the IT infrastructure, i.e., consideration must be given to enforcing electronic access control for both person-to-machine and machine-to-machine communications, and providing the appropriate levels of data privacy, device and platform integrity, and threat detection and mitigation. +3.4. Electrical Utilities Asks + + o Mixed L2 and L3 topologies + + o Deterministic behavior + + o Bounded latency and jitter + + o High availability, low recovery time + + o Redundancy, low packet loss + + o Precise timing + + o Centralized computing of deterministic paths + + o Distributed configuration may also be useful + 4. Building Automation Systems 4.1. Use Case Description A Building Automation System (BAS) manages equipment and sensors in a building for improving residents' comfort, reducing energy consumption, and responding to failures and emergencies. For example, the BAS measures the temperature of a room using sensors and then controls the HVAC (heating, ventilating, and air conditioning) to maintain a set temperature and minimize energy consumption. @@ -2711,24 +2561,26 @@ created rather infrequently, on the order of ~10 times per day / week / month. Most of these critical control/data streams get created at machine startup, however flexibility is also needed during runtime, for example when adding or removing a machine. Going forward as production systems become more flexible, we expect a significant increase in the rate at which streams are created, changed and destroyed. 8.3. Industrial M2M Future - We would like to see the various proprietary networks replaced with a - converged IP-standards-based network with deterministic properties - that can satisfy the timing, security and reliability constraints - described above. + We would like to see a converged IP-standards-based network with + deterministic properties that can satisfy the timing, security and + reliability constraints described above. Today's proprietary + networks could then be interfaced to such a network via gateways or, + in the case of new installations, devices could be connected directly + to the converged network. 8.4. Industrial M2M Asks o Converged IP-based network o Deterministic behavior (bounded latency and jitter ) o High availability (presumably through redundancy) (99.999 %) o Low message delivery time (100us - 50ms) @@ -2963,21 +2815,21 @@ . [HART] www.hartcomm.org, "Highway Addressable remote Transducer, a group of specifications for industrial process and control devices administered by the HART Foundation". [I-D.finn-detnet-architecture] Finn, N., Thubert, P., and M. Teener, "Deterministic Networking Architecture", draft-finn-detnet- - architecture-02 (work in progress), November 2015. + architecture-03 (work in progress), March 2016. [I-D.finn-detnet-problem-statement] Finn, N. and P. Thubert, "Deterministic Networking Problem Statement", draft-finn-detnet-problem-statement-04 (work in progress), October 2015. [I-D.ietf-6tisch-6top-interface] Wang, Q. and X. Vilajosana, "6TiSCH Operation Sublayer (6top) Interface", draft-ietf-6tisch-6top-interface-04 (work in progress), July 2015.