rfc9599.original   rfc9599.txt 
Transport Area Working Group B. Briscoe Internet Engineering Task Force (IETF) B. Briscoe
Internet-Draft Independent Request for Comments: 9599 Independent
Updates: 3819 (if approved) J. Kaippallimalil BCP: 89 J. Kaippallimalil
Intended status: Best Current Practice Futurewei Updates: 3819 Futurewei
Expires: 7 June 2024 5 December 2023 Category: Best Current Practice June 2024
ISSN: 2070-1721
Guidelines for Adding Congestion Notification to Protocols that Guidelines for Adding Congestion Notification to Protocols That
Encapsulate IP Encapsulate IP
draft-ietf-tsvwg-ecn-encap-guidelines-22
Abstract Abstract
The purpose of this document is to guide the design of congestion The purpose of this document is to guide the design of congestion
notification in any lower layer or tunnelling protocol that notification in any lower-layer or tunnelling protocol that
encapsulates IP. The aim is for explicit congestion signals to encapsulates IP. The aim is for explicit congestion signals to
propagate consistently from lower layer protocols into IP. Then the propagate consistently from lower-layer protocols into IP. Then, the
IP internetwork layer can act as a portability layer to carry IP internetwork layer can act as a portability layer to carry
congestion notification from non-IP-aware congested nodes up to the congestion notifications from non-IP-aware congested nodes up to the
transport layer (L4). Following these guidelines should assure transport layer (L4). Following these guidelines should assure
interworking among IP layer and lower layer congestion notification interworking among IP layer and lower-layer congestion notification
mechanisms, whether specified by the IETF or other standards bodies. mechanisms specified by the IETF or other standards bodies. This
This document is included in BCP 89 and updates the single paragraph document is included in BCP 89 and updates the single paragraph of
of advice to subnetwork designers about ECN in Section 13 of RFC advice to subnetwork designers about Explicit Congestion Notification
3819, by replacing it with a reference to the whole of this document. (ECN) in Section 13 of RFC 3819 by replacing it with a reference to
this document.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This memo documents an Internet Best Current Practice.
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Engineering Task Force
and may be updated, replaced, or obsoleted by other documents at any (IETF). It represents the consensus of the IETF community. It has
time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Further information on
BCPs is available in Section 2 of RFC 7841.
This Internet-Draft will expire on 7 June 2024. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc9599.
Copyright Notice Copyright Notice
Copyright (c) 2023 IETF Trust and the persons identified as the Copyright (c) 2024 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/ Provisions Relating to IETF Documents
license-info) in effect on the date of publication of this document. (https://trustee.ietf.org/license-info) in effect on the date of
Please review these documents carefully, as they describe your rights publication of this document. Please review these documents
and restrictions with respect to this document. Code Components carefully, as they describe your rights and restrictions with respect
extracted from this document must include Revised BSD License text as to this document. Code Components extracted from this document must
described in Section 4.e of the Trust Legal Provisions and are include Revised BSD License text as described in Section 4.e of the
provided without warranty as described in the Revised BSD License. Trust Legal Provisions and are provided without warranty as described
in the Revised BSD License.
This document may contain material from IETF Documents or IETF
Contributions published or made publicly available before November
10, 2008. The person(s) controlling the copyright in some of this
material may not have granted the IETF Trust the right to allow
modifications of such material outside the IETF Standards Process.
Without obtaining an adequate license from the person(s) controlling
the copyright in such materials, this document may not be modified
outside the IETF Standards Process, and derivative works of it may
not be created outside the IETF Standards Process, except to format
it for publication as an RFC or to translate it into languages other
than English.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction
1.1. Update to RFC 3819 . . . . . . . . . . . . . . . . . . . 5 1.1. Update to RFC 3819
1.2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2. Scope
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 8 2. Terminology
3. Modes of Operation . . . . . . . . . . . . . . . . . . . . . 9 3. Modes of Operation
3.1. Feed-Forward-and-Up Mode . . . . . . . . . . . . . . . . 10 3.1. Feed-Forward-and-Up Mode
3.2. Feed-Up-and-Forward Mode . . . . . . . . . . . . . . . . 12 3.2. Feed-Up-and-Forward Mode
3.3. Feed-Backward Mode . . . . . . . . . . . . . . . . . . . 12 3.3. Feed-Backward Mode
3.4. Null Mode . . . . . . . . . . . . . . . . . . . . . . . . 14 3.4. Null Mode
4. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion 4. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion
Notification . . . . . . . . . . . . . . . . . . . . . . 14 Notification
4.1. IP-in-IP Tunnels with Shim Headers . . . . . . . . . . . 15 4.1. IP-in-IP Tunnels with Shim Headers
4.2. Wire Protocol Design: Indication of ECN Support . . . . . 16 4.2. Wire Protocol Design: Indication of ECN Support
4.3. Encapsulation Guidelines . . . . . . . . . . . . . . . . 19 4.3. Encapsulation Guidelines
4.4. Decapsulation Guidelines . . . . . . . . . . . . . . . . 21 4.4. Decapsulation Guidelines
4.5. Sequences of Similar Tunnels or Subnets . . . . . . . . . 22 4.5. Sequences of Similar Tunnels or Subnets
4.6. Reframing and Congestion Markings . . . . . . . . . . . . 23 4.6. Reframing and Congestion Markings
5. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion 5. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion
Notification . . . . . . . . . . . . . . . . . . . . . . 25 Notification
6. Feed-Backward Mode: Guidelines for Adding Congestion 6. Feed-Backward Mode: Guidelines for Adding Congestion
Notification . . . . . . . . . . . . . . . . . . . . . . 26 Notification
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27 7. IANA Considerations
8. Security Considerations . . . . . . . . . . . . . . . . . . . 28 8. Security Considerations
9. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 28 9. Conclusions
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 29 10. References
10.1. Normative References . . . . . . . . . . . . . . . . . . 29 10.1. Normative References
10.2. Informative References . . . . . . . . . . . . . . . . . 30 10.2. Informative References
Comments Solicited . . . . . . . . . . . . . . . . . . . . . . . 34 Acknowledgements
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 34 Contributors
Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Authors' Addresses
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 35
1. Introduction 1. Introduction
The benefits of Explicit Congestion Notification (ECN) described in The benefits of Explicit Congestion Notification (ECN) described in
[RFC8087] and summarized below can only be fully realized if support [RFC8087] and summarized below can only be fully realized if support
for ECN is added to the relevant subnetwork technology, as well as to for ECN is added to the relevant subnetwork technology, as well as to
IP. When a lower layer buffer drops a packet obviously it does not IP. When a lower-layer buffer drops a packet, it obviously does not
just drop at that layer; the packet disappears from all layers. In just drop at that layer; the packet disappears from all layers. In
contrast, when active queue management (AQM) at a lower layer marks a contrast, when active queue management (AQM) at a lower layer marks a
packet with ECN, the marking needs to be explicitly propagated up the packet with ECN, the marking needs to be explicitly propagated up the
layers. The same is true if AQM marks the outer header of a packet layers. The same is true if AQM marks the outer header of a packet
that encapsulates inner tunnelled headers. Forwarding ECN is not as that encapsulates inner tunnelled headers. Forwarding ECN is not as
straightforward as other headers because it has to be assumed ECN may straightforward as other headers because it has to be assumed ECN may
be only partially deployed. If a lower layer header that contains be only partially deployed. If a lower-layer header that contains
ECN congestion indications is stripped off by a subnet egress that is ECN congestion indications is stripped off by a subnet egress that is
not ECN-aware, or if the ultimate receiver or sender is not ECN- not ECN-aware, or if the ultimate receiver or sender is not ECN-
aware, congestion needs to be indicated by dropping the packet, not aware, congestion needs to be indicated by dropping the packet, not
marking it. marking it.
The purpose of this document is to guide the addition of congestion The purpose of this document is to guide the addition of congestion
notification to any subnet technology or tunnelling protocol, so that notifications to any subnet technology or tunnelling protocol so that
lower layer AQM algorithms can signal congestion explicitly and it lower-layer AQM algorithms can signal congestion explicitly and it
will propagate consistently into encapsulated (higher layer) headers, will propagate consistently into encapsulated (higher-layer) headers;
otherwise the signals will not reach their ultimate destination. otherwise, the signals will not reach their ultimate destination.
ECN is defined in the IP header (IPv4 and IPv6) [RFC3168] to allow a ECN is defined in the IP header (IPv4 and IPv6) [RFC3168] to allow a
resource to notify the onset of queue build-up without having to drop resource to notify the onset of queue buildup without having to drop
packets, by explicitly marking a proportion of packets with the packets by explicitly marking a proportion of packets with the
congestion experienced (CE) codepoint. congestion experienced (CE) codepoint.
Given a suitable marking scheme, ECN removes nearly all congestion Given a suitable marking scheme, ECN removes nearly all congestion
loss and it cuts delays for two main reasons: loss and it cuts delays for two main reasons:
* It avoids the delay when recovering from congestion losses, which * It avoids the delay when recovering from congestion losses, which
particularly benefits small flows or real-time flows, making their particularly benefits small flows or real-time flows, making their
delivery time predictably short [RFC2884]; delivery time predictably short [RFC2884].
* As ECN is used more widely by end-systems, it will gradually * As ECN is used more widely by end systems, it will gradually
remove the need to configure a degree of delay into buffers before remove the need to configure a degree of delay into buffers before
they start to notify congestion (the cause of bufferbloat). This they start to notify congestion (the cause of bufferbloat). This
is because drop involves a trade-off between sending a timely is because drop involves a trade-off between sending a timely
signal and trying to avoid impairment, whereas ECN is solely a signal and trying to avoid impairment, whereas ECN is solely a
signal not an impairment, so there is no harm triggering it signal and not an impairment, so there is no harm triggering it
earlier. earlier.
Some lower layer technologies (e.g. MPLS, Ethernet) are used to form Some lower-layer technologies (e.g., MPLS, Ethernet) are used to form
subnetworks with IP-aware nodes only at the edges. These networks subnetworks with IP-aware nodes only at the edges. These networks
are often sized so that it is rare for interior queues to overflow. are often sized so that it is rare for interior queues to overflow.
However, until recently this was more due to the inability of TCP to However, until recently, this was more due to the inability of TCP to
saturate the links. For many years, fixes such as window scaling saturate the links. For many years, fixes such as window scaling
[RFC7323] proved hard to deploy. And the Reno variant of TCP has [RFC7323] proved hard to deploy. The Reno variant of TCP has
remained in widespread use despite its inability to scale to high remained in widespread use despite its inability to scale to high
flow rates. However, now that modern operating systems are finally flow rates. However, now that modern operating systems are finally
capable of saturating interior links, even the buffers of well- capable of saturating interior links, even the buffers of well-
provisioned interior switches will need to signal episodes of provisioned interior switches will need to signal episodes of
queuing. queuing.
Propagation of ECN is defined for MPLS [RFC5129], and has been Propagation of ECN is defined for MPLS [RFC5129] and TRILL [RFC7780]
defined for TRILL [RFC7780], [I-D.ietf-trill-ecn-support], but it [RFC9600], but it has yet to be defined for a number of other
remains to be defined for a number of other subnetwork technologies. subnetwork technologies.
Similarly, ECN propagation is yet to be defined for many tunnelling Similarly, ECN propagation is yet to be defined for many tunnelling
protocols. [RFC6040] defines how ECN should be propagated for IP-in- protocols. [RFC6040] defines how ECN should be propagated for IP-in-
IPv4 [RFC2003], IP-in-IPv6 [RFC2473] and IPsec [RFC4301] tunnels, but IPv4 [RFC2003], IP-in-IPv6 [RFC2473], and IPsec [RFC4301] tunnels,
there are numerous other tunnelling protocols with a shim and/or a but there are numerous other tunnelling protocols with a shim and/or
layer 2 header between two IP headers (IPv4 or IPv6). Some address a Layer 2 (L2) header between two IP headers (IPv4 or IPv6). Some
ECN propagation between the IP headers, but many do not. This address ECN propagation between the IP headers, but many do not.
document gives guidance on how to address ECN propagation for future This document gives guidance on how to address ECN propagation for
tunnelling protocols, and a companion standards track specification future tunnelling protocols, and a companion Standards Track
[I-D.ietf-tsvwg-rfc6040update-shim] updates those existing IP-shim- specification [RFC9601] updates existing IP-shim-(L2)-IP protocols
(L2)-IP protocols that are under IETF change control and still widely that are under IETF change control and still widely used.
used.
Incremental deployment is the most delicate aspect when adding Incremental deployment is the most delicate aspect when adding
support for ECN. The original ECN protocol in IP [RFC3168] was support for ECN. The original ECN protocol in IP [RFC3168] was
carefully designed so that a congested buffer would not mark a packet carefully designed so that a congested buffer would not mark a packet
(rather than drop it) unless both source and destination hosts were (rather than drop it) unless both source and destination hosts were
ECN-capable. Otherwise, its congestion markings would never be ECN-capable. Otherwise, its congestion markings would never be
detected and congestion would just build up further. However, to detected and congestion would just build up further. However, to
support congestion marking below the IP layer or within tunnels, it support congestion marking below the IP layer or within tunnels, it
is not sufficient to only check that the two layer 4 transport end- is not sufficient to only check that the two layer 4 transport
points support ECN; correct operation also depends on the endpoints support ECN; correct operation also depends on the
decapsulator at each subnet or tunnel egress faithfully propagating decapsulator at each subnet or tunnel egress faithfully propagating
congestion notifications to the higher layer. Otherwise, a legacy congestion notifications to the higher layer. Otherwise, a legacy
decapsulator might silently fail to propagate any ECN signals from decapsulator might silently fail to propagate any ECN signals from
the outer to the forwarded header. Then the lost signals would never the outer to the forwarded header. Then, the lost signals would
be detected and again congestion would build up further. The never be detected and congestion would build up further. The
guidelines given later require protocol designers to carefully guidelines given later require protocol designers to carefully
consider incremental deployment, and suggest various safe approaches consider incremental deployment and suggest various safe approaches
for different circumstances. for different circumstances.
Of course, the IETF does not have standards authority over every link Of course, the IETF does not have standards authority over every
layer protocol. So this document gives guidelines for designing link-layer protocol; thus, this document gives guidelines for
propagation of congestion notification across the interface between designing propagation of congestion notifications across the
IP and protocols that may encapsulate IP (i.e. that can be layered interface between IP and protocols that may encapsulate IP (i.e.,
beneath IP). Each lower layer technology will exhibit different that can be layered beneath IP). Each lower-layer technology will
issues and compromises, so the IETF or the relevant standards body exhibit different issues and compromises, so the IETF or the relevant
must be free to define the specifics of each lower layer congestion standards body must be free to define the specifics of each lower-
notification scheme. Nonetheless, if the guidelines are followed, layer congestion notification scheme. Nonetheless, if the guidelines
congestion notification should interwork between different are followed, congestion notifications should interwork between
technologies, using IP in its role as a 'portability layer'. different technologies using IP in its role as a 'portability layer'.
Therefore, the capitalized terms 'SHOULD' or 'SHOULD NOT' are often Therefore, the capitalized terms 'SHOULD' or 'SHOULD NOT' are often
used in preference to 'MUST' or 'MUST NOT', because it is difficult used in preference to 'MUST' or 'MUST NOT' because it is difficult to
to know the compromises that will be necessary in each protocol know the compromises that will be necessary in each protocol design.
design. If a particular protocol design chooses not to follow a If a particular protocol design chooses not to follow a 'SHOULD' or
'SHOULD (NOT)' given in the advice below, it MUST include a sound 'SHOULD NOT' given in the advice below, it MUST include a sound
justification. justification.
It has not been possible to give common guidelines for all lower It has not been possible to give common guidelines for all lower-
layer technologies, because they do not all fit a common pattern. layer technologies because they do not all fit a common pattern.
Instead, they have been divided into a few distinct modes of Instead, they have been divided into a few distinct modes of
operation: feed-forward-and-upward; feed-upward-and-forward; feed- operation: feed-forward-and-up, feed-up-and-forward, feed-backward,
backward; and null mode. These modes are described in Section 3, and null mode. These modes are described in Section 3, and separate
then in the subsequent sections separate guidelines are given for guidelines are given for each mode in subsequent sections.
each mode.
1.1. Update to RFC 3819 1.1. Update to RFC 3819
This document updates the brief advice to subnetwork designers about This document updates the brief advice to subnetwork designers about
ECN in Section 13 of [RFC3819], by replacing the last two paragraphs ECN in Section 13 of [RFC3819] by adding this document (RFC 9599) as
with the following sentence: an informative reference and replacing the last two paragraphs with
the following sentence:
By following the guidelines in [this document], subnetwork
designers can enable a layer-2 protocol to participate in
congestion control without dropping packets via propagation of
explicit congestion notification (ECN [RFC3168]) to receivers.
and adding [this document] as an informative reference. {RFC Editor: | By following the guidelines in [RFC9599], subnetwork designers can
Please replace both instances of [this document] above with the | enable a layer-2 protocol to participate in congestion control
number of the present RFC when published.} | without dropping packets via propagation of Explicit Congestion
| Notification (ECN) [RFC3168] to receivers.
1.2. Scope 1.2. Scope
This document only concerns wire protocol processing of explicit This document only concerns wire protocol processing of explicit
notification of congestion. It makes no changes or recommendations notification of congestion. It makes no changes or recommendations
concerning algorithms for congestion marking or for congestion concerning algorithms for congestion marking or congestion response
response, because algorithm issues should be independent of the layer because algorithm issues should be independent of the layer that the
the algorithm operates in. algorithm operates in.
The default ECN semantics are described in [RFC3168] and updated by The default ECN semantics are described in [RFC3168] and updated by
[RFC8311]. Also, the guidelines for AQM designers [RFC7567] clarify [RFC8311]. Also, the guidelines for Active Queue Management (AQM)
the semantics of both drop and ECN signals from AQM algorithms. designers [RFC7567] clarify the semantics of both drop and ECN
[RFC4774] is the appropriate best current practice specification of signals from AQM algorithms. [RFC4774] is the appropriate best
how algorithms with alternative semantics for the ECN field can be current practice specification of how algorithms with alternative
partitioned from Internet traffic that uses the default ECN semantics for the ECN field can be partitioned from Internet traffic
semantics. There are two main examples for how alternative ECN that uses the default ECN semantics. There are two main examples for
semantics have been defined in practice: how alternative ECN semantics have been defined in practice:
* RFC 4774 suggests using the ECN field in combination with a * [RFC4774] suggests using the ECN field in combination with a
Diffserv codepoint such as in PCN [RFC6660], Voice over 3G [UTRAN] Diffserv codepoint, such as in Pre-Congestion Notification (PCN)
or Voice over LTE (VoLTE) [LTE-RA]; [RFC6660], Voice over 3G [UTRAN], or Voice over LTE (VoLTE)
[LTE-RA].
* RFC 8311 suggests using the ECT(1) codepoint of the ECN field to * [RFC8311] suggests using the ECT(1) codepoint of the ECN field to
indicate alternative semantics such as for the experimental Low indicate alternative semantics, such as for the experimental Low
Latency Low Loss Scalable throughput (L4S) service [RFC9331]). Latency, Low Loss, and Scalable throughput (L4S) service
[RFC9331]).
The aim is that the default rules for encapsulating and decapsulating The aim is that the default rules for encapsulating and decapsulating
the ECN field are sufficiently generic that tunnels and subnets will the ECN field are sufficiently generic that tunnels and subnets will
encapsulate and decapsulate packets without regard to how algorithms encapsulate and decapsulate packets without regard to how algorithms
elsewhere are setting or interpreting the semantics of the ECN field. elsewhere are setting or interpreting the semantics of the ECN field.
[RFC6040] updates RFC 4774 to allow alternative encapsulation and [RFC6040] updates [RFC4774] to allow alternative encapsulation and
decapsulation behaviours to be defined for alternative ECN semantics. decapsulation behaviors to be defined for alternative ECN semantics.
However, it reinforces the same point - that it is far preferable to However, it reinforces the same point -- it is far preferable to try
try to fit within the common ECN encapsulation and decapsulation to fit within the common ECN encapsulation and decapsulation
behaviours, because expecting all lower layer technologies and behaviors because expecting all lower-layer technologies and tunnels
tunnels to be updated is likely to be completely impractical. to be updated is likely to be completely impractical.
Alternative semantics for the ECN field can be defined to depend on Alternative semantics for the ECN field can be defined to depend on
the traffic class indicated by the DSCP. Therefore, correct the traffic class indicated by the Differentiated Services Code Point
propagation of congestion signals could depend on correct propagation (DSCP). Therefore, correct propagation of congestion signals could
of the DSCP between the layers and along the path. For instance, if depend on correct propagation of the DSCP between the layers and
the meaning of the ECN field depends on the DSCP (as in PCN or VoLTE) along the path. For instance, if the meaning of the ECN field
and if the outer DSCP is stripped on descapsulation, as in the pipe depends on the DSCP (as in PCN or VoLTE) and the outer DSCP is
model of [RFC2983], the special semantics of the ECN field would be stripped on descapsulation, as in the pipe model of [RFC2983], the
lost. Similarly, if the DSCP is changed at the boundary between special semantics of the ECN field would be lost. Similarly, if the
Diffserv domains, the special ECN semantics would also be lost. This DSCP is changed at the boundary between Diffserv domains, the special
is an important implication of the localized scope of most Diffserv ECN semantics would also be lost. This is an important implication
arrangements. In this document, correct propagation of traffic class of the localized scope of most Diffserv arrangements. In this
information is assumed, while what 'correct' means and how it is document, correct propagation of traffic class information is assumed
achieved is covered elsewhere (e.g. RFC 2983) and is outside the while the meaning of 'correct' and how it is achieved is covered
scope of the present document. elsewhere (e.g., [RFC2983]) and is outside the scope of this
document.
The guidelines in this document do ensure that common encapsulation The guidelines in this document do ensure that common encapsulation
and decapsulation rules are sufficiently generic to cover cases where and decapsulation rules are sufficiently generic to cover cases where
ECT(1) is used instead of ECT(0) to identify alternative ECN ECT(1) is used instead of ECT(0) to identify alternative ECN
semantics (as in L4S [RFC9331]) and where ECN marking algorithms use semantics (as in L4S [RFC9331]) and where ECN-marking algorithms use
ECT(1) to encode 3 severity levels into the ECN field (e.g. PCN ECT(1) to encode three severity levels into the ECN field (e.g., PCN
[RFC6660]) rather than the default of 2. All these different [RFC6660]) rather than the default of two. All these different
semantics for the ECN field work because it has been possible to semantics for the ECN field work because it has been possible to
define common default decapsulation rules that allow for all cases. define common default decapsulation rules that allow for all cases.
Note that the guidelines in this document do not necessarily require Note that the guidelines in this document do not necessarily require
the subnet wire protocol to be changed to add support for congestion the subnet wire protocol to be changed to add support for congestion
notification. For instance, the Feed-Up-and-Forward Mode notifications. For instance, the feed-up-and-forward mode
(Section 3.2) and the Null Mode (Section 3.4) do not. Another way to (Section 3.2) and the null mode (Section 3.4) do not. Another way to
add congestion notification without consuming header space in the add congestion notifications without consuming header space in the
subnet protocol might be to use a parallel control plane protocol. subnet protocol might be to use a parallel control plane protocol.
This document focuses on the congestion notification interface This document focuses on the congestion notification interface
between IP and lower layer or tunnel protocols that can encapsulate between IP and lower-layer or tunnel protocols that can encapsulate
IP, where the term 'IP' includes IPv4 or IPv6, unicast, multicast or IP, where the term 'IP' includes IPv4 or IPv6, unicast, multicast, or
anycast. However, it is likely that the guidelines will also be anycast. However, it is likely that the guidelines will also be
useful when a lower layer protocol or tunnel encapsulates itself, useful when a lower-layer protocol or tunnel encapsulates itself,
e.g. Ethernet MAC in MAC ([IEEE802.1Q]; previously 802.1ah) or when e.g., Ethernet Media Access Control (MAC) in MAC ([IEEE802.1Q];
it encapsulates other protocols. In the feed-backward mode, previously 802.1ah), or when it encapsulates other protocols. In the
propagation of congestion signals for multicast and anycast packets feed-backward mode, propagation of congestion signals for multicast
is out-of-scope (because the complexity would make it unlikely to be and anycast packets is out of scope (because the complexity would
attempted). make it unlikely to be attempted).
2. Terminology 2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in "OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here. capitals, as shown here.
Further terminology used within this document: Further terminology used within this document:
Protocol data unit (PDU): Information that is delivered as a unit Protocol data unit (PDU): Information that is delivered as a unit
among peer entities of a layered network consisting of protocol among peer entities of a layered network consisting of protocol
control information (typically a header) and possibly user data control information (typically a header) and possibly user data
(payload) of that layer. The scope of this document includes (payload) of that layer. The scope of this document includes
layer 2 and layer 3 networks, where the PDU is respectively termed Layer 2 and Layer 3 networks, where the PDU is respectively termed
a frame or a packet (or a cell in ATM). PDU is a general term for a frame or a packet (or a cell in ATM). PDU is a general term for
any of these. This definition also includes a payload with a shim any of these. This definition also includes a payload with a shim
header lying somewhere between layer 2 and 3. header lying somewhere between layer 2 and 3.
Transport: The end-to-end transmission control function, Transport: The end-to-end transmission control function,
conventionally considered at layer-4 in the OSI reference model. conventionally considered at layer 4 in the OSI reference model.
Given the audience for this document will often use the word Given the audience for this document will often use the word
transport to mean low level bit carriage, whenever the term is transport to mean low-level bit carriage, the term will be
used it will be qualified, e.g. 'L4 transport'. qualified whenever it is used, e.g., 'L4 transport'.
Encapsulator: The link or tunnel endpoint function that adds an Encapsulator: The link or tunnel endpoint function that adds an
outer header to a PDU (also termed the 'link ingress', the 'subnet outer header to a PDU (also termed the 'link ingress', the 'subnet
ingress', the 'ingress tunnel endpoint' or just the 'ingress' ingress', the 'ingress tunnel endpoint', or just the 'ingress'
where the context is clear). where the context is clear).
Decapsulator: The link or tunnel endpoint function that removes an Decapsulator: The link or tunnel endpoint function that removes an
outer header from a PDU (also termed the 'link egress', the outer header from a PDU (also termed the 'link egress', the
'subnet egress', the 'egress tunnel endpoint' or just the 'egress' 'subnet egress', the 'egress tunnel endpoint', or just the
where the context is clear). 'egress' where the context is clear).
Incoming header: The header of an arriving PDU before encapsulation. Incoming header: The header of an arriving PDU before encapsulation.
Outer header: The header added to encapsulate a PDU. Outer header: The header added to encapsulate a PDU.
Inner header: The header encapsulated by the outer header. Inner header: The header encapsulated by the outer header.
Outgoing header: The header forwarded by the decapsulator. Outgoing header: The header forwarded by the decapsulator.
CE: Congestion Experienced [RFC3168] CE: Congestion Experienced [RFC3168]
skipping to change at page 9, line 4 skipping to change at line 345
Inner header: The header encapsulated by the outer header. Inner header: The header encapsulated by the outer header.
Outgoing header: The header forwarded by the decapsulator. Outgoing header: The header forwarded by the decapsulator.
CE: Congestion Experienced [RFC3168] CE: Congestion Experienced [RFC3168]
ECT: ECN-Capable (L4) Transport [RFC3168] ECT: ECN-Capable (L4) Transport [RFC3168]
Not-ECT: Not ECN-Capable (L4) Transport [RFC3168] Not-ECT: Not ECN-Capable (L4) Transport [RFC3168]
Load Regulator: For each flow of PDUs, the transport function that Load Regulator: For each flow of PDUs, the transport function that
is capable of controlling the data rate. Typically located at the is capable of controlling the data rate. Typically located at the
data source, but in-path nodes can regulate load in some data source, but in-path nodes can regulate load in some
congestion control arrangements (e.g. admission control, policing congestion control arrangements (e.g., admission control, policing
nodes or transport circuit-breakers [RFC8084]). Note the term "a nodes, or transport circuit-breakers [RFC8084]). Note that "a
function capable of controlling the load" deliberately includes a function capable of controlling the load" deliberately includes a
transport that does not actually control the load responsively but transport that does not actually control the load responsively,
ideally it ought to (e.g. a sending application without congestion but ideally it ought to (e.g., a sending application without
control that uses UDP). congestion control that uses UDP).
ECN-PDU: A PDU at the IP layer or below with a capacity to signal ECN-PDU: A PDU at the IP layer or below with a capacity to signal
congestion that is part of a congestion control feedback loop congestion that is part of a congestion control feedback loop
within which all the nodes necessary to propagate the signal back within which all the nodes necessary to propagate the signal back
to the Load Regulator are capable of doing that propagation. An to the Load Regulator are capable of doing that propagation. An
IP packet with a non-zero ECN field implies that the endpoints are IP packet with a non-zero ECN field implies that the endpoints are
ECN-capable, so this would be an ECN-PDU. However, ECN-PDU is ECN-capable, so this would be an ECN-PDU. However, ECN-PDU is
intended to be a general term for a PDU at lower layers, as well intended to be a general term for a PDU at lower layers, as well
as at the IP layer. as at the IP layer.
Not-ECN-PDU: A PDU at the IP layer or below that is part of a Not-ECN-PDU: A PDU at the IP layer or below that is part of a
congestion control feedback loop that is not capable of congestion control feedback loop that is not capable of
propagating explicit congestion notification signals back to the propagating ECN signals back to the Load Regulator because at
Load Regulator, because at least one of the nodes necessary to least one of the nodes necessary to propagate the signals is
propagate the signals is incapable of doing that propagation. incapable of doing that propagation. Note that this definition is
Note that this definition is a property of the feedback-loop, not a property of the feedback loop, not necessarily of the PDU
necessarily of the PDU itself, because in some protocols the PDU itself; the PDU will self-describe the property in some protocols,
will self-describe the property, but in others the property might but in others, the property might be carried in a separate control
be carried in a separate control-plane context that is somehow plane context that is somehow bound to the PDU.
bound to the PDU.
3. Modes of Operation 3. Modes of Operation
This section sets down the different modes by which congestion This section sets down the different modes by which congestion
information is passed between the lower layer and the higher one. It information is passed between the lower layer and the higher one. It
acts as a reference framework for the following sections, which give acts as a reference framework for the following sections that give
normative guidelines for designers of explicit congestion normative guidelines for designers of ECN protocols, taking each mode
notification protocols, taking each mode in turn: in turn:
Feed-Forward-and-Up: Nodes feed forward congestion notification Feed-Forward-and-Up: Nodes feed forward congestion notifications
towards the egress within the lower layer then up and along the towards the egress within the lower layer, then up and along the
layers towards the end-to-end destination at the transport layer. layers towards the end-to-end destination at the transport layer.
The following local optimisation is possible: The following local optimization is possible:
Feed-Up-and-Forward: A lower layer switch feeds-up congestion Feed-Up-and-Forward: A lower-layer switch feeds up congestion
notification directly into the higher layer (e.g. into the ECN notifications directly into the higher layer (e.g., into the
field in the IP header), irrespective of whether the node is at ECN field in the IP header), irrespective of whether the node
the egress of a subnet. is at the egress of a subnet.
Feed-Backward: Nodes feed back congestion signals towards the Feed-Backward: Nodes feed back congestion signals towards the
ingress of the lower layer and (optionally) attempt to control ingress of the lower layer and (optionally) attempt to control
congestion within their own layer. congestion within their own layer.
Null: Nodes cannot experience congestion at the lower layer except Null: Nodes cannot experience congestion at the lower layer except
at ingress nodes (which are IP-aware or equivalently higher-layer- at ingress nodes (which are IP-aware or equivalently higher-layer-
aware). aware).
3.1. Feed-Forward-and-Up Mode 3.1. Feed-Forward-and-Up Mode
Like IP and MPLS, many subnet technologies are based on self- Like IP and MPLS, many subnet technologies are based on self-
contained protocol data units (PDUs) or frames sent unreliably. They contained PDUs or frames sent unreliably. They provide no feedback
provide no feedback channel at the subnetwork layer, instead relying channel at the subnetwork layer, instead relying on higher layers
on higher layers (e.g. TCP) to feed back loss signals. (e.g., TCP) to feed back loss signals.
In these cases, ECN may best be supported by standardising explicit In these cases, ECN may best be supported by standardising explicit
notification of congestion into the lower layer protocol that carries notification of congestion into the lower-layer protocol that carries
the data forwards. Then a specification is needed for how the egress the data forwards. Then, a specification is needed for how the
of the lower layer subnet propagates this explicit signal into the egress of the lower-layer subnet propagates this explicit signal into
forwarded upper layer (IP) header. This signal continues forwards the forwarded upper-layer (IP) header. This signal continues
until it finally reaches the destination transport (at L4). Then forwards until it finally reaches the destination transport (at L4).
typically the destination will feed this congestion notification back Typically, the destination will feed this congestion notification
to the source transport using an end-to-end protocol (e.g. TCP). back to the source transport using an end-to-end protocol (e.g.,
This is the arrangement that has already been used to add ECN to IP- TCP). This is the arrangement that has already been used to add ECN
in-IP tunnels [RFC6040], IP-in-MPLS and MPLS-in-MPLS [RFC5129]. to IP-in-IP tunnels [RFC6040], IP-in-MPLS, and MPLS-in-MPLS
[RFC5129].
This mode is illustrated in Figure 1. Along the middle of the This mode is illustrated in Figure 1. Along the middle of the
figure, layers 2, 3 and 4 of the protocol stack are shown, and one figure, layers 2, 3, and 4 of the protocol stack are shown. One
packet is shown along the bottom as it progresses across the network packet is shown along the bottom as it progresses across the network
from source to destination, crossing two subnets connected by a from source to destination, crossing two subnets connected by a
router, and crossing two switches on the path across each subnet. router and crossing two switches on the path across each subnet.
Congestion at the output of the first switch (shown as *) leads to a Congestion at the output of the first switch (shown as *) leads to a
congestion marking in the L2 header (shown as C in the illustration congestion marking in the L2 header (shown as C in the illustration
of the packet). The chevrons show the progress of the resulting of the packet). The chevrons show the progress of the resulting
congestion indication. It is propagated from link to link across the congestion indication. It is propagated from link to link across the
subnet in the L2 header, then when the router removes the marked L2 subnet in the L2 header. Then, when the router removes the marked L2
header, it propagates the marking up into the L3 (IP) header. The header, it propagates the marking up into the L3 (IP) header. The
router forwards the marked L3 header into subnet B. The L2 protocol router forwards the marked L3 header into subnet B. The L2 protocol
used in subnet B does not support ECN, but the signal proceeds across used in subnet B does not support ECN, but the signal proceeds across
it in the L3 header. it in the L3 header.
Note that there is no implication that each 'C' marking is encoded Note that there is no implication that each 'C' marking is encoded
the same; a different encoding might be used for the 'C' marking in the same; a different encoding might be used for the 'C' marking in
each protocol. each protocol.
Finally, for completeness, we show the L3 marking arriving at the Finally, for completeness, we show the L3 marking arriving at the
destination, where the host transport protocol (e.g. TCP) feeds it destination, where the host transport protocol (e.g., TCP) feeds it
back to the source in the L4 acknowledgement (the 'C' at L4 in the back to the source in the L4 acknowledgement (the 'C' at L4 in the
packet at the top of the diagram). packet at the top of the diagram).
_ _ _ _ _ _
/_______ | | |C| ACK Packet (V) /_______ | | |C| ACK Packet (V)
\ |_|_|_| \ |_|_|_|
+---+ layer: 2 3 4 header +---+ +---+ layer: 2 3 4 header +---+
| <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4
| | +---+ | ^ | | | +---+ | ^ |
| | . . . . . . Packet U. . | >>|>>> Packet U >>>>>>>>>>>>|>^ |L3 | | . . . . . . Packet U. . | >>|>>> Packet U >>>>>>>>>>>>|>^ |L3
| | +---+ +---+ | ^ | +---+ +---+ | | | | +---+ +---+ | ^ | +---+ +---+ | |
| | | *|>>>>>|>>>|>>>>>|>^ | | | | | | |L2 | | | *|>>>>>|>>>|>>>>>|>^ | | | | | | |L2
|___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___|
source subnet A router subnet B dest source subnet A router subnet B dest
__ _ _ _| __ _ _ _| __ _ _| __ _ _ _| __ _ _ _| __ _ _ _| __ _ _| __ _ _ _|
| | | | | | | | |C| | | |C| | | |C| | Data________\ | | | | | | | | |C| | | |C| | | |C| | Data________\
|__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| Packet (U) / |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| Packet (U) /
layer:4 3 2A 4 3 2A 4 3 4 3 2B layer:4 3 2A 4 3 2A 4 3 4 3 2B
header header
Figure 1: Feed-Forward-and-Up Mode Figure 1: Feed-Forward-and-Up Mode
Of course, modern networks are rarely as simple as this text-book Of course, modern networks are rarely as simple as this textbook
example, often involving multiple nested layers. For example, a 3GPP example, often involving multiple nested layers. For example, a
mobile network may have two IP-in-IP (GTP [GTPv1]) tunnels in series Third Generation Partnership Project (3GPP) mobile network may have
and an MPLS backhaul between the base station and the first router. two IP-in-IP (GTP) [GTPv1] tunnels in series and an MPLS backhaul
Nonetheless, the example illustrates the general idea of feeding between the base station and the first router. Nonetheless, the
congestion notification forward then upward whenever a header is example illustrates the general idea of feeding congestion
removed at the egress of a subnet. notifications forward then upward whenever a header is removed at the
egress of a subnet.
Note that the FECN (forward ECN ) bit in Frame Relay [Buck00] and the Note that the Forward Explicit Congestion Notification (FECN) bit in
explicit forward congestion indication (EFCI [ITU-T.I.371]) bit in Frame Relay [Buck00] and the Explicit Forward Congestion Indication
ATM user data cells follow a feed-forward pattern. However, in ATM, (EFCI) [ITU-T.I.371] bit in ATM user data cells follow a feed-forward
this arrangement is only part of a feed-forward-and-backward pattern pattern. However, in ATM, this arrangement is only part of a feed-
at the lower layer, not feed-forward-and-up out of the lower layer — forward-and-backward pattern at the lower layer, not feed-forward-
the intention was never to interface to IP ECN at the subnet egress. and-up out of the lower layer -- the intention was never to interface
To our knowledge, Frame Relay FECN is solely used to detect where to IP ECN at the subnet egress. To our knowledge, Frame Relay FECN
more capacity should be provisioned. is solely used to detect where more capacity should be provisioned.
3.2. Feed-Up-and-Forward Mode 3.2. Feed-Up-and-Forward Mode
Ethernet is particularly difficult to extend incrementally to support Ethernet is particularly difficult to extend incrementally to support
explicit congestion notification. One way to support ECN in such ECN. One way to support ECN in such cases is to use so-called 'Layer
cases has been to use so called 'layer-3 switches'. These are 3 switches'. These are Ethernet switches that dig into the Ethernet
Ethernet switches that dig into the Ethernet payload to find an IP payload to find an IP header and manipulate or act on certain IP
header and manipulate or act on certain IP fields (specifically fields (specifically Diffserv and ECN). For instance, in Data Center
Diffserv & ECN). For instance, in Data Center TCP [RFC8257], layer-3 TCP [RFC8257], Layer 3 switches are configured to mark the ECN field
switches are configured to mark the ECN field of the IP header within of the IP header within the Ethernet payload when their output buffer
the Ethernet payload when their output buffer becomes congested. becomes congested. With respect to switching, a Layer 3 switch acts
With respect to switching, a layer-3 switch acts solely on the solely on the addresses in the Ethernet header; it does not use IP
addresses in the Ethernet header; it does not use IP addresses, and addresses and it does not decrement the TTL field in the IP header.
it does not decrement the TTL field in the IP header.
_ _ _ _ _ _
/_______ | | |C| ACK packet (V) /_______ | | |C| ACK packet (V)
\ |_|_|_| \ |_|_|_|
+---+ layer: 2 3 4 header +---+ +---+ layer: 2 3 4 header +---+
| <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4
| | +---+ | ^ | | | +---+ | ^ |
| | . . . >>>> Packet U >>>|>>>|>>> Packet U >>>>>>>>>>>>|>^ |L3 | | . . . >>>> Packet U >>>|>>>|>>> Packet U >>>>>>>>>>>>|>^ |L3
| | +--^+ +---+ | v| +---+ +---+ | ^ | | | +--^+ +---+ | v| +---+ +---+ | ^ |
| | | *| | | | >|>>>>>|>>>|>>>>>|>>>|>>>>>|>^ |L2 | | | *| | | | >|>>>>>|>>>|>>>>>|>>>|>>>>>|>^ |L2
|___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___|
source subnet E router subnet F dest source subnet E router subnet F dest
__ _ _ _| __ _ _ _| __ _ _| __ _ _ _| __ _ _ _| __ _ _ _| __ _ _| __ _ _ _|
| | | | | | | |C| | | | |C| | | |C|C| Data________\ | | | | | | | |C| | | | |C| | | |C|C| Data________\
|__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| Packet (U) / |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| Packet (U) /
layer:4 3 2 4 3 2 4 3 4 3 2 layer:4 3 2 4 3 2 4 3 4 3 2
header header
Figure 2: Feed-Up-and-Forward Mode Figure 2: Feed-Up-and-Forward Mode
By comparing Figure 2 with Figure 1, it can be seen that subnet E By comparing Figure 2 with Figure 1, it can be seen that subnet E
(perhaps a subnet of layer-3 Ethernet switches) works in feed-up-and- (perhaps a subnet of Layer 3 Ethernet switches) works in feed-up-and-
forward mode by notifying congestion directly into L3 at the point of forward mode by notifying congestion directly into L3 at the point of
congestion, even though the congested switch does not otherwise act congestion, even though the congested switch does not otherwise act
at L3. In this example, the technology in subnet F (e.g. MPLS) does at L3. In this example, the technology in subnet F (e.g., MPLS) does
support ECN natively, so when the router adds the layer-2 header it support ECN natively, so when the router adds the Layer 2 header it
copies the ECN marking from L3 to L2 as well, as shown by the 'C's in copies the ECN marking from L3 to L2 as well, as shown by the 'C's in
both layers. both layers.
3.3. Feed-Backward Mode 3.3. Feed-Backward Mode
In some layer 2 technologies, explicit congestion notification has In some Layer 2 technologies, ECN has been defined for use internally
been defined for use internally within the subnet with its own within the subnet with its own feedback and load regulation;
feedback and load regulation, but typically the interface with IP for typically, the interface with IP for ECN has not been defined.
ECN has not been defined.
For instance, for the available bit-rate (ABR) service in ATM, the For instance, for the Available Bit Rate (ABR) service in ATM, the
relative rate mechanism was one of the more popular mechanisms for relative rate mechanism was one of the more popular mechanisms for
managing traffic, tending to supersede earlier designs. In this managing traffic, tending to supersede earlier designs. In this
approach ATM switches send special resource management (RM) cells in approach, ATM switches send special resource management (RM) cells in
both the forward and backward directions to control the ingress rate both the forward and backward directions to control the ingress rate
of user data into a virtual circuit. If a switch buffer is of user data into a virtual circuit. If a switch buffer is
approaching congestion or is congested it sends an RM cell back approaching congestion or is congested, it sends an RM cell back
towards the ingress with respectively the No Increase (NI) or towards the ingress with the No Increase (NI) or Congestion
Congestion Indication (CI) bit set in its message type field Indication (CI) bit set in its message type field [ATM-TM-ABR],
[ATM-TM-ABR]. The ingress then holds or decreases its sending bit- respectively. The ingress then holds or decreases its sending bit-
rate accordingly. rate accordingly.
_ _ _ _ _ _
/_______ | | |C| ACK packet (X) /_______ | | |C| ACK packet (X)
\ |_|_|_| \ |_|_|_|
+---+ layer: 2 3 4 header +---+ +---+ layer: 2 3 4 header +---+
| <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet X <<<<<<<<<<<<<|<< |L4 | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet X <<<<<<<<<<<<<|<< |L4
| | +---+ | ^ | | | +---+ | ^ |
| | | *|>>> Packet W >>>>>>>>>>>>|>^ |L3 | | | *|>>> Packet W >>>>>>>>>>>>|>^ |L3
| | +---+ +---+ | | +---+ +---+ | | | | +---+ +---+ | | +---+ +---+ | |
skipping to change at page 13, line 46 skipping to change at line 569
2 2
__ _ _ _ __ _ _ _ __ _ _ __ _ _ _ earlier __ _ _ _ __ _ _ _ __ _ _ __ _ _ _ earlier
| | | | | | | | | | | | | | | | | | | data________\ | | | | | | | | | | | | | | | | | | | data________\
|__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) / |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) /
layer: 4 3 2 4 3 2 4 3 4 3 2 layer: 4 3 2 4 3 2 4 3 4 3 2
header header
Figure 3: Feed-Backward Mode Figure 3: Feed-Backward Mode
ATM's feed-backward approach does not fit well when layered beneath ATM's feed-backward approach does not fit well when layered beneath
IP's feed-forward approach — unless the initial data source is the IP's feed-forward approach unless the initial data source is the same
same node as the ATM ingress. Figure 3 shows the feed-backward node as the ATM ingress. Figure 3 shows the feed-backward approach
approach being used in subnet H. If the final switch on the path is being used in subnet H. If the final switch on the path is congested
congested (*), it does not feed-forward any congestion indications on (*), it does not feed forward any congestion indications on the
packet (U). Instead it sends a control cell (V) back to the router packet (U). Instead, it sends a control cell (V) back to the router
at the ATM ingress. at the ATM ingress.
However, the backward feedback does not reach the original data However, the backward feedback does not reach the original data
source directly because IP does not support backward feedback (and source directly because IP does not support backward feedback (and
subnet G is independent of subnet H). Instead, the router in the subnet G is independent of subnet H). Instead, the router in the
middle throttles down its sending rate but the original data sources middle throttles down its sending rate, but the original data sources
don't reduce their rates. The resulting rate mismatch causes the don't reduce their rates. The resulting rate mismatch causes the
middle router's buffer at layer 3 to back up until it becomes middle router's buffer at layer 3 to back up until it becomes
congested, which it signals forwards on later data packets at layer 3 congested, which it signals forwards on later data packets at layer 3
(e.g. packet W). Note that the forward signal from the middle router (e.g., packet W). Note that the forward signal from the middle
is not triggered directly by the backward signal. Rather, it is router is not triggered directly by the backward signal. Rather, it
triggered by congestion resulting from the middle router's mismatched is triggered by congestion resulting from the middle router's
rate response to the backward signal. mismatched rate response to the backward signal.
In response to this later forward signalling, end-to-end feedback at In response to this later forward signalling, end-to-end feedback at
layer-4 finally completes the tortuous path of congestion indications layer 4 finally completes the tortuous path of congestion indications
back to the origin data source, as before. back to the origin data source as before.
Quantized congestion notification (QCN [IEEE802.1Q]) would suffer Quantized Congestion Notification (QCN) [IEEE802.1Q] would suffer
from similar problems if extended to multiple subnets. However, from from similar problems if extended to multiple subnets. However, QCN
the start QCN was clearly characterized as solely applicable to a was clearly characterized as solely applicable to a single subnet
single subnet (see Section 6). from the start (see Section 6).
3.4. Null Mode 3.4. Null Mode
Often link and physical layer resources are 'non-blocking' by design. Link and physical-layer resources are often 'non-blocking' by design.
In these cases congestion notification may be implemented but it does Congestion notifications may be implemented in these cases, but it
not need to be deployed at the lower layer; ECN in IP would be does not need to be deployed at the lower layer; ECN in IP would be
sufficient. sufficient.
A degenerate example is a point-to-point Ethernet link. Excess A degenerate example is a point-to-point Ethernet link. Excess
loading of the link merely causes the queue from the higher layer to loading of the link merely causes the queue from the higher layer to
back up, while the lower layer remains immune to congestion. Even a back up, while the lower layer remains immune to congestion. Even a
whole meshed subnetwork can be made immune to interior congestion by whole meshed subnetwork can be made immune to interior congestion by
limiting ingress capacity and sufficient sizing of interior links, limiting ingress capacity and sufficient sizing of interior links,
e.g. a non-blocking fat-tree network [Leiserson85]. An alternative e.g., a non-blocking fat-tree network [Leiserson85]. An alternative
to fat links near the root is numerous thin links with multi-path to fat links near the root is numerous thin links with multi-path
routing to ensure even worst-case patterns of load cannot congest any routing to ensure even worst-case patterns of load cannot congest any
link, e.g. a Clos network [Clos53]. link, e.g., a Clos network [Clos53].
4. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion 4. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion
Notification Notification
Feed-forward-and-up is the mode already used for signalling ECN up Feed-forward-and-up is the mode already used for signalling ECN up
the layers through MPLS into IP [RFC5129] and through IP-in-IP the layers through MPLS into IP [RFC5129] and through IP-in-IP
tunnels [RFC6040], whether encapsulating with IPv4 [RFC2003], IPv6 tunnels [RFC6040], whether encapsulating with IPv4 [RFC2003], IPv6
[RFC2473] or IPsec [RFC4301]. These RFCs take a consistent approach [RFC2473], or IPsec [RFC4301]. These RFCs take a consistent approach
and the following guidelines are designed to ensure this consistency and the following guidelines are designed to ensure this consistency
continues as ECN support is added to other protocols that encapsulate continues as ECN support is added to other protocols that encapsulate
IP. The guidelines are also designed to ensure compliance with the IP. The guidelines are also designed to ensure compliance with the
more general best current practice for the design of alternate ECN more general best current practice for the design of alternate ECN
schemes given in [RFC4774] and extended by [RFC8311]. schemes given in [RFC4774] and extended by [RFC8311].
The rest of this section is structured as follows: The rest of this section is structured as follows:
* Section 4.1 addresses the most straightforward cases, where * Section 4.1 addresses the most straightforward cases, where
[RFC6040] can be applied directly to add ECN to tunnels that are [RFC6040] can be applied directly to add ECN to tunnels that are
effectively IP-in-IP tunnels, but with shim header(s) between the effectively IP-in-IP tunnels, but with a shim header(s) between
IP headers. the IP headers.
* The subsequent sections give guidelines for adding ECN to a subnet * The subsequent sections give guidelines for adding ECN to a subnet
technology that uses feed-forward-and-up mode like IP, but it is technology that uses feed-forward-and-up mode like IP, but it is
not so similar to IP that [RFC6040] rules can be applied directly. not so similar to IP that [RFC6040] rules can be applied directly.
Specifically: Specifically:
- Sections 4.2, 4.3 and 4.4 respectively address how to add ECN - Sections 4.2, 4.3, and 4.4 address how to add ECN support to
support to the wire protocol and to the encapsulators and the wire protocol and to the encapsulators and decapsulators at
decapsulators at the ingress and egress of the subnet. the ingress and egress of the subnet, respectively.
- Section 4.5 deals with the special, but common, case of - Section 4.5 deals with the special but common case of sequences
sequences of tunnels or subnets that all use the same of tunnels or subnets that all use the same technology.
technology
- Section 4.6 deals with the question of reframing when IP - Section 4.6 deals with the question of reframing when IP
packets do not map 1:1 into lower layer frames. packets do not map 1:1 into lower-layer frames.
4.1. IP-in-IP Tunnels with Shim Headers 4.1. IP-in-IP Tunnels with Shim Headers
A common pattern for many tunnelling protocols is to encapsulate an A common pattern for many tunnelling protocols is to encapsulate an
inner IP header with shim header(s) then an outer IP header. A shim inner IP header with a shim header(s) then an outer IP header. A
header is defined as one that is not sufficient alone to forward the shim header is defined as one that is not sufficient alone to forward
packet as an outer header. Another common pattern is for a shim to the packet as an outer header. Another common pattern is for a shim
encapsulate a layer 2 (L2) header, which in turn encapsulates (or to encapsulate an L2 header, which in turn encapsulates (or might
might encapsulate) an IP header. [I-D.ietf-tsvwg-rfc6040update-shim] encapsulate) an IP header. [RFC9601] clarifies that [RFC6040] is
clarifies that RFC 6040 is just as applicable when there are shim(s) just as applicable when there is a shim(s) and possibly an L2 header
and possibly a L2 header between two IP headers. between two IP headers.
However, it is not always feasible or necessary to propagate ECN However, it is not always feasible or necessary to propagate ECN
between IP headers when separated by a shim. For instance, it might between IP headers when separated by a shim. For instance, it might
be too costly to dig to arbitrary depths to find an inner IP header, be too costly to dig to arbitrary depths to find an inner IP header,
there may be little or no congestion within the tunnel by design (see there may be little or no congestion within the tunnel by design (see
null mode in Section 3.4 above), or a legacy implementation might not null mode in Section 3.4 above), or a legacy implementation might not
support ECN. In cases where a tunnel does not support ECN, it is support ECN. In cases where a tunnel does not support ECN, it is
important that the ingress does not copy the ECN field from an inner important that the ingress does not copy the ECN field from an inner
IP header to an outer. Therefore Section 4 of IP header to an outer. Therefore Section 4 of [RFC9601] requires
[I-D.ietf-tsvwg-rfc6040update-shim] requires network operators to network operators to configure the ingress of a tunnel that does not
configure the ingress of a tunnel that does not support ECN so that support ECN so that it zeros the ECN field in the outer IP header.
it zeros the ECN field in the outer IP header.
Nonetheless, in many cases it is feasible to propagate the ECN field Nonetheless, in many cases it is feasible to propagate the ECN field
between IP headers separated by shim header(s) and/or a L2 header. between IP headers separated by a shim header(s) and/or an L2 header.
Particularly in the typical case when the outer IP header and the Particularly in the typical case when the outer IP header and the
shim(s) are added (or removed) as part of the same procedure. Even shim(s) is added (or removed) as part of the same procedure. Even if
if the shim(s) encapsulate a L2 header, it is often possible to find the shim(s) encapsulates an L2 header, it is often possible to find
an inner IP header within the L2 PDU and propagate ECN between that an inner IP header within the L2 PDU and propagate ECN between that
and the outer IP header. This can be thought of as a special case of and the outer IP header. This can be thought of as a special case of
the feed-up-and-forward mode (Section 3.2), so the guidelines for the feed-up-and-forward mode (Section 3.2), so the guidelines for
this mode apply (Section 5). this mode apply (Section 5).
Numerous shim protocols have been defined for IP tunnelling. More Numerous shim protocols have been defined for IP tunnelling. More
recent ones e.g. Geneve [RFC8926] and Generic UDP Encapsulation recent ones, e.g., Geneve [RFC8926] and Generic UDP Encapsulation
(GUE) [I-D.ietf-intarea-gue] cite and follow RFC 6040. And some (GUE) [INTAREA-GUE] cite and follow [RFC6040]. Some earlier ones,
earlier ones, e.g. CAPWAP [RFC5415] and LISP [RFC9300], cite RFC e.g., CAPWAP [RFC5415] and LISP [RFC9300], cite [RFC3168], which is
3168, which is compatible with RFC 6040. compatible with [RFC6040].
However, as Section 9.3 of [RFC3168] pointed out, ECN support needs However, as Section 9.3 of [RFC3168] pointed out, ECN support needs
to be defined for many earlier shim-based tunnelling protocols, e.g. to be defined for many earlier shim-based tunnelling protocols, e.g.,
L2TPv2 [RFC2661], L2TPv3 [RFC3931], GRE [RFC2784], PPTP [RFC2637], L2TPv2 [RFC2661], L2TPv3 [RFC3931], GRE [RFC2784], PPTP [RFC2637],
GTP [GTPv1], [GTPv1-U], [GTPv2-C] and Teredo [RFC4380] as well as GTP [GTPv1] [GTPv1-U], [GTPv2-C], and Teredo [RFC4380], as well as
some recent ones, e.g. VXLAN [RFC7348], NVGRE [RFC7637] and NSH some recent ones, e.g., VXLAN [RFC7348], NVGRE [RFC7637], and NSH
[RFC8300]. [RFC8300].
All these IP-based encapsulations can be updated in one shot by All these IP-based encapsulations can be updated in one shot by
simple reference to RFC 6040. However, it would not be appropriate simple reference to [RFC6040]. However, it would not be appropriate
to update all these protocols from within the present guidance to update all these protocols from within the present guidance
document. Instead a companion specification document. Instead, a companion specification [RFC9601] has the
[I-D.ietf-tsvwg-rfc6040update-shim] has been prepared that has the appropriate Standards Track status to update Standards Track
appropriate standards track status to update standards track
protocols. For those that are not under IETF change control protocols. For those that are not under IETF change control
[I-D.ietf-tsvwg-rfc6040update-shim] can only recommend that the [RFC9601] can only recommend that the relevant body updates them.
relevant body updates them.
4.2. Wire Protocol Design: Indication of ECN Support 4.2. Wire Protocol Design: Indication of ECN Support
This section is intended to guide the redesign of any lower layer This section is intended to guide the redesign of any lower-layer
protocol that encapsulate IP to add native ECN support at the lower protocol that encapsulates IP to add native ECN support at the lower
layer. It reflects the approaches used in [RFC6040] and in layer. It reflects the approaches used in [RFC6040] and in
[RFC5129]. Therefore IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS [RFC5129]. Therefore, IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS
encapsulations that already comply with [RFC6040] or [RFC5129] will encapsulations that already comply with [RFC6040] or [RFC5129] will
already satisfy this guidance. already satisfy this guidance.
A lower layer (or subnet) congestion notification system: A lower-layer (or subnet) congestion notification system:
1. SHOULD NOT apply explicit congestion notifications to PDUs that 1. SHOULD NOT apply ECNs to PDUs that are destined for legacy
are destined for legacy layer-4 transport implementations that layer-4 transport implementations that will not understand ECN;
will not understand ECN, and and
2. SHOULD NOT apply explicit congestion notifications to PDUs if the 2. SHOULD NOT apply ECNs to PDUs if the egress of the subnet might
egress of the subnet might not propagate congestion notifications not propagate congestion notifications onward into the higher
onward into the higher layer. layer.
We use the term ECN-PDUs for a PDU on a feedback loop that will We use the term ECN-PDUs for a PDU on a feedback loop that will
propagate congestion notification properly because it meets both propagate congestion notifications properly because it meets both
the above criteria. And a Not-ECN-PDU is a PDU on a feedback the above criteria. Additionally, a Not-ECN-PDU is a PDU on a
loop that does not meet at least one of the criteria, and will feedback loop that does not meet at least one of the criteria,
therefore not propagate congestion notification properly. A and therefore will not propagate congestion notifications
corollary of the above is that a lower layer congestion properly. A corollary of the above is that a lower-layer
notification protocol: congestion notification protocol:
3. SHOULD be able to distinguish ECN-PDUs from Not-ECN-PDUs. 3. SHOULD be able to distinguish ECN-PDUs from Not-ECN-PDUs.
Note that there is no need for all interior nodes within a subnet to Note that there is no need for all interior nodes within a subnet to
be able to mark congestion explicitly. A mix of ECN and drop signals be able to mark congestion explicitly. A mix of ECN and drop signals
from different nodes is fine. However, if _any_ interior nodes might from different nodes is fine. However, if _any_ interior nodes might
generate ECN markings, guideline 2 above says that all relevant generate ECN markings, Guideline 2 above says that all relevant
egress node(s) SHOULD be able to propagate those markings up to the egress nodes SHOULD be able to propagate those markings up to the
higher layer. higher layer.
In IP, if the ECN field in each PDU is cleared to the Not-ECT (not In IP, if the ECN field in each PDU is cleared to the Not ECN-Capable
ECN-capable transport) codepoint, it indicates that the L4 transport Transport (Not-ECT) codepoint, it indicates that the L4 transport
will not understand congestion markings. A congested buffer must not will not understand congestion markings. A congested buffer must not
mark these Not-ECT PDUs, and therefore has to signal congestion by mark these Not-ECT PDUs; therefore, it has to signal congestion by
increasingly applying drop instead. increasingly applying drop instead.
The mechanism a lower layer uses to distinguish the ECN-capability of The mechanism a lower layer uses to distinguish the ECN capability of
PDUs need not mimic that of IP. The above guidelines merely say that PDUs need not mimic that of IP. The above guidelines merely say that
the lower layer system, as a whole, should achieve the same outcome. the lower-layer system as a whole should achieve the same outcome.
For instance, ECN-capable feedback loops might use PDUs that are For instance, ECN-capable feedback loops might use PDUs that are
identified by a particular set of labels or tags. Alternatively, identified by a particular set of labels or tags. Alternatively,
logical link protocols that use flow state might determine whether a logical-link protocols that use flow state might determine whether a
PDU can be congestion marked by checking for ECN-support in the flow PDU can be congestion marked by checking for ECN-support in the flow
state. Other protocols might depend on out-of-band control signals. state. Other protocols might depend on out-of-band control signals.
The per-domain checking of ECN support in MPLS [RFC5129] is a good The per-domain checking of ECN support in MPLS [RFC5129] is a good
example of a way to avoid sending congestion markings to L4 example of a way to avoid sending congestion markings to L4
transports that will not understand them, without using any header transports that will not understand them without using any header
space in the subnet protocol. space in the subnet protocol.
In MPLS, header space is extremely limited, therefore RFC5129 does In MPLS, header space is extremely limited; therefore, [RFC5129] does
not provide a field in the MPLS header to indicate whether the PDU is not provide a field in the MPLS header to indicate whether the PDU is
an ECN-PDU or a Not-ECN-PDU. Instead, interior nodes in a domain are an ECN-PDU or a Not-ECN-PDU. Instead, interior nodes in a domain are
allowed to set explicit congestion indications without checking allowed to set explicit congestion indications without checking
whether the PDU is destined for a L4 transport that will understand whether the PDU is destined for a L4 transport that will understand
them. Nonetheless, this is made safe by requiring that the network them. Nonetheless, this is made safe by requiring that the network
operator upgrades all decapsulating edges of a whole domain at once, operator upgrades all decapsulating edges of a whole domain at once
as soon as even one switch within the domain is configured to mark as soon as even one switch within the domain is configured to mark
rather than drop some PDUs during congestion. Therefore, any edge rather than drop some PDUs during congestion. Therefore, any edge
node that might decapsulate a packet will be capable of checking node that might decapsulate a packet will be capable of checking
whether the higher layer transport is ECN-capable. When whether the higher-layer transport is ECN-capable. When
decapsulating a CE-marked packet, if the decapsulator discovers that decapsulating a CE-marked packet, if the decapsulator discovers that
the higher layer (inner header) indicates the transport is not ECN- the higher layer (inner header) indicates the transport is not ECN-
capable, it drops the packet — effectively on behalf of the earlier capable, it drops the packet -- effectively on behalf of the earlier
congested node (see Decapsulation Guideline 1 in Section 4.4). congested node (see Decapsulation Guideline 1 in Section 4.4).
It was only appropriate to define such an incremental deployment It was only appropriate to define such an incremental deployment
strategy because MPLS is targeted solely at professional operators, strategy because MPLS is targeted solely at professional operators
who can be expected to ensure that a whole subnetwork is consistently who can be expected to ensure that a whole subnetwork is consistently
configured. This strategy might not be appropriate for other link configured. This strategy might not be appropriate for other link
technologies targeted at zero-configuration deployment or deployment technologies targeted at zero-configuration deployment or deployment
by the general public (e.g. Ethernet). For such 'plug-and-play' by the general public (e.g., Ethernet). For such 'plug-and-play'
environments it will be necessary to invent a failsafe approach that environments, it will be necessary to invent a fail-safe approach
ensures congestion markings will never fall into black holes, no that ensures congestion markings will never fall into black holes, no
matter how inconsistently a system is put together. Alternatively, matter how inconsistently a system is put together. Alternatively,
congestion notification relying on correct system configuration could congestion notifications relying on correct system configuration
be confined to flavours of Ethernet intended only for professional could be confined to flavours of Ethernet intended only for
network operators, such as Provider Backbone Bridges (PBB professional network operators, such as Provider Backbone Bridges
[IEEE802.1Q]; previously 802.1ah). (PBB) ([IEEE802.1Q]; previously 802.1ah).
ECN support in TRILL [I-D.ietf-trill-ecn-support] provides a good ECN support in TRansparent Interconnection of Lots of Links (TRILL)
example of how to add ECN to a lower layer protocol without relying [RFC9600] provides a good example of how to add ECN to a lower-layer
on careful and consistent operator configuration. TRILL provides an protocol without relying on careful and consistent operator
extension header word with space for flags of different categories configuration. TRILL provides an extension header word with space
depending on whether logic to understand the extension is critical. for flags of different categories depending on whether logic to
The congestion experienced marking has been defined as a 'critical understand the extension is critical. The congestion-experienced
ingress-to-egress' flag. So if a transit RBridge sets this flag on a marking has been defined as a 'critical ingress-to-egress' flag. So
frame and an egress RBridge does not have any logic to process it, it if a transit RBridge sets this flag on a frame and an egress RBridge
will drop it; which is the desired default action anyway. Therefore does not have any logic to process it, it will drop it; which is the
TRILL RBridges can be updated with support for ECN in no particular desired default action anyway. Therefore, TRILL RBridges can be
order and, at the egress of the TRILL campus, congestion notification updated with support for ECN in no particular order and, at the
will be propagated to IP as ECN whenever ECN logic has been egress of the TRILL campus, congestion notifications will be
implemented at the egress, or as drop otherwise. propagated to IP as ECN whenever ECN logic has been implemented at
the egress, or as drop otherwise.
QCN [IEEE802.1Q] is not intended to extend beyond a single subnet, or QCN [IEEE802.1Q] is not intended to extend beyond a single subnet or
to interoperate with ECN. Nonetheless, the way QCN indicates to interoperate with ECN. Nonetheless, the way QCN indicates to lower-
lower layer devices that the end-points will not understand QCN layer devices that the endpoints will not understand QCN provides
provides another example that a lower layer protocol designer might another example that a lower-layer protocol designer might be able to
be able to mimic for their scenario. An operator can define certain mimic for their scenario. An operator can define certain Priority
Priority Code Points (PCPs [IEEE802.1Q]; previously 802.1p) to Code Points (PCPs) ([IEEE802.1Q]; previously 802.1p) to indicate non-
indicate non-QCN frames and an ingress bridge is required to map QCN frames and an ingress bridge is required to map arriving not-QCN-
arriving not-QCN-capable IP packets to one of these non-QCN PCPs. capable IP packets to one of these non-QCN PCPs.
When drop for non-ECN traffic is deferred to the egress of a subnet, When drop for non-ECN traffic is deferred to the egress of a subnet,
it cannot necessarily be assumed that one ECN mark is equivalent to it cannot necessarily be assumed that one ECN mark is equivalent to
one drop, as was originally required by [RFC3168]. [RFC8311] updated one drop, as was originally required by [RFC3168]. [RFC8311] updated
RFC 3168, to allow experimentation with congestion markings that are [RFC3168] to allow experimentation with congestion markings that are
not equivalent to drop, in particular for L4S [RFC9331]. ECN support not equivalent to drop, particularly for L4S [RFC9331]. ECN support
in TRILL [I-D.ietf-trill-ecn-support] is a good example of a way to in TRILL [RFC9600] is a good example of a way to defer drop to the
defer drop to the egress of a subnet both when marks are equivalent egress of a subnet both when marks are equivalent to drops (as in
to drops (as in RFC 3168) and when they are not (as in L4S). The ECN [RFC3168]) and when they are not (as in L4S). The ECN scheme for
scheme for MPLS [RFC5129] was defined before L4S, so it only MPLS [RFC5129] was defined before L4S, so it only currently supports
currently supports deferred drop that is equivalent to ECN-marking. deferred drop that is equivalent to ECN marking. Nonetheless, in
Nonetheless, in principle, MPLS (and potentially future L2 protocols) principle, MPLS (and potentially future L2 protocols) could support
could support L4S marking and copy TRILL's approach for determining L4S marking and copy TRILL's approach for determining the drop level
the drop level of any non-ECN traffic at the subnet egress. of any non-ECN traffic at the subnet egress.
4.3. Encapsulation Guidelines 4.3. Encapsulation Guidelines
This section is intended to guide the redesign of any node that This section is intended to guide the redesign of any node that
encapsulates IP with a lower layer header when adding native ECN encapsulates IP with a lower-layer header when adding native ECN
support to the lower layer protocol. It reflects the approaches used support to the lower-layer protocol. It reflects the approaches used
in [RFC6040] and in [RFC5129]. Therefore IP-in-IP tunnels or IP-in- in [RFC6040] and [RFC5129]. Therefore, IP-in-IP tunnels or IP-in-
MPLS or MPLS-in-MPLS encapsulations that already comply with MPLS or MPLS-in-MPLS encapsulations that already comply with
[RFC6040] or [RFC5129] will already satisfy this guidance. [RFC6040] or [RFC5129] will already satisfy this guidance.
1. Egress Capability Check: A subnet ingress needs to be sure that 1. Egress Capability Check: A subnet ingress needs to be sure that
the corresponding egress of a subnet will propagate any the corresponding egress of a subnet will propagate any
congestion notification added to the outer header across the congestion notifications added to the outer header across the
subnet. This is necessary in addition to checking that an subnet. This is necessary in addition to checking that an
incoming PDU indicates an ECN-capable (L4) transport. Examples incoming PDU indicates an ECN-capable (L4) transport. Examples
of how this guarantee might be provided include: of how this guarantee might be provided include:
* by configuration (e.g. if any label switches in a domain * by configuration (e.g., if any label switches in a domain
support ECN marking, [RFC5129] requires all egress nodes to supports ECN marking, [RFC5129] requires all egress nodes to
have been configured to propagate ECN) have been configured to propagate ECN).
* by the ingress explicitly checking that the egress propagates * by the ingress explicitly checking that the egress propagates
ECN (e.g. an early attempt to add ECN support to TRILL used ECN (e.g., an early attempt to add ECN support to TRILL used
IS-IS to check path capabilities before adding ECN extension IS-IS to check path capabilities before adding ECN extension
flags to each frame [RFC7780]). flags to each frame [RFC7780]).
* by inherent design of the protocol (e.g. by encoding ECN * by inherent design of the protocol (e.g., by encoding ECN
marking on the outer header in such a way that a legacy egress marking on the outer header in such a way that a legacy egress
that does not understand ECN will consider the PDU corrupt or that does not understand ECN will consider the PDU corrupt or
invalid and discard it, thus at least propagating a form of invalid and discard it; thus, at least propagating a form of
congestion signal). congestion signal).
2. Egress Fails Capability Check: If the ingress cannot guarantee 2. Egress Fails Capability Check: If the ingress cannot guarantee
that the egress will propagate congestion notification, the that the egress will propagate congestion notifications, the
ingress SHOULD disable ECN at the lower layer when it forwards ingress SHOULD disable ECN at the lower layer when it forwards
the PDU. An example of how the ingress might disable ECN at the the PDU. An example of how the ingress might disable ECN at the
lower layer would be by setting the outer header of the PDU to lower layer would be by setting the outer header of the PDU to
identify it as a Not-ECN-PDU, assuming the subnet technology identify it as a Not-ECN-PDU, assuming the subnet technology
supports such a concept. supports such a concept.
3. Standard Congestion Monitoring Baseline: Once the ingress to a 3. Standard Congestion Monitoring Baseline: Once the ingress to a
subnet has established that the egress will correctly propagate subnet has established that the egress will correctly propagate
ECN, on encapsulation it SHOULD encode the same level of ECN, it SHOULD encode the same level of congestion in outer
congestion in outer headers as is arriving in incoming headers. headers as is arriving in incoming headers on encapsulation. For
For example, it might copy any incoming congestion notification example, it might copy any incoming congestion notifications into
into the outer header of the lower layer protocol. the outer header of the lower-layer protocol.
This ensures that bulk congestion monitoring of outer headers This ensures that bulk congestion monitoring of outer headers
(e.g. by a network management node monitoring ECN in passing (e.g., by a network management node monitoring ECN in passing
frames) will measure congestion accumulated along the whole frames) will measure congestion accumulated along the whole
upstream path — starting from the Load Regulator not just upstream path, starting from the Load Regulator and not just
starting from the ingress of the subnet. A node that is not the starting from the ingress of the subnet. A node that is not the
Load Regulator SHOULD NOT re-initialize the level of CE markings Load Regulator SHOULD NOT re-initialize the level of CE markings
in the outer to zero. in the outer header to zero.
It would still also be possible to measure congestion introduced It would still also be possible to measure congestion introduced
across one subnet (or tunnel) by subtracting the level of CE across one subnet (or tunnel) by subtracting the level of CE
markings on inner headers from that on outer headers (see markings on inner headers from that on outer headers (see
Appendix C of [RFC6040]). For example: Appendix C of [RFC6040]). For example:
* If this guideline has been followed and if the level of CE * If this guideline has been followed and if the level of CE
markings is 0.4% on the outer and 0.1% on the inner, 0.4% markings is 0.4% on the outer header and 0.1% on the inner
congestion has been introduced across all the networks since header, 0.4% congestion has been introduced across all the
the load regulator, and 0.3% (= 0.4% - 0.1%) has been networks since the Load Regulator, and 0.3% (= 0.4% - 0.1%)
introduced since the ingress to the current subnet (or has been introduced since the ingress to the current subnet
tunnel); (or tunnel).
* Without this guideline, if the subnet ingress had re- * Without this guideline, if the subnet ingress had re-
initialized the outer congestion level to zero, the outer and initialized the outer congestion level to zero, the outer and
inner would measure 0.1% and 0.3%. It would still be possible inner headers would measure 0.1% and 0.3%. It would still be
to infer that the congestion introduced since the Load possible to infer that the congestion introduced since the
Regulator was 0.4% (= 0.1% + 0.3%). But only if the Load Regulator was 0.4% (= 0.1% + 0.3%), but only if the
monitoring system somehow knows whether the subnet ingress re- monitoring system somehow knows whether the subnet ingress re-
initialized the congestion level. initialized the congestion level.
As long as subnet and tunnel technologies use the standard As long as subnet and tunnel technologies use the standard
congestion monitoring baseline in this guideline, monitoring congestion monitoring baseline in this guideline, monitoring
systems will know to use the former approach, rather than having systems will know to use the former approach rather than having
to "somehow know" which approach to use. to "somehow know" which approach to use.
4.4. Decapsulation Guidelines 4.4. Decapsulation Guidelines
This section is intended to guide the redesign of any node that This section is intended to guide the redesign of any node that
decapsulates IP from within a lower layer header when adding native decapsulates IP from within a lower-layer header when adding native
ECN support to the lower layer protocol. It reflects the approaches ECN support to the lower-layer protocol. It reflects the approaches
used in [RFC6040] and in [RFC5129]. Therefore IP-in-IP tunnels or used in [RFC6040] and in [RFC5129]. Therefore, IP-in-IP tunnels or
IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with
[RFC6040] or [RFC5129] will already satisfy this guidance. [RFC6040] or [RFC5129] will already satisfy this guidance.
A subnet egress SHOULD NOT simply copy congestion notification from A subnet egress SHOULD NOT simply copy congestion notifications from
outer headers to the forwarded header. It SHOULD calculate the outer headers to the forwarded header. It SHOULD calculate the
outgoing congestion notification field from the inner and outer outgoing congestion notification field from the inner and outer
headers using the following guidelines. If there is any conflict, headers using the following guidelines. If there is any conflict,
rules earlier in the list take precedence over rules later in the rules earlier in the list take precedence over rules later in the
list: list.
1. If the arriving inner header is a Not-ECN-PDU it implies the L4 1. If the arriving inner header is a Not-ECN-PDU, it implies the L4
transport will not understand explicit congestion markings. transport will not understand explicit congestion markings.
Then: Then:
* If the outer header carries an explicit congestion marking, it * If the outer header carries an explicit congestion marking, it
is likely that a protocol error has occurred, so drop is the is likely that a protocol error has occurred, so drop is the
only indication of congestion that the L4 transport will only indication of congestion that the L4 transport will
understand. If the congestion marking is the most severe understand. If the congestion marking is the most severe
possible, the packet MUST be dropped. However, if congestion possible, the packet MUST be dropped. However, if congestion
can be marked with multiple levels of severity and the can be marked with multiple levels of severity and the
packet's marking is not the most severe, this requirement can packet's marking is not the most severe, this requirement can
be relaxed to: the packet SHOULD be dropped. be relaxed to: the packet SHOULD be dropped.
* If the outer is an ECN-PDU that carries no indication of * If the outer is an ECN-PDU that carries no indication of
congestion or a Not-ECN-PDU the PDU SHOULD be forwarded, but congestion or a Not-ECN-PDU the PDU SHOULD be forwarded, but
still as a Not-ECN-PDU. still as a Not-ECN-PDU.
2. If the outer header does not support explicit congestion 2. If the outer header does not support ENC (a Not-ECN-PDU), but the
notification (a Not-ECN-PDU), but the inner header does (an ECN- inner header does (an ECN-PDU), the inner header SHOULD be
PDU), the inner header SHOULD be forwarded unchanged. forwarded unchanged.
3. In some lower layer protocols congestion may be signalled as a 3. In some lower-layer protocols, congestion may be signalled as a
numerical level, such as in the control frames of quantized numerical level, such as in the control frames of QCN
congestion notification (QCN [IEEE802.1Q]). If such a multi-bit [IEEE802.1Q]. If such a multi-bit encoding encapsulates an ECN-
encoding encapsulates an ECN-capable IP data packet, a function capable IP data packet, a function will be needed to convert the
will be needed to convert the quantized congestion level into the quantized congestion level into the frequency of congestion
frequency of congestion markings in outgoing IP packets. markings in outgoing IP packets.
4. Congestion indications might be encoded by a severity level. For 4. Congestion indications might be encoded by a severity level. For
instance increasing levels of congestion might be encoded by instance, increasing levels of congestion might be encoded by
numerically increasing indications, e.g. pre-congestion numerically increasing indications, e.g., PCN can be encoded in
notification (PCN) can be encoded in each PDU at three severity each PDU at three severity levels in IP or MPLS [RFC6660] and the
levels in IP or MPLS [RFC6660] and the default encapsulation and default encapsulation and decapsulation rules [RFC6040] are
decapsulation rules [RFC6040] are compatible with this compatible with this interpretation of the ECN field.
interpretation of the ECN field.
If the arriving inner header is an ECN-PDU, where the inner and If the arriving inner header is an ECN-PDU, where the inner and
outer headers carry indications of congestion of different outer headers carry indications of congestion of different
severity, the more severe indication SHOULD be forwarded in severity, the more severe indication SHOULD be forwarded in
preference to the less severe. preference to the less severe.
5. The inner and outer headers might carry a combination of 5. The inner and outer headers might carry a combination of
congestion notification fields that should not be possible given congestion notification fields that should not be possible given
any currently used protocol transitions. For instance, if any currently used protocol transitions. For instance, if
Encapsulation Guideline 3 in Section 4.3 had been followed, it Encapsulation Guideline 3 in Section 4.3 had been followed, it
should not be possible to have a less severe indication of should not be possible to have a less severe indication of
congestion in the outer than in the inner. It MAY be appropriate congestion in the outer header than in the inner header. It MAY
to log unexpected combinations of headers and possibly raise an be appropriate to log unexpected combinations of headers and
alarm. possibly raise an alarm.
If a safe outgoing codepoint can be defined for such a PDU, the If a safe outgoing codepoint can be defined for such a PDU, the
PDU SHOULD be forwarded rather than dropped. Some implementers PDU SHOULD be forwarded rather than dropped. Some implementers
discard PDUs with currently unused combinations of headers just discard PDUs with currently unused combinations of headers just
in case they represent an attack. However, an approach using in case they represent an attack. However, an approach using
alarms and policy-mediated drop is preferable to hard-coded drop, alarms and policy-mediated drop is preferable to hard-coded drop
so that operators can keep track of possible attacks but so that operators can keep track of possible attacks, but
currently unused combinations are not precluded from future use currently unused combinations are not precluded from future use
through new standards actions. through new standards actions.
4.5. Sequences of Similar Tunnels or Subnets 4.5. Sequences of Similar Tunnels or Subnets
In some deployments, particularly in 3GPP networks, an IP packet may In some deployments, particularly in 3GPP networks, an IP packet may
traverse two or more IP-in-IP tunnels in sequence that all use traverse (in sequence) two or more IP-in-IP tunnels that all use
identical technology (e.g. GTP). identical technology (e.g., GTP).
In such cases, it would be sufficient for every encapsulation and In such cases, it would be sufficient for every encapsulation and
decapsulation in the chain to comply with RFC 6040. Alternatively, decapsulation in the chain to comply with [RFC6040]. Alternatively,
as an optimisation, a node that decapsulates a packet and immediately as an optimization, a node that decapsulates a packet and immediately
re-encapsulates it for the next tunnel MAY copy the incoming outer re-encapsulates it for the next tunnel MAY copy the incoming outer
ECN field directly to the outgoing outer and the incoming inner ECN ECN field directly to the outgoing outer and the incoming inner ECN
field directly to the outgoing inner. Then the overall behavior field directly to the outgoing inner. Then, the overall behavior
across the sequence of tunnel segments would still be consistent with across the sequence of tunnel segments would still be consistent with
RFC 6040. [RFC6040].
Appendix C of RFC6040 describes how a tunnel egress can monitor how Appendix C of [RFC6040] describes how a tunnel egress can monitor how
much congestion has been introduced within a tunnel. A network much congestion has been introduced within a tunnel. A network
operator might want to monitor how much congestion had been operator might want to monitor how much congestion had been
introduced within a whole sequence of tunnels. Using the technique introduced within a whole sequence of tunnels. Using the technique
in Appendix C of RFC6040 at the final egress, the operator could in Appendix C of [RFC6040] at the final egress, the operator could
monitor the whole sequence of tunnels, but only if the above monitor the whole sequence of tunnels, but only if the above
optimisation were used consistently along the sequence of tunnels, in optimization were used consistently along the sequence of tunnels, in
order to make it appear as a single tunnel. Therefore, tunnel order to make it appear as a single tunnel. Therefore, tunnel
endpoint implementations SHOULD allow the operator to configure endpoint implementations SHOULD allow the operator to configure
whether this optimisation is enabled. whether this optimization is enabled.
When ECN support is added to a subnet technology, consideration When ECN support is added to a subnet technology, consideration
SHOULD be given to a similar optimisation between subnets in sequence SHOULD be given to a similar optimization between subnets in sequence
if they all use the same technology. if they all use the same technology.
4.6. Reframing and Congestion Markings 4.6. Reframing and Congestion Markings
The guidance in this section is worded in terms of framing The guidance in this section is worded in terms of framing
boundaries, but it applies equally whether the protocol data units boundaries, but it applies equally whether the PDUs are frames,
are frames, cells or packets. cells, or packets.
Where an AQM marks the ECN field of IP packets as they queue into a Where an AQM marks the ECN field of IP packets as they queue into a
layer-2 link, there will be no problem with framing boundaries, Layer 2 link, there will be no problem with framing boundaries
because the ECN markings would be applied directly to IP packets. because the ECN markings would be applied directly to IP packets.
The guidance in this section is only applicable where an ECN The guidance in this section is only applicable where an ECN
capability is being added to a layer-2 protocol so that layer-2 capability is being added to a Layer 2 protocol so that Layer 2
frames can be ECN-marked by an AQM at layer-2. This would only be frames can be ECN-marked by an AQM at layer 2. This would only be
necessary where AQM will be applied at pure layer-2 nodes (without necessary where AQM will be applied at pure Layer 2 nodes (without IP
IP-awareness). awareness).
Where ECN marking has had to be applied at non-IP-aware nodes and Where ECN marking has had to be applied at non-IP-aware nodes and
framing boundaries do not necessarily align with packet boundaries, framing boundaries do not necessarily align with packet boundaries,
the decapsulating IP forwarding node SHOULD propagate ECN markings the decapsulating IP forwarding node SHOULD propagate ECN markings
from layer-2 frame headers to IP packets that may have different from Layer 2 frame headers to IP packets that may have different
boundaries as a consequence of reframing. boundaries as a consequence of reframing.
Two possible design goals for propagating congestion indications, Two possible design goals for propagating congestion indications,
described in Section 5.3 of [RFC3168] and Section 2.4 of [RFC7141], described in Section 5.3 of [RFC3168] and Section 2.4 of [RFC7141],
are: are:
1. approximate preservation of the presence (and therefore timing) 1. approximate preservation of the presence (and therefore timing)
of congestion marks on the L2 frames used to construct an IP of congestion marks on the L2 frames used to construct an IP
packet; packet;
a. at high frequency of congestion marking, approximate a. at high frequency of congestion marking, approximate
preservation of the proportion of congestion marks arriving preservation of the proportion of congestion marks arriving
and departing; and departing;
b. at low frequency of congestion marking, approximate b. at low frequency of congestion marking, approximate
preservation of the timing of congestion marks arriving and preservation of the timing of congestion marks arriving and
departing. departing.
In either case, an implementation SHOULD ensure that any new incoming In either case, an implementation SHOULD ensure that any new incoming
congestion indication is propagated immediately, not held awaiting congestion indication is propagated immediately and not held awaiting
the possibility of further congestion indications to be sufficient to the possibility of further congestion indications to be sufficient to
indicate congestion on an outgoing PDU [RFC7141]. Nonetheless, to indicate congestion on an outgoing PDU [RFC7141]. Nonetheless, to
facilitate pipelined implementation, it would be acceptable for facilitate pipelined implementation, it would be acceptable for
congestion marks to propagate to a slightly later IP packet. congestion marks to propagate to a slightly later IP packet.
At decapsulation in either case: At decapsulation in either case, ECN-marking propagation logically
occurs before application of Decapsulation Guideline 1 in
* ECN marking propagation logically occurs before application of Section 4.4. For instance, if ECN-marking propagation would cause an
Decapsulation Guideline 1 in Section 4.4. For instance, if ECN ECN congestion indication to be applied to an IP packet that is a
marking propagation would cause an ECN congestion indication to be Not-ECN-PDU, then that IP packet is dropped in accordance with
applied to an IP packet that is a Not-ECN-PDU, then that IP packet Guideline 1:
is dropped in accordance with Guideline 1;
* where a mix of ECN-PDUs and non-ECN-PDUs arrives to construct the * where a mix of ECN-PDUs and non-ECN-PDUs arrives to construct the
same IP packet, the decapsulation spec SHOULD require that packet same IP packet, the decapsulation specification SHOULD require
to be discarded. that packet to be discarded.
* where a mix of different types of ECN-PDUs arrives to construct * where a mix of different types of ECN-PDUs arrives to construct
the same IP packet, e.g. a mix of frames that map to ECT(0) and the same IP packet, e.g., a mix of frames that map to ECT(0) and
ECT(1) IP packets, the decapsulation spec might consider this a ECT(1) IP packets, the decapsulation specification might consider
protocol error. But, if the lower layer protocol has defined such this a protocol error. But, if the lower-layer protocol has
a mix of types of ECN-PDU as valid, it SHOULD require the defined such a mix of types of ECN-PDU as valid, it SHOULD require
resulting IP packet to be set to either ECT(0) or ECT(1). In this the resulting IP packet to be set to either ECT(0) or ECT(1). In
case, it SHOULD take into account that the RFC series has so far this case, it SHOULD take into account that the RFC Series has so
allowed ECT(0) and ECT(1) to be considered equivalent [RFC3168], far allowed ECT(0) and ECT(1) to be considered equivalent
or ECT(1) can provide a less severe congestion marking than CE [RFC3168]; otherwise, ECT(1) can provide a less severe congestion
[RFC6040], or ECT(1) can indicate an unmarked but ECN-capable marking than CE [RFC6040] or ECT(1) can indicate an unmarked but
packet that is subject to a different marking algorithm to ECT(0) ECN-capable packet that is subject to a different marking
packets, for example L4S [RFC8311] [RFC9331]. algorithm to ECT(0) packets, e.g., L4S [RFC8311] [RFC9331].
The following are two ways that goal 1 might be achieved, but they The following are two ways that goal 1 might be achieved, but they
are not intended to be the only ways: are not intended to be the only ways:
* Every IP PDU that is constructed, in whole or in part, from an L2 * Every IP PDU that is constructed, in whole or in part, from an L2
frame that is marked with a congestion signal, has that signal frame that is marked with a congestion signal has that signal
propagated to it; propagated to it.
* Every L2 frame that is marked with a congestion signal, propagates * Every L2 frame that is marked with a congestion signal propagates
that signal to one IP PDU which is constructed, in whole or in that signal to one IP PDU that is constructed from it in whole or
part, from it. If multiple IP PDUs meet this description, the in part. If multiple IP PDUs meet this description, the choice
choice can be made arbitrarily but ought to be consistent. can be made arbitrarily but ought to be consistent.
The following gives one way that goal 2 might be achieved, but it is The following gives one way that goal 2 might be achieved, but it is
not intended to be the only way: not intended to be the only way:
* For each of the streams of frames that encapsulate the IP packets * For each of the streams of frames that encapsulate the IP packets
of each IP-ECN codepoint and follow the same path through the of each IP-ECN codepoint and follow the same path through the
subnet, a counter ('in') tracks octets arriving within the payload subnet, a counter ('in') tracks octets arriving within the payload
of marked L2 frames and another ('out') tracks octets departing in of marked L2 frames and another ('out') tracks octets departing in
marked IP packets. While 'in' exceeds 'out', forwarded IP packets marked IP packets. While 'in' exceeds 'out', forwarded IP packets
are ECN-marked. If 'out' exceeds 'in' for longer than a timeout, are ECN-marked. If 'out' exceeds 'in' for longer than a timeout,
both counters are zeroed, to ensure that the start of the next both counters are zeroed to ensure that the start of the next
congestion episode propagates immediately. The 'out' counter congestion episode propagates immediately. The 'out' counter
includes octets in reconstructed IP packets that would have been includes octets in reconstructed IP packets that would have been
marked, but had to be dropped because they were Not-ECN-PDUs (by marked, but had to be dropped because they were Not-ECN-PDUs (by
Decapsulation Guideline 1 in Section 4.4). Decapsulation Guideline 1 in Section 4.4).
Generally, the number of L2 frames may be higher (e.g. ATM), similar Generally, the number of L2 frames may be higher (e.g., ATM) than,
to, or lower (e.g. 802.11 aggregation at a L2-only station) than the similar to, or lower (e.g., 802.11 aggregation at an L2-only station)
number of IP PDUs, and this distinction may influence the choice of than the number of IP PDUs; this distinction may influence the choice
mechanism. of mechanism.
5. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion 5. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion
Notification Notification
The guidance in this section is applicable, for example, when IP The guidance in this section is applicable, for example, when IP
packets: packets:
* are encapsulated in Ethernet headers, which have no support for * are encapsulated in Ethernet headers, which have no support for
ECN; ECN;
* are forwarded by the eNode-B (base station) of a 3GPP radio access * are forwarded by the eNode-B (base station) of a 3GPP radio access
network, which is required to apply ECN marking during congestion, network, which is required to apply ECN marking during congestion
[LTE-RA], [UTRAN], but the Packet Data Convergence Protocol (PDCP) [LTE-RA] [UTRAN], but the Packet Data Convergence Protocol (PDCP)
that encapsulates the IP header over the radio access has no that encapsulates the IP header over the radio access has no
support for ECN. support for ECN.
This guidance also generalizes to encapsulation by other subnet This guidance also generalizes to encapsulation by other subnet
technologies with no native support for explicit congestion technologies with no native support for ECN at the lower layer, but
notification at the lower layer, but with support for finding and with support for finding and processing an IP header. It is unlikely
processing an IP header. It is unlikely to be applicable or to be applicable or necessary for IP-in-IP encapsulation, where feed-
necessary for IP-in-IP encapsulation, where feed-forward-and-up mode forward-and-up mode based on [RFC6040] would be more appropriate.
based on [RFC6040] would be more appropriate.
Marking the IP header while switching at layer-2 (by using a layer-3 Marking the IP header while switching at layer 2 (by using a Layer 3
switch) or while forwarding in a radio access network seems to switch) or while forwarding in a radio access network seems to
represent a layering violation. However, it can be considered as a represent a layering violation. However, it can be considered as a
benign optimisation if the guidelines below are followed. Feed-up- benign optimization if the guidelines below are followed. Feed-up-
and-forward is certainly not a general alternative to implementing and-forward is certainly not a general alternative to implementing
feed-forward congestion notification in the lower layer, because: feed-forward congestion notification in the lower layer, because:
* IPv4 and IPv6 are not the only layer-3 protocols that might be * IPv4 and IPv6 are not the only Layer 3 protocols that might be
encapsulated by lower layer protocols encapsulated by lower-layer protocols.
* Link-layer encryption might be in use, making the layer-2 payload * Link-layer encryption might be in use, making the Layer 2 payload
inaccessible inaccessible.
* Many Ethernet switches do not have 'layer-3 switch' capabilities * Many Ethernet switches do not have 'Layer 3 switch' capabilities,
so they cannot read or modify an IP payload so they cannot read or modify an IP payload.
* It might be costly to find an IP header (IPv4 or IPv6) when it may * It might be costly to find an IP header (IPv4 or IPv6) when it may
be encapsulated by more than one lower layer header, e.g. be encapsulated by more than one lower-layer header, e.g.,
Ethernet MAC in MAC ([IEEE802.1Q]; previously 802.1ah). Ethernet MAC in MAC ([IEEE802.1Q]; previously 802.1ah).
Nonetheless, configuring lower layer equipment to look for an ECN Nonetheless, configuring lower-layer equipment to look for an ECN
field in an encapsulated IP header is a useful optimisation. If the field in an encapsulated IP header is a useful optimization. If the
implementation follows the guidelines below, this optimisation does implementation follows the guidelines below, this optimization does
not have to be confined to a controlled environment such as within a not have to be confined to a controlled environment, e.g., within a
data centre; it could usefully be applied on any network — even if data centre; it could usefully be applied on any network -- even if
the operator is not sure whether the above issues will never apply: the operator is not sure whether the above issues will never apply:
1. If a native lower-layer congestion notification mechanism exists 1. If a native lower-layer congestion notification mechanism exists
for a subnet technology, it is safe to mix feed-up-and-forward for a subnet technology, it is safe to mix feed-up-and-forward
with feed-forward-and-up on other switches in the same subnet. with feed-forward-and-up on other switches in the same subnet.
However, it will generally be more efficient to use the native However, it will generally be more efficient to use the native
mechanism. mechanism.
2. The depth of the search for an IP header SHOULD be limited. If 2. The depth of the search for an IP header SHOULD be limited. If
an IP header is not found soon enough, or an unrecognized or an IP header is not found soon enough, or an unrecognized or
unreadable header is encountered, the switch SHOULD resort to an unreadable header is encountered, the switch SHOULD resort to an
alternative means of signalling congestion (e.g. drop, or the alternative means of signalling congestion (e.g., drop or the
native lower layer mechanism if available). native lower-layer mechanism if available).
3. It is sufficient to use the first IP header found in the stack; 3. It is sufficient to use the first IP header found in the stack;
the egress of the relevant tunnel can propagate congestion the egress of the relevant tunnel can propagate congestion
notification upwards to any more deeply encapsulated IP headers notification upwards to any more deeply encapsulated IP headers
later. later.
6. Feed-Backward Mode: Guidelines for Adding Congestion Notification 6. Feed-Backward Mode: Guidelines for Adding Congestion Notification
It can be seen from Section 3.3 that congestion notification in a It can be seen from Section 3.3 that congestion notifications in a
subnet using feed-backward mode has generally not been designed to be subnet using feed-backward mode have generally not been designed to
directly coupled with IP layer congestion notification. The subnet be directly coupled with IP-layer congestion notifications. The
attempts to minimize congestion internally, and if the incoming load subnet attempts to minimize congestion internally, and if the
at the ingress exceeds the capacity somewhere through the subnet, the incoming load at the ingress exceeds the capacity somewhere through
layer 3 buffer into the ingress backs up. Thus, a feed-backward mode the subnet, the Layer 3 buffer into the ingress backs up. Thus, a
subnet is in some sense similar to a null mode subnet, in that there feed-backward mode subnet is in some sense similar to a null mode
is no need for any direct interaction between the subnet and higher subnet, in that there is no need for any direct interaction between
layer congestion notification. Therefore no detailed protocol design the subnet and higher-layer congestion notifications. Therefore, no
guidelines are appropriate. Nonetheless, a more general guideline is detailed protocol design guidelines are appropriate. Nonetheless, a
appropriate: more general guideline is appropriate:
A subnetwork technology intended to eventually interface to IP | A subnetwork technology intended to eventually interface to IP
SHOULD NOT be designed using only the feed-backward mode, which is | SHOULD NOT be designed using only the feed-backward mode, which is
certainly best for a stand-alone subnet, but would need to be | certainly best for a stand-alone subnet, but would need to be
modified to work efficiently as part of the wider Internet, | modified to work efficiently as part of the wider Internet because
because IP uses feed-forward-and-up mode. | IP uses feed-forward-and-up mode.
The feed-backward approach at least works beneath IP, where the term The feed-backward approach at least works beneath IP, where the term
'works' is used only in a narrow functional sense because feed- 'works' is used only in a narrow functional sense because feed-
backward can result in very inefficient and sluggish congestion backward can result in very inefficient and sluggish congestion
control — except if it is confined to the subnet directly connected control -- except if it is confined to the subnet directly connected
to the original data source, when it is faster than feed-forward. It to the original data source when it is faster than feed-forward. It
would be valid to design a protocol that could work in feed-backward would be valid to design a protocol that could work in feed-backward
mode for paths that only cross one subnet, and in feed-forward-and-up mode for paths that only cross one subnet, and in feed-forward-and-up
mode for paths that cross subnets. mode for paths that cross subnets.
In the early days of TCP/IP, a similar feed-backward approach was In the early days of TCP/IP, a similar feed-backward approach was
tried for explicit congestion signalling, using source-quench (SQ) tried for explicit congestion signalling using source-quench (SQ)
ICMP control packets. However, SQ fell out of favour and is now ICMP control packets. However, SQ fell out of favour and is now
formally deprecated [RFC6633]. The main problem was that it is hard formally deprecated [RFC6633]. The main problem was that it is hard
for a data source to tell the difference between a spoofed SQ message for a data source to tell the difference between a spoofed SQ message
and a quench request from a genuine buffer on the path. It is also and a quench request from a genuine buffer on the path. It is also
hard for a lower layer buffer to address an SQ message to the hard for a lower-layer buffer to address an SQ message to the
original source port number, which may be buried within many layers original source port number, which may be buried within many layers
of headers, and possibly encrypted. of headers and possibly encrypted.
QCN (also known as backward congestion notification, BCN; see QCN (also known as Backward Congestion Notification (BCN); see
Sections 30–33 of [IEEE802.1Q]; previously known as 802.1Qau) uses a Sections 30-33 of [IEEE802.1Q], previously known as 802.1Qau) uses a
feed-backward mode structurally similar to ATM's relative rate feed-backward mode that is structurally similar to ATM's relative
mechanism. However, QCN confines its applicability to scenarios such rate mechanism. However, QCN confines its applicability to scenarios
as some data centres where all endpoints are directly attached by the such as some data centres where all endpoints are directly attached
same Ethernet technology. If a QCN subnet were later connected into by the same Ethernet technology. If a QCN subnet were later
a wider IP-based internetwork (e.g. when attempting to interconnect connected into a wider IP-based internetwork (e.g., when attempting
multiple data centres) it would suffer the inefficiency shown in to interconnect multiple data centres) it would suffer the
Figure 3. inefficiency shown in Figure 3.
7. IANA Considerations 7. IANA Considerations
This section is to be removed before publishing as an RFC. This document has no IANA actions.
This memo includes no request to IANA.
8. Security Considerations 8. Security Considerations
If a lower layer wire protocol is redesigned to include explicit If a lower-layer wire protocol is redesigned to include explicit
congestion signalling in-band in the protocol header, care SHOULD be congestion signalling in-band in the protocol header, care SHOULD be
taken to ensure that the field used is specified as mutable during taken to ensure that the field used is specified as mutable during
transit. Otherwise interior nodes signalling congestion would transit. Otherwise, interior nodes signalling congestion would
invalidate any authentication protocol applied to the lower layer invalidate any authentication protocol applied to the lower-layer
header — by altering a header field that had been assumed as header -- by altering a header field that had been assumed as
immutable. immutable.
The redesign of protocols that encapsulate IP in order to propagate The redesign of protocols that encapsulate IP in order to propagate
congestion signals between layers raises potential signal integrity congestion signals between layers raises potential signal integrity
concerns. Experimental or proposed approaches exist for assuring the concerns. Experimental or proposed approaches exist for assuring the
end-to-end integrity of in-band congestion signals, e.g.: end-to-end integrity of in-band congestion signals, such as:
* Congestion exposure (ConEx) for networks to audit that their * Congestion exposure (ConEx) for networks to audit that their
congestion signals are not being suppressed by other networks or congestion signals are not being suppressed by other networks or
by receivers, and for networks to police that senders are by receivers, and to police that senders are responding
responding sufficiently to the signals, irrespective of the L4 sufficiently to the signals, irrespective of the L4 transport
transport protocol used [RFC7713]. protocol used [RFC7713].
* A test for a sender to detect whether a network or the receiver is * A test for a sender to detect whether a network or the receiver is
suppressing congestion signals (for example see 2nd para of suppressing congestion signals (for example, see the second
Section 20.2 of [RFC3168]). paragraph of Section 20.2 of [RFC3168]).
Given these end-to-end approaches are already being specified, it Given these end-to-end approaches are already being specified, it
would make little sense to attempt to design hop-by-hop congestion would make little sense to attempt to design hop-by-hop congestion
signal integrity into a new lower layer protocol, because end-to-end signal integrity into a new lower-layer protocol because end-to-end
integrity inherently achieves hop-by-hop integrity. integrity inherently achieves hop-by-hop integrity.
Section 6 gives vulnerability to spoofing as one of the reasons for Section 6 gives vulnerability to spoofing as one of the reasons for
deprecating feed-backward mode. deprecating feed-backward mode.
9. Conclusions 9. Conclusions
Following the guidance in this document enables ECN support to be Following the guidance in this document enables ECN support to be
extended consistently to numerous protocols that encapsulate IP (IPv4 extended consistently to numerous protocols that encapsulate IP (IPv4
and IPv6), so that IP continues to fulfil its role as an end-to-end and IPv6) so that IP continues to fulfil its role as an end-to-end
interoperability layer. This includes: interoperability layer. This includes:
* A wide range of tunnelling protocols including those with various * A wide range of tunnelling protocols, including those with various
forms of shim header between two IP headers, possibly also forms of a shim header between two IP headers, possibly also
separated by a L2 header; separated by an L2 header;
* A wide range of subnet technologies, particularly those that work * A wide range of subnet technologies, particularly those that work
in the same 'feed-forward-and-up' mode that is used to support ECN in the same 'feed-forward-and-up' mode that is used to support ECN
in IP and MPLS. in IP and MPLS.
Guidelines have been defined for supporting propagation of ECN Guidelines have been defined for supporting propagation of ECN
between Ethernet and IP on so-called Layer-3 Ethernet switches, using between Ethernet and IP on so-called Layer 3 Ethernet switches using
a 'feed-up-and-forward' mode. This approach could enable other a 'feed-up-and-forward' mode. This approach could enable other
subnet technologies to pass ECN signals into the IP layer, even if subnet technologies to pass ECN signals into the IP layer, even if
they do not support ECN natively. they do not support ECN natively.
Finally, attempting to add ECN to a subnet technology in feed- Finally, attempting to add ECN to a subnet technology in feed-
backward mode is deprecated except in special cases, due to its backward mode is deprecated except in special cases due to its likely
likely sluggish response to congestion. sluggish response to congestion.
10. References 10. References
10.1. Normative References 10.1. Normative References
[I-D.ietf-trill-ecn-support]
Eastlake, D. E. and B. Briscoe, "TRILL (TRansparent
Interconnection of Lots of Links): ECN (Explicit
Congestion Notification) Support", Work in Progress,
Internet-Draft, draft-ietf-trill-ecn-support-07, 25
February 2018, <https://datatracker.ietf.org/doc/html/
draft-ietf-trill-ecn-support-07>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>. <https://www.rfc-editor.org/info/rfc2119>.
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
of Explicit Congestion Notification (ECN) to IP", of Explicit Congestion Notification (ECN) to IP",
RFC 3168, DOI 10.17487/RFC3168, September 2001, RFC 3168, DOI 10.17487/RFC3168, September 2001,
<https://www.rfc-editor.org/info/rfc3168>. <https://www.rfc-editor.org/info/rfc3168>.
skipping to change at page 30, line 13 skipping to change at line 1322
2008, <https://www.rfc-editor.org/info/rfc5129>. 2008, <https://www.rfc-editor.org/info/rfc5129>.
[RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion [RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion
Notification", RFC 6040, DOI 10.17487/RFC6040, November Notification", RFC 6040, DOI 10.17487/RFC6040, November
2010, <https://www.rfc-editor.org/info/rfc6040>. 2010, <https://www.rfc-editor.org/info/rfc6040>.
[RFC7141] Briscoe, B. and J. Manner, "Byte and Packet Congestion [RFC7141] Briscoe, B. and J. Manner, "Byte and Packet Congestion
Notification", BCP 41, RFC 7141, DOI 10.17487/RFC7141, Notification", BCP 41, RFC 7141, DOI 10.17487/RFC7141,
February 2014, <https://www.rfc-editor.org/info/rfc7141>. February 2014, <https://www.rfc-editor.org/info/rfc7141>.
[RFC9600] Eastlake, D. and B. Briscoe, "TRILL (TRansparent
Interconnection of Lots of Links): ECN (Explicit
Congestion Notification) Support", RFC 9600,
DOI 10.17487/RFC9600, June 2024,
<https://www.rfc-editor.org/info/rfc9600>.
10.2. Informative References 10.2. Informative References
[ATM-TM-ABR] [ATM-TM-ABR]
Cisco, "Understanding the Available Bit Rate (ABR) Service Cisco, "Understanding the Available Bit Rate (ABR) Service
Category for ATM VCs", Design Technote 10415, 5 June 2005, Category for ATM VCs", Design Technote 10415, June 2005,
<https://www.cisco.com/c/en/us/support/docs/asynchronous- <https://www.cisco.com/c/en/us/support/docs/asynchronous-
transfer-mode-atm/atm-traffic- transfer-mode-atm/atm-traffic-
management/10415-atmabr.html>. management/10415-atmabr.html>.
[Buck00] Buckwalter, J.T., "Frame Relay: Technology and Practice", [Buck00] Buckwalter, J.T., "Frame Relay: Technology and Practice",
Pub. Addison Wesley ISBN-13: 978-0201485240, 2000. Addison-Wesley Professional, ISBN-13 978-0201485240, 2000.
[Clos53] Clos, C., "A Study of Non-Blocking Switching Networks", [Clos53] Clos, C., "A Study of Non-Blocking Switching Networks",
Bell Systems Technical Journal 32(2):406–424, March 1953. The Bell System Technical Journal, Vol. 32, Issue 2,
DOI 10.1002/j.1538-7305.1953.tb01433.x, March 1953,
<https://doi.org/10.1002/j.1538-7305.1953.tb01433.x>.
[GTPv1] 3GPP, "GPRS Tunnelling Protocol (GTP) across the Gn and Gp [GTPv1] 3GPP, "General Packet Radio Service (GPRS); GPRS
interface", Technical Specification TS 29.060. Tunnelling Protocol (GTP) across the Gn and Gp interface",
Technical Specification 29.060.
[GTPv1-U] 3GPP, "General Packet Radio System (GPRS) Tunnelling [GTPv1-U] 3GPP, "General Packet Radio System (GPRS) Tunnelling
Protocol User Plane (GTPv1-U)", Technical Specification TS Protocol User Plane (GTPv1-U)", Technical
29.281. Specification 29.281.
[GTPv2-C] 3GPP, "Evolved General Packet Radio Service (GPRS) [GTPv2-C] 3GPP, "3GPP Evolved Packet System (EPS); Evolved General
Tunnelling Protocol for Control plane (GTPv2-C)", Packet Radio Service (GPRS) Tunnelling Protocol for
Technical Specification TS 29.274. Control plane (GTPv2-C); Stage 3", Technical
Specification 29.274.
[I-D.ietf-intarea-gue] [IEEE802.1Q]
IEEE, "IEEE Standard for Local and Metropolitan Area
Network--Bridges and Bridged Networks", IEEE Std 802.1Q-
2018, DOI 10.1109/IEEESTD.2018.8403927, July 2018,
<https://doi.org/10.1109/IEEESTD.2018.8403927>.
[INTAREA-GUE]
Herbert, T., Yong, L., and O. Zia, "Generic UDP Herbert, T., Yong, L., and O. Zia, "Generic UDP
Encapsulation", Work in Progress, Internet-Draft, draft- Encapsulation", Work in Progress, Internet-Draft, draft-
ietf-intarea-gue-09, 26 October 2019, ietf-intarea-gue-09, 26 October 2019,
<https://datatracker.ietf.org/doc/html/draft-ietf-intarea- <https://datatracker.ietf.org/doc/html/draft-ietf-intarea-
gue-09>. gue-09>.
[I-D.ietf-tsvwg-rfc6040update-shim]
Briscoe, B., "Propagating Explicit Congestion Notification
Across IP Tunnel Headers Separated by a Shim", Work in
Progress, Internet-Draft, draft-ietf-tsvwg-rfc6040update-
shim-22, 29 October 2023,
<https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-
rfc6040update-shim-22>.
[IEEE802.1Q]
IEEE, "IEEE Standard for Local and Metropolitan Area
Networks—Virtual Bridged Local Area Networks—Amendment 6:
Provider Backbone Bridges", IEEE Std 802.1Q-2018, July
2018, <https://ieeexplore.ieee.org/document/8403927>.
[ITU-T.I.371] [ITU-T.I.371]
ITU-T, "Traffic Control and Congestion Control in B-ISDN", ITU-T, "Traffic control and congestion control in B-ISDN",
ITU-T Rec. I.371 (03/04), March 2004, ITU-T Recommendation I.371, March 2004,
<https://www.itu.int/rec/T-REC-I.371>. <https://www.itu.int/rec/T-REC-I.371-200403-I/en>.
[Leiserson85] [Leiserson85]
Leiserson, C.E., "Fat-trees: universal networks for Leiserson, C.E., "Fat-trees: Universal networks for
hardware-efficient supercomputing", IEEE Transactions on hardware-efficient supercomputing", IEEE Transactions on
Computers 34(10):892–901, October 1985. Computers, Vol. C-34, Issue 10,
DOI 10.1109/TC.1985.6312192, October 1985,
<https://doi.org/10.1109/TC.1985.6312192>.
[LTE-RA] 3GPP, "Evolved Universal Terrestrial Radio Access (E-UTRA) [LTE-RA] 3GPP, "Evolved Universal Terrestrial Radio Access (E-UTRA)
and Evolved Universal Terrestrial Radio Access Network and Evolved Universal Terrestrial Radio Access Network
(E-UTRAN); Overall description; Stage 2", Technical (E-UTRAN); Overall description; Stage 2", Technical
Specification TS 36.300. Specification 36.300.
[RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003,
DOI 10.17487/RFC2003, October 1996, DOI 10.17487/RFC2003, October 1996,
<https://www.rfc-editor.org/info/rfc2003>. <https://www.rfc-editor.org/info/rfc2003>.
[RFC2473] Conta, A. and S. Deering, "Generic Packet Tunneling in [RFC2473] Conta, A. and S. Deering, "Generic Packet Tunneling in
IPv6 Specification", RFC 2473, DOI 10.17487/RFC2473, IPv6 Specification", RFC 2473, DOI 10.17487/RFC2473,
December 1998, <https://www.rfc-editor.org/info/rfc2473>. December 1998, <https://www.rfc-editor.org/info/rfc2473>.
[RFC2637] Hamzeh, K., Pall, G., Verthein, W., Taarud, J., Little, [RFC2637] Hamzeh, K., Pall, G., Verthein, W., Taarud, J., Little,
skipping to change at page 34, line 21 skipping to change at line 1527
Cabellos, Ed., "The Locator/ID Separation Protocol Cabellos, Ed., "The Locator/ID Separation Protocol
(LISP)", RFC 9300, DOI 10.17487/RFC9300, October 2022, (LISP)", RFC 9300, DOI 10.17487/RFC9300, October 2022,
<https://www.rfc-editor.org/info/rfc9300>. <https://www.rfc-editor.org/info/rfc9300>.
[RFC9331] De Schepper, K. and B. Briscoe, Ed., "The Explicit [RFC9331] De Schepper, K. and B. Briscoe, Ed., "The Explicit
Congestion Notification (ECN) Protocol for Low Latency, Congestion Notification (ECN) Protocol for Low Latency,
Low Loss, and Scalable Throughput (L4S)", RFC 9331, Low Loss, and Scalable Throughput (L4S)", RFC 9331,
DOI 10.17487/RFC9331, January 2023, DOI 10.17487/RFC9331, January 2023,
<https://www.rfc-editor.org/info/rfc9331>. <https://www.rfc-editor.org/info/rfc9331>.
[UTRAN] 3GPP, "UTRAN Overall Description", Technical [RFC9601] Briscoe, B., "Propagating Explicit Congestion Notification
Specification TS 25.401. Across IP Tunnel Headers Separated by a Shim", RFC 9601,
DOI 10.17487/RFC9601, June 2024,
Comments Solicited <https://www.rfc-editor.org/info/rfc9601>.
This section is to be removed before publishing as an RFC.
Comments and questions are encouraged and very welcome. They can be [UTRAN] 3GPP, "UTRAN overall description", Technical
addressed to the IETF Transport Area working group mailing list Specification 25.401.
<tsvwg@ietf.org>, and/or to the authors.
Acknowledgements Acknowledgements
Thanks to Gorry Fairhurst and David Black for extensive reviews. Thanks to Gorry Fairhurst and David Black for extensive reviews.
Thanks also to the following reviewers: Joe Touch, Andrew McGregor, Thanks also to the following reviewers: Joe Touch, Andrew McGregor,
Richard Scheffenegger, Ingemar Johansson, Piers O'Hanlon, Donald Richard Scheffenegger, Ingemar Johansson, Piers O'Hanlon, Donald
Eastlake, Jonathan Morton, Markku Kojo, Sebastian Möller, Martin Duke Eastlake 3rd, Jonathan Morton, Markku Kojo, Sebastian Möller, Martin
and Michael Welzl, who pointed out that lower layer congestion Duke, and Michael Welzl, who pointed out that lower-layer congestion
notification signals may have different semantics to those in IP. notification signals may have different semantics to those in IP.
Thanks are also due to the tsvwg chairs, TSV ADs and IETF liaison Thanks are also due to the Transport and Services Working Group
people such as Eric Gray, Dan Romascanu and Gonzalo Camarillo for (tsvwg) chairs, TSV ADs and IETF liaison people such as Eric Gray,
helping with the liaisons with the IEEE and 3GPP. And thanks to Dan Romascanu and Gonzalo Camarillo for helping with the liaisons
Georg Mayer and particularly to Erik Guttman for the extensive search with the IEEE and 3GPP. And thanks to Georg Mayer and particularly
and categorisation of any 3GPP specifications that cite ECN to Erik Guttman for the extensive search and categorisation of any
specifications. Thanks also to the Area Reviewers Dan Harkins, Paul 3GPP specifications that cite ECN specifications. Thanks also to the
Kyzivat, Sue Hares and Dale Worley. Area Reviewers Dan Harkins, Paul Kyzivat, Sue Hares, and Dale Worley.
Bob Briscoe was part-funded by the European Community under its Bob Briscoe was part-funded by the European Community under its
Seventh Framework Programme through the Trilogy project (ICT-216372) Seventh Framework Programme through the Trilogy project (ICT-216372)
for initial drafts then through the Reducing Internet Transport and the Reducing Internet Transport Latency (RITE) project (ICT-
Latency (RITE) project (ICT-317700), and for final drafts (from -18) 317700) for earlier draft versions of this document. For later draft
he was funded by Apple Inc. The views expressed here are solely those versions of this document, he was funded by Apple Inc. The views
of the authors. expressed here are solely those of the authors.
Contributors Contributors
Pat Thaler Pat Thaler
Broadcom Corporation (retired) Broadcom Corporation (retired)
CA CA
USA United States of America
Pat was a co-author of this draft, but retired before its Pat was a coauthor of this draft, but retired before its publication.
publication.
Authors' Addresses Authors' Addresses
Bob Briscoe Bob Briscoe
Independent Independent
United Kingdom United Kingdom
Email: ietf@bobbriscoe.net Email: ietf@bobbriscoe.net
URI: https://bobbriscoe.net/ URI: https://bobbriscoe.net/
John Kaippallimalil John Kaippallimalil
 End of changes. 216 change blocks. 
672 lines changed or deleted 640 lines changed or added

This html diff was produced by rfcdiff 1.48.