1.. SPDX-License-Identifier: GPL-2.0
2
3====================================
4Netfilter's flowtable infrastructure
5====================================
6
7This documentation describes the Netfilter flowtable infrastructure which allows
8you to define a fastpath through the flowtable datapath. This infrastructure
9also provides hardware offload support. The flowtable supports for the layer 3
10IPv4 and IPv6 and the layer 4 TCP and UDP protocols.
11
12Overview
13--------
14
15Once the first packet of the flow successfully goes through the IP forwarding
16path, from the second packet on, you might decide to offload the flow to the
17flowtable through your ruleset. The flowtable infrastructure provides a rule
18action that allows you to specify when to add a flow to the flowtable.
19
20A packet that finds a matching entry in the flowtable (ie. flowtable hit) is
21transmitted to the output netdevice via neigh_xmit(), hence, packets bypass the
22classic IP forwarding path (the visible effect is that you do not see these
23packets from any of the Netfilter hooks coming after ingress). In case that
24there is no matching entry in the flowtable (ie. flowtable miss), the packet
25follows the classic IP forwarding path.
26
27The flowtable uses a resizable hashtable. Lookups are based on the following
28n-tuple selectors: layer 2 protocol encapsulation (VLAN and PPPoE), layer 3
29source and destination, layer 4 source and destination ports and the input
30interface (useful in case there are several conntrack zones in place).
31
32The 'flow add' action allows you to populate the flowtable, the user selectively
33specifies what flows are placed into the flowtable. Hence, packets follow the
34classic IP forwarding path unless the user explicitly instruct flows to use this
35new alternative forwarding path via policy.
36
37The flowtable datapath is represented in Fig.1, which describes the classic IP
38forwarding path including the Netfilter hooks and the flowtable fastpath bypass.
39
40::
41
42					 userspace process
43					  ^              |
44					  |              |
45				     _____|____     ____\/___
46				    /          \   /         \
47				    |   input   |  |  output  |
48				    \__________/   \_________/
49					 ^               |
50					 |               |
51      _________      __________      ---------     _____\/_____
52     /         \    /          \     |Routing |   /            \
53  -->  ingress  ---> prerouting ---> |decision|   | postrouting |--> neigh_xmit
54     \_________/    \__________/     ----------   \____________/          ^
55       |      ^                          |               ^                |
56   flowtable  |                     ____\/___            |                |
57       |      |                    /         \           |                |
58    __\/___   |                    | forward |------------                |
59    |-----|   |                    \_________/                            |
60    |-----|   |                 'flow offload' rule                       |
61    |-----|   |                   adds entry to                           |
62    |_____|   |                     flowtable                             |
63       |      |                                                           |
64      / \     |                                                           |
65     /hit\_no_|                                                           |
66     \ ? /                                                                |
67      \ /                                                                 |
68       |__yes_________________fastpath bypass ____________________________|
69
70	       Fig.1 Netfilter hooks and flowtable interactions
71
72The flowtable entry also stores the NAT configuration, so all packets are
73mangled according to the NAT policy that is specified from the classic IP
74forwarding path. The TTL is decremented before calling neigh_xmit(). Fragmented
75traffic is passed up to follow the classic IP forwarding path given that the
76transport header is missing, in this case, flowtable lookups are not possible.
77TCP RST and FIN packets are also passed up to the classic IP forwarding path to
78release the flow gracefully. Packets that exceed the MTU are also passed up to
79the classic forwarding path to report packet-too-big ICMP errors to the sender.
80
81Example configuration
82---------------------
83
84Enabling the flowtable bypass is relatively easy, you only need to create a
85flowtable and add one rule to your forward chain::
86
87	table inet x {
88		flowtable f {
89			hook ingress priority 0; devices = { eth0, eth1 };
90		}
91		chain y {
92			type filter hook forward priority 0; policy accept;
93			ip protocol tcp flow add @f
94			counter packets 0 bytes 0
95		}
96	}
97
98This example adds the flowtable 'f' to the ingress hook of the eth0 and eth1
99netdevices. You can create as many flowtables as you want in case you need to
100perform resource partitioning. The flowtable priority defines the order in which
101hooks are run in the pipeline, this is convenient in case you already have a
102nftables ingress chain (make sure the flowtable priority is smaller than the
103nftables ingress chain hence the flowtable runs before in the pipeline).
104
105The 'flow offload' action from the forward chain 'y' adds an entry to the
106flowtable for the TCP syn-ack packet coming in the reply direction. Once the
107flow is offloaded, you will observe that the counter rule in the example above
108does not get updated for the packets that are being forwarded through the
109forwarding bypass.
110
111You can identify offloaded flows through the [OFFLOAD] tag when listing your
112connection tracking table.
113
114::
115
116	# conntrack -L
117	tcp      6 src=10.141.10.2 dst=192.168.10.2 sport=52728 dport=5201 src=192.168.10.2 dst=192.168.10.1 sport=5201 dport=52728 [OFFLOAD] mark=0 use=2
118
119
120Layer 2 encapsulation
121---------------------
122
123Since Linux kernel 5.13, the flowtable infrastructure discovers the real
124netdevice behind VLAN and PPPoE netdevices. The flowtable software datapath
125parses the VLAN and PPPoE layer 2 headers to extract the ethertype and the
126VLAN ID / PPPoE session ID which are used for the flowtable lookups. The
127flowtable datapath also deals with layer 2 decapsulation.
128
129You do not need to add the PPPoE and the VLAN devices to your flowtable,
130instead the real device is sufficient for the flowtable to track your flows.
131
132Bridge and IP forwarding
133------------------------
134
135Since Linux kernel 5.13, you can add bridge ports to the flowtable. The
136flowtable infrastructure discovers the topology behind the bridge device. This
137allows the flowtable to define a fastpath bypass between the bridge ports
138(represented as eth1 and eth2 in the example figure below) and the gateway
139device (represented as eth0) in your switch/router.
140
141::
142
143                      fastpath bypass
144               .-------------------------.
145              /                           \
146              |           IP forwarding   |
147              |          /             \ \/
148              |       br0               eth0 ..... eth0
149              .       / \                          *host B*
150               -> eth1  eth2
151                   .           *switch/router*
152                   .
153                   .
154                 eth0
155               *host A*
156
157The flowtable infrastructure also supports for bridge VLAN filtering actions
158such as PVID and untagged. You can also stack a classic VLAN device on top of
159your bridge port.
160
161If you would like that your flowtable defines a fastpath between your bridge
162ports and your IP forwarding path, you have to add your bridge ports (as
163represented by the real netdevice) to your flowtable definition.
164
165Counters
166--------
167
168The flowtable can synchronize packet and byte counters with the existing
169connection tracking entry by specifying the counter statement in your flowtable
170definition, e.g.
171
172::
173
174	table inet x {
175		flowtable f {
176			hook ingress priority 0; devices = { eth0, eth1 };
177			counter
178		}
179	}
180
181Counter support is available since Linux kernel 5.7.
182
183Hardware offload
184----------------
185
186If your network device provides hardware offload support, you can turn it on by
187means of the 'offload' flag in your flowtable definition, e.g.
188
189::
190
191	table inet x {
192		flowtable f {
193			hook ingress priority 0; devices = { eth0, eth1 };
194			flags offload;
195		}
196	}
197
198There is a workqueue that adds the flows to the hardware. Note that a few
199packets might still run over the flowtable software path until the workqueue has
200a chance to offload the flow to the network device.
201
202You can identify hardware offloaded flows through the [HW_OFFLOAD] tag when
203listing your connection tracking table. Please, note that the [OFFLOAD] tag
204refers to the software offload mode, so there is a distinction between [OFFLOAD]
205which refers to the software flowtable fastpath and [HW_OFFLOAD] which refers
206to the hardware offload datapath being used by the flow.
207
208The flowtable hardware offload infrastructure also supports for the DSA
209(Distributed Switch Architecture).
210
211Limitations
212-----------
213
214The flowtable behaves like a cache. The flowtable entries might get stale if
215either the destination MAC address or the egress netdevice that is used for
216transmission changes.
217
218This might be a problem if:
219
220- You run the flowtable in software mode and you combine bridge and IP
221  forwarding in your setup.
222- Hardware offload is enabled.
223
224More reading
225------------
226
227This documentation is based on the LWN.net articles [1]_\ [2]_. Rafal Milecki
228also made a very complete and comprehensive summary called "A state of network
229acceleration" that describes how things were before this infrastructure was
230mainlined [3]_ and it also makes a rough summary of this work [4]_.
231
232.. [1] https://lwn.net/Articles/738214/
233.. [2] https://lwn.net/Articles/742164/
234.. [3] http://lists.infradead.org/pipermail/lede-dev/2018-January/010830.html
235.. [4] http://lists.infradead.org/pipermail/lede-dev/2018-January/010829.html
236