pflow: add RFC8158 NAT support

Extend pflow(4) to send NAT44 Session Create and Delete events.
This applies only to IPFIX (i.e. proto version 10), and requires no
user configuration.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D43114

pf: allow pflow to be activated per rule

Only generate ipfix/netflow reports (through pflow) for the rules where
this is enabled. Reports can also be enabled globally through 'set
state-default pflow'.

Obtained from: OpenBSD
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D43108

pfctl: fix incorrect mask on dynamic address

A PF rule using an IPv4 address followed by an IPv6 address and then a
dynamic address, e.g. "pass from {192.0.2.1 2001:db8::1} to (pppoe0)",
will have an incorrect /32 mask applied to the dynamic address.

MFC after: 3 weeks
Obtained from: OpenBSD
See also: https://ftp.openbsd.org/pub/OpenBSD/patches/5.6/common/007_pfctl.patch.sig
Sponsored by: Rubicon Communications, LLC ("Netgate")
Event: Oslo Hackathon at Modirum

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

pfctl: SCTP can have port numbers

MFC after: 3 weeks
Sponsored by: Orange Business Services
Differential Revision: https://reviews.freebsd.org/D40861

pfctl: match expand_label_addr() prototype to definition

Sponsored by: Rubicon Communications, LLC ("Netgate")

pf: introduce ridentifier and labels to ether rules

Make Ethernet rules more similar to the usual layer 3 rules by also
allowing ridentifier and labels to be set on them.

Reviewed by: kp
Sponsored by: Rubicon Communications, LLC ("Netgate")

pf: backport OpenBSD syntax of "scrub" option for "match" and "pass" rules

Introduce the OpenBSD syntax of "scrub" option for "match" and "pass"
rules and the "set reassemble" flag. The patch is backward-compatible,
pf.conf can be still written in FreeBSD-style.

Obtained from: OpenBSD
MFC after: never
Sponsored by: InnoGames GmbH
Differential Revision: https://reviews.freebsd.org/D38025

pf: drop support for fragment crop|drop-ovl

We removed the code for these modes back in 2015, but converted such
configurations to 'scrub fragment reassemble'. It's been long enough,
drop the backwards compatibility glue too.

Reviewed by: mjg
MFC after: never
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D37460

pf: allow scrub rules without fragment reassemble

scrub rules have defaulted to handling fragments for a long time, but
since we removed "fragment crop" and "fragment drop-ovl" in 64b3b4d611
this has become less obvious and more expensive ("reassemble" being the
more expensive option, even if it's the one the vast majority of users
should be using).

Extend the 'scrub' syntax to allow fragment reassembly to be disabled,
while retaining the other scrub behaviour (e.g. TTL changes, random-id,
..) using 'scrub fragment no reassemble'.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D37459

pf: bridge-to

Allow pf (l2) to be used to redirect ethernet packets to a different
interface.

The intended use case is to send 802.1x challenges out to a side
interface, to enable AT&T links to function with pfSense as a gateway,
rather than the AT&T provided hardware.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D37193

pfctl: fix recrusive printing of ethernet anchors

Similar to the preceding fix for layer three rules, ensure that we
recursively list wildcard anchors for ethernet rules.

MFC after: 3 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D36417

pfctl: fix recrusive printing of anchors

Fix a couple of problems with printing of anchors, in particular recursive
printing, both of inline anchors and when requested explicitly with a '*'
in the anchor.
- Correct recursive printing of wildcard anchors (recurse into child anchors
rather than rules, which don't exist)
- Print multi-part anchor paths correctly (pr6065)
- Fix comments and prevent users from specifying multi-component names for
inline anchors.

tested by phessler
ok henning

Also fix the relevant pfctl test case to reflect the new (and now
correct) behaviour).

MFC after: 3 weeks
Obtained from: OpenBSD (mcbride, f9a568a27c740528301ca3419316c85a9fc7f1de)
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D36416

pfctl: fix FOM_ICMP/POM_STICKYADDRESS clash

pass inet proto icmp icmp-type {unreach}
pass route-to (if0 127.0.0.1/8) sticky-address inet

The wrong struct was being tested. The parser tries to prevent
"sticky-address sticky-address" syntax but was actually cross-rule
enforcing that ICMP filter cannot be before the use of "sticky-address"
in next rule.

MFC after: 2 weeks
Reviewed by: kp
Differential Revision: https://reviews.freebsd.org/D36050

pf: support matching on tags for Ethernet rules

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D35362

pf: allow the use of tables in ethernet rules

Allow tables to be used for the l3 source/destination matching.
This requires taking the PF_RULES read lock.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D34917

pfctl(8): Fix a typo in a comment

- s/steping/stepping/

MFC after: 3 days

pf: ether l3 rules can only use addresses

Disallow the use of tables in ethernet rules. Using tables requires
taking the PF_RULES lock. Moreover, the current table code isn't ready
to deal with ethernet rules.

Disallow their use for now.

Sponsored by: Rubicon Communications, LLC ("Netgate")

pf: support basic L3 filtering in the Ethernet rules

Allow filtering based on the source or destination IP/IPv6 address in
the Ethernet layer rules.

Reviewed by: pauamma_gundo.com (man), debdrup (man)
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D34482

pfctl: print ethernet rules when called with '-n'

Just as pfctl already does for other rules we print the ethernet rules
we would have loaded if '-n' is specified.

Sponsored by: Rubicon Communications, LLC ("Netgate")

pf: support masking mac addresses

When filtering Ethernet packets allow rules to specify a mac address
with a mask. This indicates which bits of the specified address are
significant. This allows users to do things like filter based on device
manufacturer.

Sponsored by: Rubicon Communications, LLC ("Netgate")

pf: add anchor support for ether rules

Support anchors in ether rules.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D32482

pfctl: support lists of mac addresses

Teach the 'ether' rules to accept { mac1, mac2, ... } lists, similar to
the lists of interfaces or IP addresses we already supported for layer 3
filtering.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D32481

pf: support dummynet on L2 rules

Allow packets to be tagged with dummynet information. Note that we do
not apply dummynet shaping on the L2 traffic, but instead mark it for
dummynet processing in the L3 code. This is the same approach as we take
for ALTQ.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D32222

pfctl: Print Ethernet rules

Extent pfctl to be able to read configured Ethernet filtering rules from
the kernel and print them.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D31738

pfctl: improve error reporting for routehost

If an invalid (i.e. overly long) interface name is specified error out
immediately, rather than in expand_rule() so we point at the incorrect
line.

PR: 260958
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D34008

pf: Introduce ridentifier

Allow users to set a number on rules which will be exposed as part of
the pflog header.
The intent behind this is to allow users to correlate rules across
updates (remember that pf rules continue to exist and match existing
states, even if they're removed from the active ruleset) and pflog.

Obtained from: pfSense
MFC after: 3 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D32750

pfctl: delay label macro expansion until after rule optimisation

We used to expand the $nr macro in labels into the rule number prior to
the optimisation step. This would occasionally produce incorrect rule
numbers in the labels.

Delay all macro expansion until after the optimisation step to ensure
that we expand the correct values.

MFC after: 1 week
Reported by: Özkan KIRIK <ozkan.kirik@gmail.com>
Differential Revision: https://reviews.freebsd.org/D32488

pfctl: Remove unused variable

MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")

pfctl: userspace adaptive syncookies configration

Hook up the userspace bits to configure syncookies in adaptive mode.

MFC after: 1 week
Sponsored by: Modirum MDPay
Differential Revision: https://reviews.freebsd.org/D32136

pf: support dummynet

Allow pf to use dummynet pipes and queues.

We re-use the currently unused IPFW_IS_DUMMYNET flag to allow dummynet
to tell us that a packet is being re-injected after being delayed. This
is needed to avoid endlessly looping the packet between pf and dummynet.

MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D31904

pfctl: syncookie configuration

pfctl and libpfctl code required to enable/disable the syncookie
feature.

MFC after: 1 week
Sponsored by: Modirum MDPay
Differential Revision: https://reviews.freebsd.org/D31140

pf: match keyword support

Support the 'match' keyword.
Note that support is limited to adding queuing information, so without
ALTQ support in the kernel setting match rules is pointless.

For the avoidance of doubt: this is NOT full support for the match
keyword as found in OpenBSD's pf. That could potentially be built on top
of this, but this commit is NOT that.

MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D31115

pfctl: cache getprotobynumber results

As for example pfctl -ss keeps calling it, it saves a lot of overhead
from elided parsing of /etc/nsswitch.conf and /etc/protocols.

Sample result when running a pre-nvlist binary with nfs root and dumping
7 mln states:
before: 24.817u 62.993s 1:28.52 99.1%
after: 8.064u 1.117s 0:18.87 48.5%

Idea by Jim Thompson

Reviewed by: kp
Sponsored by: Rubicon Communications, LLC ("Netgate")

pfctl: Ensure parent queue is configured for FAIRQ

We failed to account for the FAIRQ scheduler in expand_altq(), which led
it to be set up without its parent queue.

MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D30347

pf: Allow multiple labels to be set on a rule

Allow up to 5 labels to be set on each rule.
This offers more flexibility in using labels. For example, it replaces
the customer 'schedule' keyword used by pfSense to terminate states
according to a schedule.

Reviewed by: glebius
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D29936

pf: Optionally attempt to preserve rule counter values across ruleset updates

Usually rule counters are reset to zero on every update of the ruleset.
With keepcounters set pf will attempt to find matching rules between old
and new rulesets and preserve the rule counters.

MFC after: 4 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D29780

pf: Implement the NAT source port selection of MAP-E Customer Edge

MAP-E (RFC 7597) requires special care for selecting source ports
in NAT operation on the Customer Edge because a part of bits of the port
numbers are used by the Border Relay to distinguish another side of the
IPv4-over-IPv6 tunnel.

PR: 254577
Reviewed by: kp
Differential Revision: https://reviews.freebsd.org/D29468

libpfctl: Switch to pfctl_rule

Stop using the kernel's struct pf_rule, switch to libpfctl's pfctl_rule.
Now that we use nvlists to communicate with the kernel these structures
can be fully decoupled.

Reviewed by: glebius
MFC after: 4 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D29644

pfctl: Move ioctl abstraction functions into libpfctl

Introduce a library to wrap the pf ioctl interface.

MFC after: 4 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D29562

pfctl: Add missing 'va' code point name

Add the 'va' (voice-admit, RFC5865) symbolic name.

Reviewed by: rgrimes, gbe (man page)
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D29069

pfctl: Point users to net.pf.request_maxcount if large requests are rejected

The kernel will reject very large tables to avoid resource exhaustion
attacks. Some users run into this limit with legitimate table
configurations.

The error message in this case was not very clear:

pf.conf:1: cannot define table nets: Invalid argument
pfctl: Syntax error in config file: pf rules not loaded

If a table definition fails we now check the request_maxcount sysctl,
and if we've tried to create more than that point the user at
net.pf.request_maxcount:

pf.conf:1: cannot define table nets: too many elements.
Consider increasing net.pf.request_maxcount.
pfctl: Syntax error in config file: pf rules not loaded

PR: 235076
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D18909

pfctl: Do not allow whitespace in macro names

i.e. "this is" = "a variable" is not valid. It was accepted by the
parser, but the variable could not be used afterwards.

Obtained from: OpenBSD

pfctl: Fix line numbers when \ is used inside ""

PR: 201520
Obtained from: OpenBSD
MFC after: 2 weeks

pfctl: Dup strings

When we set the ifname we have to copy the string, rather than just keep
the pointer.

PR: 231323
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D17507

Extended pf(4) ioctl interface and pfctl(8) to allow bandwidths of
2^32 bps or greater to be used. Prior to this, bandwidth parameters
would simply wrap at the 2^32 boundary. The computations in the HFSC
scheduler and token bucket regulator have been modified to operate
correctly up to at least 100 Gbps. No other algorithms have been
examined or modified for correct operation above 2^32 bps (some may
have existing computation resolution or overflow issues at rates below
that threshold). pfctl(8) will now limit non-HFSC bandwidth
parameters to 2^32 - 1 before passing them to the kernel.

The extensions to the pf(4) ioctl interface have been made in a
backwards-compatible way by versioning affected data structures,
supporting all versions in the kernel, and implementing macros that
will cause existing code that consumes that interface to use version 0
without source modifications. If version 0 consumers of the interface
are used against a new kernel that has had bandwidth parameters of
2^32 or greater configured by updated tools, such bandwidth parameters
will be reported as 2^32 - 1 bps by those old consumers.

All in-tree consumers of the pf(4) interface have been updated. To
update out-of-tree consumers to the latest version of the interface,
define PFIOC_USE_LATEST ahead of any includes and use the code of
pfctl(8) as a guide for the ioctls of interest.

PR: 211730
Reviewed by: jmallett, kp, loos
MFC after: 2 weeks
Relnotes: yes
Sponsored by: RG Nets
Differential Revision: https://reviews.freebsd.org/D16782

pf: Support "return" statements in passing rules when they fail.

Normally pf rules are expected to do one of two things: pass the traffic or
block it. Blocking can be silent - "drop", or loud - "return", "return-rst",
"return-icmp". Yet there is a 3rd category of traffic passing through pf:
Packets matching a "pass" rule but when applying the rule fails. This happens
when redirection table is empty or when src node or state creation fails. Such
rules always fail silently without notifying the sender.

Allow users to configure this behaviour too, so that pf returns an error packet
in these cases.

PR: 226850
Submitted by: Kajetan Staszkiewicz <vegeta tuxpowered.net>
MFC after: 1 week
Sponsored by: InnoGames GmbH

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

pfctl: teach route-to to deal with interfaces with multiple addresses

The route_host parsing code set the interface name, but only for the first
node_host in the list. If that one happened to be the inet6 address and the
rule wanted an inet address it'd get removed by remove_invalid_hosts() later
on, and we'd have no interface name.

We must set the interface name for all node_host entries in the list, not just
the first one.

PR: 223208
MFC after: 2 weeks

Fix the parsing of NPt binat rules.

In this specific case the src address can be set to any, which was not
accepted prior to this commit.

pfSense bug report: https://redmine.pfsense.org/issues/6985
Reviewed by: kp
Obtained from: pfSense
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC (Netgate)

pf: port extended DSCP support from OpenBSD

Ignore the ECN bits on 'tos' and 'set-tos' and allow to use
DCSP names instead of having to embed their TOS equivalents
as plain numbers.

Obtained from: OpenBSD
Sponsored by: OPNsense
Differential Revision: https://reviews.freebsd.org/D8165

pf: remove fastroute tag

The tag fastroute came from ipf and was removed in OpenBSD in 2011. The code
allows to skip the in pfil hooks and completely removes the out pfil invoke,
albeit looking up a route that the IP stack will likely find on its own.
The code between IPv4 and IPv6 is also inconsistent and marked as "XXX"
for years.

Submitted by: Franco Fichtner <franco@opnsense.org>
Differential Revision: https://reviews.freebsd.org/D8058

Fix a regression in pf.conf while parsing the 'interval' keyword.

The bug was introduced by r287009.

PR: 210924
Submitted by: kp@
Sponsored by: Rubicon Communications (Netgate)
Pointy hat to: loos

pfctl: Make most global variables static.

This will make it easier to link as a library.

Submitted by: Christian Mauderer <christian.mauderer@embedded-brains.de>

pfctl: Allow TOS bits to be cleared

TOS value 0 is valid, so use 256 as an invalid value rather than zero.
This allows users to enforce TOS == 0 with pf.

Reported by: Radek Krejča <radek.krejca@starnet.cz>

pf: Filter on and set vlan PCP values

Adopt the OpenBSD syntax for setting and filtering on VLAN PCP values. This
introduces two new keywords: 'set prio' to set the PCP value, and 'prio' to
filter on it.

Reviewed by: allanjude, araujo
Approved by: re (gjb)
Obtained from: OpenBSD (mostly)
Differential Revision: https://reviews.freebsd.org/D6786

sbin: minor spelling fixes.

No functional change.

Make pfctl(8) more flexible when parsing bandwidth values.

This is the current behaviour in OpenBSD and a similar patch exist in
pfSense too.

Obtained from: OpenBSD (partly - rev. 1.625)
MFC after: 2 weeks
Sponsored by: Rubicon Communications (Netgate)

pf: Remove support for 'scrub fragment crop|drop-ovl'

The crop/drop-ovl fragment scrub modes are not very useful and likely to confuse
users into making poor choices.
It's also a fairly large amount of complex code, so just remove the support
altogether.

Users who have 'scrub fragment crop|drop-ovl' in their pf configuration will be
implicitly converted to 'scrub fragment reassemble'.

Reviewed by: gnn, eri
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D3466

Add ALTQ(9) support for the CoDel algorithm.

CoDel is a parameterless queue discipline that handles variable bandwidth
and RTT.

It can be used as the single queue discipline on an interface or as a sub
discipline of existing queue disciplines such as PRIQ, CBQ, HFSC, FAIRQ.

Differential Revision: https://reviews.freebsd.org/D3272
Reviewd by: rpaulo, gnn (previous version)
Obtained from: pfSense
Sponsored by: Rubicon Communications (Netgate)

ALTQ FAIRQ discipline import from DragonFLY

Differential Revision: https://reviews.freebsd.org/D2847
Reviewed by: glebius, wblock(manpage)
Approved by: gnn(mentor)
Obtained from: pfSense
Sponsored by: Netgate

Move ALTQ from contrib to net/altq. The ALTQ code is for many years
discontinued by its initial authors. In FreeBSD the code was already
slightly edited during the pf(4) SMP project. It is about to be edited
more in the projects/ifnet. Moving out of contrib also allows to remove
several hacks to the make glue.

Reviewed by: net@

o Create directory sys/netpfil, where all packet filters should
reside, and move there ipfw(4) and pf(4).

o Move most modified parts of pf out of contrib.

Actual movements:

sys/contrib/pf/net/*.c -> sys/netpfil/pf/
sys/contrib/pf/net/*.h -> sys/net/
contrib/pf/pfctl/*.c -> sbin/pfctl
contrib/pf/pfctl/*.h -> sbin/pfctl
contrib/pf/pfctl/pfctl.8 -> sbin/pfctl
contrib/pf/pfctl/*.4 -> share/man/man4
contrib/pf/pfctl/*.5 -> share/man/man5

sys/netinet/ipfw -> sys/netpfil/ipfw

The arguable movement is pf/net/*.h -> sys/net. There are
future plans to refactor pf includes, so I decided not to
break things twice.

Not modified bits of pf left in contrib: authpf, ftp-proxy,
tftp-proxy, pflogd.

The ipfw(4) movement is planned to be merged to stable/9,
to make head and stable match.

Discussed with: bz, luigi