pf: allow pflow to be activated per rule

Only generate ipfix/netflow reports (through pflow) for the rules where
this is enabled. Reports can also be enabled globally through 'set
state-default pflow'.

Obtained from: OpenBSD
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D43108

sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

pf: expose more syncookie state information to userspace

Allow userspace to retrieve low and high water marks, as well as the
current number of half open states.

MFC after: 1 week
Sponsored by: Modirum MDPay

pf: support SCTP-specific timeouts

Allow SCTP state timeouts to be configured independently from TCP state
timeouts.

Reviewed by: tuexen
MFC after: 1 week
Sponsored by: Orange Business Services
Differential Revision: https://reviews.freebsd.org/D42393

pfctl: fix incorrect mask on dynamic address

A PF rule using an IPv4 address followed by an IPv6 address and then a
dynamic address, e.g. "pass from {192.0.2.1 2001:db8::1} to (pppoe0)",
will have an incorrect /32 mask applied to the dynamic address.

MFC after: 3 weeks
Obtained from: OpenBSD
See also: https://ftp.openbsd.org/pub/OpenBSD/patches/5.6/common/007_pfctl.patch.sig
Sponsored by: Rubicon Communications, LLC ("Netgate")
Event: Oslo Hackathon at Modirum

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

pf: introduce ridentifier and labels to ether rules

Make Ethernet rules more similar to the usual layer 3 rules by also
allowing ridentifier and labels to be set on them.

Reviewed by: kp
Sponsored by: Rubicon Communications, LLC ("Netgate")

pfctl: match geticmpcodeby*()/print_fromto() prototypes to definitions

The definitions use sa_family_t, not u_int8_t.

Sponsored by: Rubicon Communications, LLC ("Netgate")

pf: backport OpenBSD syntax of "scrub" option for "match" and "pass" rules

Introduce the OpenBSD syntax of "scrub" option for "match" and "pass"
rules and the "set reassemble" flag. The patch is backward-compatible,
pf.conf can be still written in FreeBSD-style.

Obtained from: OpenBSD
MFC after: never
Sponsored by: InnoGames GmbH
Differential Revision: https://reviews.freebsd.org/D38025

pf: allow scrub rules without fragment reassemble

scrub rules have defaulted to handling fragments for a long time, but
since we removed "fragment crop" and "fragment drop-ovl" in 64b3b4d611
this has become less obvious and more expensive ("reassemble" being the
more expensive option, even if it's the one the vast majority of users
should be using).

Extend the 'scrub' syntax to allow fragment reassembly to be disabled,
while retaining the other scrub behaviour (e.g. TTL changes, random-id,
..) using 'scrub fragment no reassemble'.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D37459

pf: bridge-to

Allow pf (l2) to be used to redirect ethernet packets to a different
interface.

The intended use case is to send 802.1x challenges out to a side
interface, to enable AT&T links to function with pfSense as a gateway,
rather than the AT&T provided hardware.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D37193

pf: expose syncookie active/inactive status

When syncookies are in adaptive mode they may be active or inactive.
Expose this status to users.

Suggested by: Guido van Rooij
Sponsored by: Rubicon Communications, LLC ("Netgate")

pfctl: fix recrusive printing of anchors

Fix a couple of problems with printing of anchors, in particular recursive
printing, both of inline anchors and when requested explicitly with a '*'
in the anchor.
- Correct recursive printing of wildcard anchors (recurse into child anchors
rather than rules, which don't exist)
- Print multi-part anchor paths correctly (pr6065)
- Fix comments and prevent users from specifying multi-component names for
inline anchors.

tested by phessler
ok henning

Also fix the relevant pfctl test case to reflect the new (and now
correct) behaviour).

MFC after: 3 weeks
Obtained from: OpenBSD (mcbride, f9a568a27c740528301ca3419316c85a9fc7f1de)
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D36416

pf: stop resolving hosts as dns that use ":" modifier

When the interface does not exist avoid passing host with special pf
modifiers to DNS resolution as they come up empty anyway.

Reviewed by: kp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D35429

pf: support matching on tags for Ethernet rules

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D35362

pfctl: always print 'l3' source/destination

While the kernel only performs the L3 check for
ETHERTYPE_IP/ETHERTYPE_IP6 we should always print the source and
destination addresses.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D34918

pf: fixup match rules

Ensure that we can set and print match rules in ethernet rules.

Sponsored by: Rubicon Communications, LLC ("Netgate")

pf: support basic L3 filtering in the Ethernet rules

Allow filtering based on the source or destination IP/IPv6 address in
the Ethernet layer rules.

Reviewed by: pauamma_gundo.com (man), debdrup (man)
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D34482

pf: support masking mac addresses

When filtering Ethernet packets allow rules to specify a mac address
with a mask. This indicates which bits of the specified address are
significant. This allows users to do things like filter based on device
manufacturer.

Sponsored by: Rubicon Communications, LLC ("Netgate")

pf: add anchor support for ether rules

Support anchors in ether rules.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D32482

pf: support dummynet on L2 rules

Allow packets to be tagged with dummynet information. Note that we do
not apply dummynet shaping on the L2 traffic, but instead mark it for
dummynet processing in the L3 code. This is the same approach as we take
for ALTQ.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D32222

pfctl: Don't print (ether) to / from if they're not set

If we're not filtering on a specific MAC address don't print it at all,
rather than showing an all-zero address.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D31749

pfctl: Print Ethernet rules

Extent pfctl to be able to read configured Ethernet filtering rules from
the kernel and print them.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D31738

libpfctl: fix creatorid endianness

We provide the hostid (which is the state creatorid) to the kernel as a
big endian number (see pfctl/pfctl.c pfctl_set_hostid()), so convert it
back to system endianness when we get it from the kernel.

This avoids a confusing mismatch between the value the user configures
and the value displayed in the state.

MFC after: 3 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D33989

pf: Introduce ridentifier

Allow users to set a number on rules which will be exposed as part of
the pflog header.
The intent behind this is to allow users to correlate rules across
updates (remember that pf rules continue to exist and match existing
states, even if they're removed from the active ruleset) and pflog.

Obtained from: pfSense
MFC after: 3 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D32750

pfctl: userspace adaptive syncookies configration

Hook up the userspace bits to configure syncookies in adaptive mode.

MFC after: 1 week
Sponsored by: Modirum MDPay
Differential Revision: https://reviews.freebsd.org/D32136

pf: support dummynet

Allow pf to use dummynet pipes and queues.

We re-use the currently unused IPFW_IS_DUMMYNET flag to allow dummynet
to tell us that a packet is being re-injected after being delayed. This
is needed to avoid endlessly looping the packet between pf and dummynet.

MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D31904

pfctl: print counters in decimal

795d78a46713 pfctl: Don't use PRIu64 mistakenly changed these to be
printed as hexadecimal numbers.

Reported by: Florian Smeets
MFC after: 4 days
Sponsored by: Rubicon Communications, LLC ("Netgate")

pfctl: Don't use PRIu64

Rather than PRIu64 we can just treat the data as uintmax_t, and use %jx
instead.

MFC after: 1 week
Suggested by: kib
Sponsored by: Rubicon Communications, LLC ("Netgate")

Fix -Wformat errors in pfctl on 32-bit architectures

Use PRIu64 to printf(3) uint64_t quantities, otherwise this will result
in "error: format specifies type 'unsigned long' but the argument has
type 'uint64_t' (aka 'unsigned long long') [-Werror,-Wformat]" on 32-bit
architectures.

Fixes: 80078d9d38fd
MFC after: 1 week

pfctl: use libpfctl to retrieve pf status

Rather than call DIOCGETSTATUS ourselves use the new libpfctl functions.

MFC after: 1 week
Sponsored by: Modirum MDPay
Differential Revision: https://reviews.freebsd.org/D31697

pfctl: syncookie configuration

pfctl and libpfctl code required to enable/disable the syncookie
feature.

MFC after: 1 week
Sponsored by: Modirum MDPay
Differential Revision: https://reviews.freebsd.org/D31140

pf: match keyword support

Support the 'match' keyword.
Note that support is limited to adding queuing information, so without
ALTQ support in the kernel setting match rules is pointless.

For the avoidance of doubt: this is NOT full support for the match
keyword as found in OpenBSD's pf. That could potentially be built on top
of this, but this commit is NOT that.

MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D31115

pfctl: cache getprotobynumber results

As for example pfctl -ss keeps calling it, it saves a lot of overhead
from elided parsing of /etc/nsswitch.conf and /etc/protocols.

Sample result when running a pre-nvlist binary with nfs root and dumping
7 mln states:
before: 24.817u 62.993s 1:28.52 99.1%
after: 8.064u 1.117s 0:18.87 48.5%

Idea by Jim Thompson

Reviewed by: kp
Sponsored by: Rubicon Communications, LLC ("Netgate")

pfctl: Revert "pfctl: Another set skip <group> fix"

This reverts commit 0c156a3c32cd0d9168570da5686ddc96abcbbc5a.
This fix broke using '<ifgroup>:network' in tables.

MFC after: 1 week

pf: Allow multiple labels to be set on a rule

Allow up to 5 labels to be set on each rule.
This offers more flexibility in using labels. For example, it replaces
the customer 'schedule' keyword used by pfSense to terminate states
according to a schedule.

Reviewed by: glebius
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D29936

pf: Implement the NAT source port selection of MAP-E Customer Edge

MAP-E (RFC 7597) requires special care for selecting source ports
in NAT operation on the Customer Edge because a part of bits of the port
numbers are used by the Border Relay to distinguish another side of the
IPv4-over-IPv6 tunnel.

PR: 254577
Reviewed by: kp
Differential Revision: https://reviews.freebsd.org/D29468

pfctl, libpfctl: introduce pfctl_pool

Introduce pfctl_pool to be able to extend the pool part of the pf rule
without breaking the ABI.

Reviewed by: kp
MFC after: 4 weeks
Differential Revision: https://reviews.freebsd.org/D29721

libpfctl: Switch to pfctl_rule

Stop using the kernel's struct pf_rule, switch to libpfctl's pfctl_rule.
Now that we use nvlists to communicate with the kernel these structures
can be fully decoupled.

Reviewed by: glebius
MFC after: 4 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D29644

pfctl: Another set skip <group> fix

When retrieving the list of group members we cannot simply use
ifa_lookup(), because it expects the interface to have an IP (v4 or v6)
address. This means that interfaces with no address are not found.
This presents as interfacing being alternately marked as skip and not
whenever the rules are re-loaded.

Happily we only need to fix ifa_grouplookup(). Teach it to also accept
AF_LINK (i.e. interface) node_hosts.

PR: 250994
MFC after: 3 days

pfctl: Call ifa_load() before ifa_grouplookup()

ifa_grouplookup() uses the data loaded in ifa_load() (through is_a_group()), so
we must call ifa_load() before we can rely on any of the data it populates.

Submitted by: Nick Rogers
MFC after: 1 week
Sponsored by: RG Nets

pfctl: Remove unused variable

Submitted by: Nick Rogers
MFC after: 1 week
Sponsored by: RG Nets

pfctl: improve rule load times with thousands of interfaces

r343287 / D18759 introduced ifa_add_groups_to_map() which is now run by
ifa_load/ifa_lookup/host_if. When loading an anchor or ruleset via pfctl that
does NOT contain ifnames as hosts, host() still ends up iterating all
interfaces twice, grabbing SIOCGIFGROUP ioctl twice for each. This adds an
unnecessary amount of time on systems with thousands or tens of thousands of
interfaces.

Prioritize the IPv4/6 check over the interface name lookup, which skips loading
the iftab and iterating all interfaces when the configuration does not contain
interface names.

Submitted by: Nick Rogers
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D24100

pfctl: Fix ifa_grouplookup()

Setting the length of the request got lost in r343287, which means SIOCGIFGMEMB
gives us the required length, but does not copy the names of the group members.
As a result we don't get a correct list of group members, and 'set skip on
<ifgroup>' broke.

This produced all sorts of very unexpected results, because we would end up
applying 'set skip' to unexpected interfaces.

X-MFC-with: r343287

Reduce pf.conf parsing cost for configs that define N queues from O(N^2) to O(N)

The number of syscalls made during parsing of any config that
defines tables is also reduced, and incorrect warnings that HFSC
parent queue bandwidths were smaller than the sum of their child
bandwidths have been fixed.

Reviewed by: kp
MFC after: 1 week
Sponsored by: RG Nets
Differential Revision: https://reviews.freebsd.org/D18759

pfctl: Populate ifname in ifa_lookup()

pfctl_adjust_skip_ifaces() relies on this name.

MFC after: 2 weeks

pf: Make ':0' ignore link-local v6 addresses too

When users mark an interface to not use aliases they likely also don't
want to use the link-local v6 address there.

PR: 201695
Submitted by: Russell Yount <Russell.Yount AT gmail.com>
Differential Revision: https://reviews.freebsd.org/D17633

pf: Return non-zero from 'status' if pf is not enabled

In the pf rc.d script the output of `/etc/rc.d/pf status` or `/etc/rc.d/pf
onestatus` always provided an exit status of zero. This made it fiddly to
programmatically determine if pf was running or not.

Return a non-zero status if the pf module is not loaded, extend pfctl to have
an option to return an error status if pf is not enabled.

PR: 228632
Submitted by: James Park-Watt <jimmypw AT gmail.com>
MFC after: 1 week

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

pf: remove fastroute tag

The tag fastroute came from ipf and was removed in OpenBSD in 2011. The code
allows to skip the in pfil hooks and completely removes the out pfil invoke,
albeit looking up a route that the IP stack will likely find on its own.
The code between IPv4 and IPv6 is also inconsistent and marked as "XXX"
for years.

Submitted by: Franco Fichtner <franco@opnsense.org>
Differential Revision: https://reviews.freebsd.org/D8058

pfctl: Make most global variables static.

This will make it easier to link as a library.

Submitted by: Christian Mauderer <christian.mauderer@embedded-brains.de>

pfctl: Use const where possible.

This adds const qualifiers where it is possible.

Submitted by: Christian Mauderer <christian.mauderer@embedded-brains.de>

pf: Filter on and set vlan PCP values

Adopt the OpenBSD syntax for setting and filtering on VLAN PCP values. This
introduces two new keywords: 'set prio' to set the PCP value, and 'prio' to
filter on it.

Reviewed by: allanjude, araujo
Approved by: re (gjb)
Obtained from: OpenBSD (mostly)
Differential Revision: https://reviews.freebsd.org/D6786

Use nitems() from sys/param.h.

MFC after: 2 weeks.

pf: Remove support for 'scrub fragment crop|drop-ovl'

The crop/drop-ovl fragment scrub modes are not very useful and likely to confuse
users into making poor choices.
It's also a fairly large amount of complex code, so just remove the support
altogether.

Users who have 'scrub fragment crop|drop-ovl' in their pf configuration will be
implicitly converted to 'scrub fragment reassemble'.

Reviewed by: gnn, eri
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D3466

Use feature_present(3) to determine whether to open an INET or an
INET6 socket when needed to allow pfctl to work on noinet and noinet6
kernels (and try to provide a fallback using AF_LINK as best effort).
Adjust the Makefile to also respect relevant src.conf(5) options
for compile time decisions on INET and INET6 support.

Reviewed by: glebius (no objections)
MFC after: 1 week

o Create directory sys/netpfil, where all packet filters should
reside, and move there ipfw(4) and pf(4).

o Move most modified parts of pf out of contrib.

Actual movements:

sys/contrib/pf/net/*.c -> sys/netpfil/pf/
sys/contrib/pf/net/*.h -> sys/net/
contrib/pf/pfctl/*.c -> sbin/pfctl
contrib/pf/pfctl/*.h -> sbin/pfctl
contrib/pf/pfctl/pfctl.8 -> sbin/pfctl
contrib/pf/pfctl/*.4 -> share/man/man4
contrib/pf/pfctl/*.5 -> share/man/man5

sys/netinet/ipfw -> sys/netpfil/ipfw

The arguable movement is pf/net/*.h -> sys/net. There are
future plans to refactor pf includes, so I decided not to
break things twice.

Not modified bits of pf left in contrib: authpf, ftp-proxy,
tftp-proxy, pflogd.

The ipfw(4) movement is planned to be merged to stable/9,
to make head and stable match.

Discussed with: bz, luigi