mlx5: Introduce new destination type TABLE_TYPE

This new destination type supports flow transition between different
table types, e.g. from NIC_RX to RDMA_RX or from RDMA_TX to NIC_TX.

In addition add driver support to be able to query the capability for
this new destination type.

Signed-off-by: Patrisious Haddad <phaddad@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

mlx5: add fs_counters

Signed-off-by: Raed Salem <raeds@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

mlx5: Add packet reformat support to flow rules

Allow attaching a packet reformat action to a flow rule.

Signed-off-by: Mark Bloch <mbloch@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

mlx5: Add modify header support to flow rules

Allow attaching a modify header to a flow rule.

Signed-off-by: Mark Bloch <mbloch@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

mlx5: Add packet reformat allocation support

Add support to allocating a packet reformat context.

Signed-off-by: Mark Bloch <mbloch@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

mlx5: Add modify header support

Add support to allocating a modify header context.

Signed-off-by: Mark Bloch <mbloch@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

mlx5: Implement diagostic counters as sysctl(8) nodes.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5en: Implement TLS RX support.

TLS RX support is modeled after TLS TX support. The basic structures and layouts
are almost identical, except that the send tag created filters RX traffic and
not TX traffic.

The TLS RX tag keeps track of past TLS records up to a certain limit,
approximately 1 Gbyte of TCP data. TLS records of same length are joined
into a single database record.

Regularly the HW is queried for TLS RX progress information. The TCP sequence
number gotten from the HW is then matches against the database of TLS TCP
sequence number records and lengths. If a match is found a static params WQE
is queued on the IQ and the hardware should immediately resume decrypting TLS
data until the next non-sequential TCP packet arrives.

Offloading TLS RX data is supported for untagged, prio-tagged, and
regular VLAN traffic.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5: Implement mlx5_nic_vport_update_local_lb()

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5: Implement flow steering helper functions for TCP sockets.

This change adds convenience functions to setup a flow steering rule based on
a TCP socket. The helper function gets all the address information from the
socket and returns a steering rule, to be used with HW TLS RX offload.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5/mlx5en: Add SQ remap support

Add support to map an SQ to a specific schedule queue using a
special WQE as performance enhancement.

SQ remap operation is handled by a privileged internal queue, IQ,
and the mapping is enabled from one rate to another.

The transition from paced to non-paced should however always go
through FW.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5: Properly define the reg_umr_sq networking offload capability bit.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5en: Make the receive packet indirection table, RQT, static instead of dynamic.

Allocate the RQT once, pointing all initial entries to the drop RQN.
When opening the channels simplify modify the RQT, directing all traffic
to the new RQNs. Similarly when closing the channels point all RQT entries
back to the so-called drop RQN.

MFC after: 1 week
Sponsored by: NVIDIA Networking

ibcore: Kernel space update based on Linux 5.7-rc1.

Overview:

This is the first stage of a RDMA stack upgrade introducing kernel
changes only based on Linux 5.7-rc1.

This patch is based on about four main areas of work:
- Update of the IB uobjects system:
- The memory holding so-called AH, CQ, PD, SRQ and UCONTEXT objects
is now managed by ibcore. This also require some changes in the
kernel verbs API. The updated verbs changes are typically about
initialize and deinitialize objects, and remove allocation and
free of memory.

- Update of the uverbs IOCTL framework:
- The parsing and handling of user-space commands has been
completely refactored to integrate with the updated IB uobjects
system.

- Various changes and updates to the generic uverbs interfaces in
device drivers including the new uAPI surface.

- The mlx5_ib_devx.c in mlx5ib and related mlx5 core changes.

Dependencies:

- The mlx4ib driver code has been updated with the minimum changes
needed.

- The mlx5ib driver code has been updated with the minimum changes
needed including DV support.

Compatibility:

- All user-space facing APIs are backwards compatible after this
change.

- All kernel-space facing RDMA APIs are backwards compatible after
this change, with exception of ib_create_ah() and ib_destroy_ah()
which takes a new flag.

- The "ib_device_ops" structure exist, but only contains the driver ID
and some structure sizes.

Differences from Linux:

- Infiniband drivers must use the INIT_IB_DEVICE_OPS() macro to set
the sizes needed for allocating various IB objects, when adding
IB device instances.

Security:

- PRIV_NET_RAW is needed to use raw ethernet transmit features.
- PRIV_DRIVER is needed to use other privileged operations.

Based on upstream Linux, Torvalds (5.7-rc1):
8632e9b5645bbc2331d21d892b0d6961c1a08429

MFC after: 1 week
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D31149
Sponsored by: NVIDIA Networking

mlx5: Fix for uninitialized "uid" field.

Make sure the "uid" field gets properly set when destroying DCT and QP
objects by making a copy of the field when creating such objects.

MFC after: 1 week
Reviewed by: kib
Sponsored by: Mellanox Technologies // NVIDIA Networking

mlx5en: Configure relaxed PCI read and write ordering for ethernet.

This may improve performance in some configurations.

MFC after: 1 week
Reviewed by: kib
Sponsored by: Mellanox Technologies // NVIDIA Networking

mlx5: Add new timestamp mode bits.

These fields declare which timestamp mode is supported
by the device per RQ/SQ/QP.

In addition add the ts_format field to the select the mode
for RQ/SQ/QP.

Linux commit:
a6a217dddcd544f6b75f0e2a60b6e84c1d494b7e

MFC after: 1 week
Reviewed by: kib
Sponsored by: Mellanox Technologies // NVIDIA Networking

mlx5: Fix PBMC register mapping

Import Linux commit 534b1204ca4694db1093b15cf3e79a99fcb6a6da

Add reserved mapping to cover all the register in order to avoid setting
arbitrary values to newer FW which implements the reserved fields.

Reviewed by: hselasky
Sponsored by: Mellanox Technologies // NVIDIA Networking
MFC after: 1 week

mlx5: Fix PPLM register mapping

Import Linux commit ce28f0fd670ddffcd564ce7119bdefbaf08f02d3:
Add reserved mapping to cover all the register in order to avoid
setting arbitrary values to newer FW which implements the reserved
fields.

Taken from: https://patches.linaro.org/patch/417255/
Reviewed by: hselasky
Sponsored by: Mellanox Technologies // NVIDIA Networking
MFC after: 1 week

Update user access region, UAR, APIs in the core in mlx5core.

This change include several changes as listed below all related to UAR.
UAR is a special PCI memory area where the so-called doorbell register and
blue flame register live. Blue flame is a feature for sending small packets
more efficiently via a PCI memory page, instead of using PCI DMA.

- All structures and functions named xxx_uuars were renamed into xxx_bfreg.
- Remove partially implemented Blueflame support from mlx5en(4) and mlx5ib.
- Implement blue flame register allocator.
- Use blue flame register allocator in mlx5ib.
- A common UAR page is now allocated by the core to support doorbell register
writes for all of mlx5en and mlx5ib, instead of allocating one UAR per
sendqueue.
- Add support for DEVX query UAR.
- Add support for 4K UAR for libmlx5.

Linux commits:
7c043e908a74ae0a935037cdd984d0cb89b2b970
2f5ff26478adaff5ed9b7ad4079d6a710b5f27e7
0b80c14f009758cefeed0edff4f9141957964211
30aa60b3bd12bd79b5324b7b595bd3446ab24b52
5fe9dec0d045437e48f112b8fa705197bd7bc3c0
0118717583cda6f4f36092853ad0345e8150b286
a6d51b68611e98f05042ada662aed5dbe3279c1e

MFC after: 1 week
Sponsored by: Mellanox Technologies // NVIDIA Networking

mlx5_core: add mlx5_query_pddr().

And use it in mlx5_query_pddr_range_info() instead of direct register
access.

Sponsored by: Mellanox Technologies - Nvidia
MFC after: 1 week

mlx5_core: Import PDDR register definitions

PDDR (Port Diagnostics Database Register) is used to read the physical
layer debug database, which contains helpful troubleshooting information
regarding the state of the link.

PDDR register can only be queried when PCAM register reports it as
supported in its register mask. A new helper macro was added to
the MLX5_CAP_* infrastructure in order to access this mask.

Sponsored by: Mellanox Technologies - Nvidia
MFC after: 1 week

Sync with Linux packet pacing enhancements in mlx5en(4).

Linux commit:
05d3ac978ed25b753bfe34fe76c50c31ee506a82

MFC after: 1 week
Sponsored by: Mellanox Technologies

Update the TLS capability bit after recent PRM changes in mlx5en(4).

A CX6-DX firmware version equal to or newer than 12.27.0372 is
now required.

Sponsored by: Mellanox Technologies

Add support for reading temperature in mlx5en(4).

MFC after: 1 week
Sponsored by: Mellanox Technologies

Add basic support for TCP/IP based hardware TLS offload to mlx5core.

The hardware offload is primarily targeted for TLS v1.2 and v1.3,
using AES 128/256 bit pre-shared keys. This patch adds all the needed
hardware structures, capabilites and firmware commands.

Sponsored by: Mellanox Technologies

Add sysctl(8) to get and set forward error correction, FEC, configuration
in mlx5en(4).

MFC after: 3 days
Sponsored by: Mellanox Technologies

Update definitons for PPTB and PBMC registers layouts in mlx5core.

Submitted by: kib@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Define MLX5_VSC_DOMAIN_SCAN_CRSPACE.

Submitted by: kib@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Move mlx5_ifc_vsc_space_bits and mlx5_ifc_vsc_addr_bits to mlx5_ifc.h.

Submitted by: kib@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Add Firmware Reset Level, MFRL, register accessors in mlx5core.

Submitted by: kib@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Update performance counter bits in mlx5core.

MFC after: 3 days
Sponsored by: Mellanox Technologies

Implement reading PCI power status in mlx5core.

Implement a watchdog as part of the healtcare subsystem which
reads the PCI power status during startup and upon the PCI
power status change event and store it into the core device
structure. This value is then exported to user-space via a
read-only SYSCTL. A dmesg print has been added to inform
the admin about the PCI power status.

MFC after: 3 days
Sponsored by: Mellanox Technologies

Add support for 200Gb ethernet speeds to mlx5core.

Submitted by: slavash@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Add vnic steering drop statistics in mlx5en(4).

MFC after: 3 days
Sponsored by: Mellanox Technologies

Enhance MCAM reg to allow query on access reg support in mlx5core.

Enhance MCAM to allow the driver to query which access regs are
supported. For now, expose the regs needed for FW flashing.

Linux commit:
0ab87743cc8c5bcd482daf71961ed5fc45349e01

Submitted by: slavash@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Add MCC (Management Component Control) register definitions in mlx5core.

MCC (Management Component Control) allows to control a firmware
component update.

MCDA (Management Component Data Access) allows to read and write
a firmware component.

MCQI (Management Component Query Information) allows to query
information about firmware components.

Linux commit:
4717628938423fcba0aa8fa889e9fed4eb6a655f

Submitted by: slavash@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Expose PCAM, MCAM registers infrastructure in mlx5core.

PCAM: Ports capabilities mask register.
MCAM: Management capabilities mask register.

PCAM and MCAM registers will provide information regarding firmware
support for different features, in order to avoid cases where new driver
combined with old firmware results in syndromes (for ex. PCIe counters
before this patchset).

Linux commit:
cfdcbceaeffc669b70d904d80a2df9c86c232566

Submitted by: slavash@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Add Fast teardown support to mlx5core.

Today mlx5 devices support two teardown modes:
1- Regular teardown
2- Force teardown

This change introduces the enhanced version of the "Force teardown" that
allows SW to perform teardown in a faster way without the need to reclaim
all the pages.

Fast teardown provides the following advantages:
1- Fix a FW race condition that could cause command timeout
2- Avoid moving to polling mode
3- Close the vport to prevent PCI ACK to be sent without been
scattered to memory

Linux commit:
fcd29ad17c6ff885dfae58f557e9323941e63ba2

MFC after: 3 days
Sponsored by: Mellanox Technologies

Update Ethernet extended counters in mlx5en(4).

Expose all Ethernet extended counters those counters via debug_stats
sysctl:
dev.mce.X.debug_stats

Submitted by: slavash@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Add temperature warning event to log in mlx5core.

Temperature warning event is sent by FW to indicate high temperature
as detected by one of the sensors on the board.
Add handling of this event by writing the numbers of the alert sensors
to the kernel log.

Linux commit:
1865ea9adbfaf341c5cd5d8f7d384f19948b2fe9

Submitted by: slavash@
MFC after: 3 days
Sponsored by: Mellanox Technologies

mlx5en: Add support for IFM_10G_LR and IFM_40G_ER4 media types.

Inspect the ethernet compliance code to figure out actual cable type by reading
the PDDR module info register.

Submitted by: hselasky@
Approved by: hselasky (mentor)
MFC after: 1 week
Sponsored by: Mellanox Technologies

mlx5fpga: IOCTL for FPGA temperature measurement

Submitted by: kib@
Approved by: hselasky (mentor)
MFC after: 1 week
Sponsored by: Mellanox Technologies

mlx5fpga: Initial code import.

Submitted by: kib@
Approved by: hselasky (mentor)
MFC after: 1 week
Sponsored by: Mellanox Technologies

mlx5: Implement support for configuring PCIe packet write ordering via a sysctl.

Submitted by: hselasky@
Approved by: hselasky (mentor)
MFC after: 1 week
Sponsored by: Mellanox Technologies

Implement support for Differentiated Service Code Point, DSCP, in mlx5en(4).

The DSCP feature is controlled using a set of sysctl(8) fields under
the qos sysctl directory entry for mlx5en(4).

For Routable RoCE QPs, the DSCP should be set in the QP's address path.
The DSCP's value is derived from the traffic class.

Linux commit:
ed88451e1f2d400fd6a743d0a481631cf9f97550

MFC after: 1 week
Sponsored by: Mellanox Technologies

Add support for fast unload in shutdown flow in mlx5core.

This patch accumulates the following Linux commits:

- 8812c24d28f4972c4f2b9998bf30b1f2a1b62adf
net/mlx5: Add fast unload support in shutdown flow
- 59211bd3b6329c3e5f4a90ac3d7f87ffa7867073
net/mlx5: Split the load/unload flow into hardware and software flows
- 4525abeaae54560254a1bb8970b3d4c225d32ef4
net/mlx5: Expose command polling interface

Submitted by: Matthew Finlay <matt@mellanox.com>
MFC after: 1 week
Sponsored by: Mellanox Technologies

Add vendor specific capability interface support in mlx5core.

Add the ability to access the vendor specific space gateway in order
to support reading and writing data into the different configuration
domains.

Submitted by: Matthew Finlay <matt@mellanox.com>
MFC after: 1 week
Sponsored by: Mellanox Technologies

Add support for explicit congestion notification, ECN, to mlx5ib(4).

ECN configuration and statistics is available through a set of sysctl(8)
nodes under sys.class.infiniband.mlx5_X.cong . The ECN configuration
nodes can also be used as loader tunables.

MFC after: 1 week
Sponsored by: Mellanox Technologies

Use the autogenerated interface file for all commands in mlx5core.

This patch accumulates the following Linux commits:
- 90b3e38d048f09b22fb50bcd460cea65fd00b2d7
mlx5_core: Modify CQ moderation parameters
- 09a7d9eca1a6cf5eb4f9abfdf8914db9dbd96f08
mlx5_core: QP/XRCD commands via mlx5 ifc
- 1a412fb1caa2c1b77719ccb5ed8b0c3c2bc65da7
mlx5_core: Modify QP commands via mlx5 ifc
- ec22eb53106be1472ba6573dc900943f52f8fd1e
mlx5_core: MKey/PSV commands via mlx5 ifc
- 73b626c182dff06867ceba996a819e8372c9b2ce
mlx5_core: EQ commands via mlx5 ifc
- 20ed51c643b6296789a48adc3bc2cc875a1612cf
mlx5_core: Access register and MAD IFC commands via mlx5 ifc
- a533ed5e179cd15512d40282617909d3482a771c
mlx5_core: Pages management commands via mlx5 ifc
- b8a4ddb2e8f44f872fb93bbda2d541b27079fd2b
mlx5_core: Add MLX5_ARRAY_SET64 to fix BUILD_BUG_ON
- af1ba291c5e498973cc325c501dd8da80b234571
mlx5_core: Refactor internal SRQ API
- b06e7de8a9d8d1d540ec122bbdf2face2a211634
mlx5_core: Refactor device capability function
- c4f287c4a6ac489c18afc4acc4353141a8c53070
mlx5_core: Unify and improve command interface

Submitted by: Matthew Finlay <matt@mellanox.com>
MFC after: 1 week
Sponsored by: Mellanox Technologies

Implement rate limit per traffic class in mlx5core.

Add support for rate limiting traffic class via sysctl.

Submitted by: Slava Shwartsman <slavash@mellanox.com>
MFC after: 1 week
Sponsored by: Mellanox Technologies

Add more and update existing mlx5 core firmware structure definitions and bits.
This change is part of coming ibcore and mlx5ib updates.

Sponsored by: Mellanox Technologies
MFC after: 1 week

Set ATOMIC endian mode in mlx5 core.

The hardware is capable of 2 requestor endianness modes for standard 8
byte atomics: BE (0x0) and host endianness (0x1). Read the supported
modes from hca atomic capabilities and configure HW to host endianness
mode if supported.

Sponsored by: Mellanox Technologies
MFC after: 1 week

Refactor the flowsteering APIs used by mlx5en(4). This change is needed by
the coming ibcore and mlx5ib updates in order to support traffic redirection
to so-called raw ethernet QPs.

Remove unused E-switch related routines and files while at it.

Sponsored by: Mellanox Technologies
MFC after: 1 week

Update firmware interface structures and definitions adding support
for new features and commands.

MFC after: 1 week
Sponsored by: Mellanox Technologies

Add more firmware related structures and update existing ones in the
MLX5 core module. Update the set and query diagnostics counter API.

Sponsored by: Mellanox Technologies
MFC after: 1 week

Update the MLX5 core module:
- Add new firmware commands and update existing ones.
- Add more firmware related structures and update existing ones.
- Some minor fixes, like adding missing \n to some prints.

Sponsored by: Mellanox Technologies
MFC after: 1 week

Add mlx5 and mlx5en driver(s) for ConnectX-4 and ConnectX-4LX cards
from Mellanox Technologies. The current driver supports ethernet
speeds up to and including 100 GBit/s. Infiniband support will be
done later.

The code added is not compiled by default, which will be done by a
separate commit.

Sponsored by: Mellanox Technologies
MFC after: 2 weeks