MFC r362045:
Make sure packets generated by raw IP code is let through by mlx5en(4).

Allow the TCP header to reside in the mbuf following the IP header.
Else such packets will get dropped.

Backtrace:
mlx5e_sq_xmit()
mlx5e_xmit()
ether_output_frame()
ether_output()
ip_output_send()
ip_output()
rip_output()
sosend_generic()
sosend()
kern_sendit()
sendit()
sys_sendto()
amd64_syscall()
fast_syscall_common()

Sponsored by: Mellanox Technologies

MFC r362044:
Extend use of unlikely() in the fast path, in mlx5en(4).

Typically the TCP/IP headers fit within the first mbuf and should not
trigger any of the error cases. Use unlikely() for these cases.

No functional change.

Sponsored by: Mellanox Technologies

MFC r362043:
Use const keyword when parsing the TCP/IP header in the fast path in mlx5en(4).

When parsing the TCP/IP header in the fast path, make it clear by using
the const keyword, no fields are to be modified inside the transmitted
packet.

No functional change.

Sponsored by: Mellanox Technologies

MFC r359653:
Count number of times transmit ring is out of buffers in mlx5en(4).

Differential Revision: https://reviews.freebsd.org/D24273
Sponsored by: Mellanox Technologies

MFC r359654:
Ensure a minimum inline size of 16 bytes in mlx5en(4).

This includes 14 bytes of ethernet header and 2 bytes of VLAN header.

This allows for making assumptions about the inline size limit
in the fast transmit path later on.

Use a signed integer variable to catch underflow.

Sponsored by: Mellanox Technologies

MFC r355447:
Fix compilation issue with mlx5core and sparc64 (gcc48):

sys/dev/mlx5/mlx5_en/mlx5_en_tx.c:335: error: requested alignment is not a constant

Sponsored by: Mellanox Technologies

MFC r348603:
Make sure the DMA tags get freed in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r341585:
mlx5en: Improve configuration of HW LRO.

In order to enable HW LRO, both the "hw_lro" sysctl in the mlx5en(4) config
space must be set, and the ifconfig(8) LRO capability must be set. Any other
settings will disable HW LRO.

Sponsored by: Mellanox Technologies

MFC r339584:
mlx5: Notify user that the ConnectX-6 shutdown its port due to power limitation

If power exceed the slot limit, or slot limit is unknown the ConnectX-6
firmware will shutdown its port.
Inform the user via debug message.

Approved by: hselasky (mentor), kib (mentor)
Sponsored by: Mellanox Technologies

MFC r338492:
Add support for receive side scaling stride, RSSS, in mlx5en(4).

The receive side scaling stride parameter is a value which define the interval
between active receive side queues. The traffic for the inactive queues is
redirected to the nearest active queue by use of modulus. The default value
of this parameter is one, which means all receive side queues are used.

The point of this feature is to redirect more traffic to fewer receive side
queues in order to take more advantage of sorted large receive offload,
sorted LRO. The sorted LRO works better when more packets are accumulated
per service interval.

Sponsored by: Mellanox Technologies

MFC r338490:
Don't stall transmit queue on drops in mlx5en(4).

When a transmitted packet is dropped don't stall the transmit queue.

Sponsored by: Mellanox Technologies

MFC r338489:
Maximum number of mbuf frags is off-by-one for worst case scenario in mlx5en(4).

Inspecting the PRM no more than 0x3F data segments, DS, of size 16 bytes is
allowed.

Worst case scenario summary of DS usage:
Header is fixed: 2 DS
Maximum inlining: 98 => (98 - 2) / 16 = 6 DS
Remainder: 0x3F - 2 - 6 = 55 DS (mbuf frags)

Previously a value of 56 DS was used and this would work in the
normal case because not all inline data area was used up.

Sponsored by: Mellanox Technologies

Bump the mlx5core, mlx5en(4) and mlx5ib driver version.

This is a direct commit.

Sponsored by: Mellanox Technologies

MFC r330657:
Use vport rather than physical-port MTU in mlx5en(4).

Set and report vport MTU rather than physical MTU,
The driver will set both vport and physical port mtu
and will rely on the query of vport mtu.

SRIOV VFs have to report their MTU to their vport manager (PF),
and this will allow them to work with any MTU they need
without failing the request.

Also for some cases where the PF is not a port owner, PF can
work with MTU less than the physical port mtu if set physical
port mtu didn't take effect.

Based on Linux upstream commit:
cd255efff9baadd654d6160e52d17ae7c568c9d3

Submitted by: Meny Yossefi <menyy@mellanox.com>
Sponsored by: Mellanox Technologies

MFC r325661:
Expose the current hardware MTU in mlx5en(4) as a separate entry
in the sysctl tree.

Sponsored by: Mellanox Technologies

Enter error state when handling bad device in mlx5core and add checks
for error state to mlx5en(4) to make live migration work.

This is a direct commit.

Sponsored by: Mellanox Technologies

MFC r336450:
Do not inline transmit headers and use HW VLAN tagging if supported by mlx5en(4).

Query the minimal inline mode supported by the card.
When creating a send queue, cache the queried mode and optimize the transmit
if no inlining is required. In this case, we can avoid touching the headers
cache line and avoid dirtying several more lines by copying headers into
the send WQEs. Also, if no inline headers are used, hardware assists in
the VLAN tag framing.

Submitted by: kib@, slavash@
Sponsored by: Mellanox Technologies

MFC r336407:
Handle jumbo frames without requiring big clusters in mlx5en(4).

The scatter list is formed by the chunks of MCLBYTES each, and larger
than default packets are returned to the stack as the mbuf chain.

Submitted by: kib@
Sponsored by: Mellanox Technologies

MFC r328591:
Move the mlx5 core device pointer first in the mlx5en priv. This help simplify
checks to recognize own network devices when using mlx5ib. This patch fixes
an issues where mlx5ib fails to recognize mceX network devices for use with
RoCE.

Sponsored by: Mellanox Technologies

MFC r324792:
The remote DMA TCP portspace selector, RDMA_PS_TCP, is used for both
iWarp and RoCE in ibcore. The selection of RDMA_PS_TCP can not be used
to indicate iWarp protocol use. Backport the proper IB device
capabilities from Linux upstream to distinguish between iWarp and
RoCE. Only allocate the additional socket required for iWarp for RDMA
IDs when at least one iWarp device present. This resolves
interopability issues between iWarp and RoCE in ibcore

Reviewed by: np @
Differential Revision: https://reviews.freebsd.org/D12563
Sponsored by: Mellanox Technologies

Use MAC-based GID format for the GID table entries in mlx5ib(4) to be
compatible with the ibcore module in FreeBSD 10-stable.

This fixes RDMA over VLAN, because the MAC-based GID format embeds
the VLAN ID into the GID itself and this is what ibcore expects when
requesting GID indexes from the GID cache.

RoCE V1.5 and V2 is not supported in 10-stable by mlx5ib(4).
This is a direct commit.

Sponsored by: Mellanox Technologies

MFC r289568, r300676, r300677, r300719, r300720 and r300721:
Implement LinuxKPI module parameters as SYSCTLs.

The bool module parameter is no longer supported, because there is no
equivalent in FreeBSD 10-stable. These are converted into "int" type.

There are two macros available which control the behaviour of the
LinuxKPI module parameters:

- LINUXKPI_PARAM_PARENT allows the consumer to set the SYSCTL parent
where the modules parameters will be created.

- LINUXKPI_PARAM_PREFIX defines a parameter name prefix, which is
added to all created module parameters.

The LinuxKPI module parameters also have a permissions value.
If any write bits are set we are allowed to modify the module
parameter runtime. Reflect this when creating the static SYSCTL
nodes.

The module_param_call() function is no longer supported.

Sponsored by: Mellanox Technologies

MFC r324202:
Make sure the doorbell lock is valid for the i386 version
of the mlx5en(4) driver.

Tested by: gallatin @
Sponsored by: Mellanox Technologies

MFC r322810 and r322830:
Add new mlx5ib(4) driver to the kernel source tree which supports
Remote DMA over Converged Ethernet, RoCE, for the ConnectX-4 series of
PCI express network cards.

There is currently no user-space support and this driver only supports
kernel side non-routable RoCE V1. The krping kernel module can be used
to test this driver. Full user-space support including RoCE V2 will be
added as part of the ongoing upgrade to ibcore from Linux 4.9. Otherwise
this driver is feature equivalent to mlx4ib(4). The mlx5ib(4) kernel
module will only be built when WITH_OFED=YES is specified.

Approved by: re (marius)
Sponsored by: Mellanox Technologies

MFC r322250:
Count drop events due to lack of PCI bandwidth as queue drops and not as
input errors in the mlx5en(4) driver. This improves the sysadmin view of
physical port errors.

Approved by: re (kib)
Submitted by: gallatin@
Sponsored by: Mellanox Technologies

MFC r322251:
Make sure the received IP header gets 32-bit aligned for short packets
in the mlx5en(4) driver.

Approved by: re (kib)
Sponsored by: Mellanox Technologies

MFC r312882, r321983 and r321984:
Use the busdma API to allocate all DMA-able memory.

The MLX5 driver has four different types of DMA allocations which are
now allocated using busdma:

1) The 4K firmware DMA-able blocks. One busdma object per 4K allocation.
2) Data for firmware commands use the 4K firmware blocks split into four 1K blocks.
3) The 4K firmware blocks are also used for doorbell pages.
4) The RQ-, SQ- and CQ- DMA rings. One busdma object per allocation.

After this patch the mlx5en driver can be used with DMAR enabled in
the FreeBSD kernel.

Sponsored by: Mellanox Technologies

MFC r312881:
Add support for device surprise removal and other PCI errors.

- When device disappears from PCI indicate error device state and:
1) Trigger command completion for all pending commands
2) Prevent new commands from executing and return:
- success for modify and remove/cleanup commands
- failure for create/query commands
3) When reclaiming pages for a device in error state don't ask FW to
return all given pages, just release the allocated memory

Sponsored by: Mellanox Technologies

MFC r312879:
Rename struct fw_page into struct mlx5_fw_page as a preparation step
for adding busdma support.

Sponsored by: Mellanox Technologies

MFC r312880:
Wait for all VFs pages to be reclaimed before closing EQ pages.

Sponsored by: Mellanox Technologies

MFC r312875:
Make fw_pages statistics counter 64-bit to avoid overflow.

Sponsored by: Mellanox Technologies

MFC r312877 and r312878:
Minor code refactor as a preparation step for suprise removal of CX-4
PCI device(s), changes:
- alloc_entry() now clears bit for page slot entry aswell
- update of cmd->ent_arr[] is now under cmd->alloc_lock
- complete command if alloc_entry() fails

Sponsored by: Mellanox Technologies

MFC r312983:
Make "desc" pointer non-constant inside the mlx5_core_diagnostics_entry
structure. This fixes compilation with amd64-xtoolchain-gcc.

PR: 216588
Sponsored by: Mellanox Technologies

MFC r312876:
Use ffs() to scan for first bit instead of using a for() loop.
Minor code refactor while at it.

Sponsored by: Mellanox Technologies

MFC r312872:
Add support for reading advanced diagnostic counters.

By default reading the diagnostic counters is disabled. The firmware
decides which counters are supported and only those supported show up
in the dev.mce.X.diagnostics sysctl tree.

To enable reading of diagnostic counters set one or more of the
following sysctls to one:

dev.mce.X.conf.diag_general_enable=1
dev.mce.X.conf.diag_pci_enable=1

Sponsored by: Mellanox Technologies

MFC r312865:
Enforce reading the consumer and producer counters once to ensure
consistent return values from the mlx5e_sq_has_room_for()
function. The two counters are incremented by different threads under
different locks.

Sponsored by: Mellanox Technologies

MFC r312537:
Remove superfluous return statement.

Sponsored by: Mellanox Technologies

MFC r312536:
Allow transmit packet bufring in software to be disabled.

- Add new sysctl node to control the transmit packet bufring.

- Add optimised version of the transmit routine which output packets
directly to the DMA ring instead of using bufring in case the transmit
lock is congested. This can reduce the number of taskswitches which in
turn influence the overall system CPU usage, depending on the
workload.

- Add " TX" suffix to debug name for transmit mutexes to silence some
witness warnings about aquiring duplicate locks having same name.

Sponsored by: Mellanox Technologies
Suggested by: gallatin @

MFC r312528:
Make draining a sendqueue more robust.

Add own state variable to track if a sendqueue is stopped or not.
This will prevent traffic from entering the sendqueue while it is
being destroyed.

Update drain function to wait for traffic to be transmitted before
returning when the link state is active.

Add extra checks in transmit path for stopped SQ's.

While at it:
- Use likely() for a mbuf pointer check.
- Remove redundant IFF_DRV_RUNNING check.

Sponsored by: Mellanox Technologies

MFC r312527:
Add runtime support for modifying the SQ and RQ completion event
moderation mode. The presence of this feature is indicated through the
firmware capabilities.

Sponsored by: Mellanox Technologies

MFC r312526:
Update firmware interface structures and definitions adding support
for new features and commands.

Sponsored by: Mellanox Technologies

MFC r317568:

Improve sysadmin visibility of physical port error counters in the
mlx5en driver.

Sponsored by: Mellanox Technologies

MFC r310387:
Add more comments regarding collection of statistics counters.

Sponsored by: Mellanox Technologies

MFC r310388:
Make a read only pointer constant.

Sponsored by: Mellanox Technologies

MFC r309406:
Remove useless NULL checks.

NULL is not returned when allocating memory passing the M_WAITOK flag.

Submitted by: trasz @
Differential Revision: https://reviews.freebsd.org/D5772
Sponsored by: Mellanox Technologies

MFC r308416:
Add timer to watch the RQ when we are out of mbufs.

The firmware/hardware does not generate additional completion
events unless we post new buffers. Use a timer to try to post
more buffers in case we are temporarily out of mbufs. Else
the receive schedule completely stops.

Sponsored by: Mellanox Technologies

MFC r308414:
Add more firmware related structures and update existing ones in the
MLX5 core module. Update the set and query diagnostics counter API.

Sponsored by: Mellanox Technologies

MFC r308413:
Query flow table capabilities according to the correct capability bit
for infiniband.

Sponsored by: Mellanox Technologies

MFC r308412:
Correct checksum fields in the "mlx5_mini_cqe8" structure. The fields
in question are currently not used.

Sponsored by: Mellanox Technologies

MFC r308411:
Ensure the firmware is notified of any host memory allocation
failures. Else firmware commands may time out waiting for host
memory.

Sponsored by: Mellanox Technologies

MFC r308409:
When a firmware command times out do not free the command structure to
avoid use after free.

Sponsored by: Mellanox Technologies

MFC r306453:
Set hardware stats flag to avoid double counting the number of incoming bytes.

Found by: Ben RUBSON <ben.rubson@gmail.com>
Sponsored by: Mellanox Technologies

MFC r305877:
mlx5en: Fix duplicate mbuf free-by-code.

When mlx5e_sq_xmit() returns an error code and the mbuf pointer is set,
we should not free the mbuf, because the caller will keep the mbuf in
the drbr. Make sure the mbuf pointer is correctly set upon function
exit.

Sponsored by: Mellanox Technologies

MFC r305876:
mlx5en: Remove unused pdev pointer.

Sponsored by: Mellanox Technologies

MFC r305875:
mlx5en: Verify port type is ethernet before creating network device

Else the mlx5en driver might attach to infiniband ports.

Sponsored by: Mellanox Technologies

MFC r305874:
mlx5en: Allow setting the software MTU size below 1500 bytes

The hardware MTU size can't be set to a value less than 1500 bytes due
to side-band management support. Allow setting the software MTU size
below 1500 bytes, thus creating a mismatch between hardware and
software MTU sizes.

Sponsored by: Mellanox Technologies

MFC r305873:
mlx5en: Factor out common sendqueue code for use with rate limiting SQs.

Try to reuse code to setup sendqueues when possible by making some static
functions global. Further split the mlx5e_close_sq_wait() function to
separate out reusable parts.

Sponsored by: Mellanox Technologies

MFC r305872:
mlx5en: Properly declare doorbell lock for 32-bit CPUs.

Sponsored by: Mellanox Technologies

MFC r305871:
mlx5en: Optimise away duplicate UAR pointers.

This change also reduces the size of the mlx5e_sq structure so that the last
queue_state element will fit into the previous cacheline and then the mlx5e_sq
structure becomes one cacheline less for amd64.

Sponsored by: Mellanox Technologies

MFC r305870:
mlx5en: Make the mlx5e_open_cq() and mlx5e_close_cq() functions global.

Make some functions and structures global to allow for code reuse
when creating rate limiting sendqueues.

Sponsored by: Mellanox Technologies

MFC r305869:
mlx5en: Minor completion queue control path code refactor.

Move setting of CQ moderation mode together with the other
CQ moderation parameters. Pass completion event vector as
a separate argument to mlx5e_open_cq(), because its value is
different for each call. Pass mlx5e_priv pointer instead of
mlx5e_channel pointer so that code can be used by rate
limiting sendqueues.

Sponsored by: Mellanox Technologies

MFC r305868:
mlx5en: Separate the sendqueue from using the mlx5e_channel structure.

This change allows for reusing the transmit path for so called
rate limited senqueues. While at it optimise some pointer lookups
in the fast path.

Sponsored by: Mellanox Technologies

MFC r305867:
Update the MLX5 core module:
- Add new firmware commands and update existing ones.
- Add more firmware related structures and update existing ones.
- Some minor fixes, like adding missing \n to some prints.

Sponsored by: Mellanox Technologies

MFC r304601:
Increase the maximum RX/TX queue size. This allows for a RX/TX queue
size of 16384 mbufs. Previously the limit was 8192.

Sponsored by: Mellanox Technologies

MFC r303870:
Fix for use after free.

Clear the device description to avoid use after free because the
bsddev is not destroyed when the mlx5en module is unloaded. Only when
the parent mlx5 module is unloaded the bsddev is destroyed. This fixes
a panic on listing sysctls which refer strings in the bsddev after the
mlx5en module has been unloaded.

Sponsored by: Mellanox Technologies

MFC r301545:
Add SR-IOV guest support to the mlx5en driver.

This patch adds the missing pieces needed for device setup using the
mlx5en driver inside a virtual machine which is providing hardware
access through SR-IOV.

Sponsored by: Mellanox Technologies

MFC r302139:
Use correct Q-counter output array.

Sponsored by: Mellanox Technologies

MFC r300282:
Verify one sysctl parameter at a time. When a mlx5en sysctl parameter
is updated only verify the changed one instead of all.

No functional change.

Sponsored by: Mellanox Technologies
Tested by: Netflix

MFC r300280:
Optimise use of doorbell and remove redundant NOPs

Store the last doorbell write in the mlx5e_sq structure and write the
doorbell to the hardware when the transmit routine finishes
transmitting all queued mbufs.

Sponsored by: Mellanox Technologies
Tested by: Netflix

MFC r300277:
Implement TX completion event interleaving.

This patch implements a sysctl which allows setting a factor, N, for
how many work queue elements can be generated before requiring a
completion event. When a completion event happens the code simulates N
completion events instead of only one. When draining a transmit queue,
N-1 NOPs are transmitted at most, to force generation of the final
completion event. Further a timer is running every HZ ticks to flush
any remaining data off the transmit queue when the tx_completion_fact
> 1.

The goal of this feature is to reduce the PCI bandwidth needed when
transmitting data.

Sponsored by: Mellanox Technologies
Tested by: Netflix

MFC r298772:
Correct some error codes to native FreeBSD ones.

Sponsored by: Mellanox Technologies
Tested by: Netflix

MFC r298771:
Add function to detect the presence of a port module and use this
function to error out early when no port module is present and doing
eeprom access. This also prevents error codes from filling up in
dmesg.

Sponsored by: Mellanox Technologies
Tested by: Netflix

MFC r294319:
Fix an issue where the network adapter could be left in down state
after changing the HW LRO sysctl when previously in up state.

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4941

MFC r294318:
Add clarifying comment about CQE zipping.

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4940

MFC r294317:
Declare local variables at top of function.

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4939

MFC r294314:
Allow RX and TX pause frames to be set through ifconfig.

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4817

MFC r292949:
Add support for modifying coalescing parameters runtime.

Sponsored by: Mellanox Technologies

MFC r292948:
Allow I2C to read address 0x51 as well as address 0x50.

Submitted by: Shahar Klein <shahark@mellanox.com>
Sponsored by: Mellanox Technologies

MFC r292946:
10G ER/LR should present itself as LR.

MFC after: 1 week
Submitted by: Shahar Klein <shahark@mellanox.com>
Sponsored by: Mellanox Technologies

MFC r292838:
Add support for CQE zipping. CQE zipping reduces PCI overhead by
coalescing and zipping multiple CQEs into a single merged CQE. The
feature is enabled by default and can be disabled by a sysctl.

Implementing this feature mlx5_cqwq_pop() has been separated from
mlx5e_get_cqe().

Submitted by: Mark Bloch <markb@mellanox.com>
Differential Revision: https://reviews.freebsd.org/D4598
Sponsored by: Mellanox Technologies

MFC r292837:
Add support for sysctl tunables to 10-stable and older. Pushed through
head first to simplify driver maintenance.

MFC after: 1 week
Submitted by: Drew Gallatin <gallatin@freebsd.org>
Differential Revision: https://reviews.freebsd.org/D4552
Sponsored by: Mellanox Technologies

MFC r292835:
Make the eeprom dump function more readable and rename variables for
better clarity.

Submitted by: Daria Genzel <dariaz@mellanox.com>
Differential Revision: https://reviews.freebsd.org/D4551
Sponsored by: Mellanox Technologies

MFC r291939:
Update the mlx5 shared driver code to the latest version, which
include the following list of changes:

- Added eswitch ACL table management
Introduce API for managing ACL table.
This API include the following features:
1) vlan filter - for VST/VGT+ support.
2) spoofcheck.
3) robust functionality to allow/drop general untagged/tagged traffic.
4) support for both ingress and egress ACL types.

- Added loopback filter to the vacl table.

- Added multicast list set in the vPort context

- Added promiscuous mode set in the vPort context

- Set the vlan list in vPort context
1) Check caps if VLAN list is not longer than FW supports
2) Set MODIFY_NIC_VPORT_CONTEXT command

- Changed MLX5_EEPROM_MAX_BYTES from 48 to 32 so that a single EEPROM
reading cannot cross the 128-byte boundary. Previously reading the
MCIA register was done in batches of 48 bytes. The third reading
would then by-pass the 127th byte, which means that part of the low
page and part of the high page would be read at the same time, which
created a bug:
1st: 0-47 bytes
2nd: 48-95 bytes
3rd: 96-143 bytes

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4411

MFC r291938:
Add full support for Receive Side Scaling, RSS, to the mlx5en
driver. This includes binding all interrupt and worker threads
according to the RSS configuration, setting up correct Toeplitz
hashing keys as given by RSS and setting the correct mbuf
hashtype for all received traffic.

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4410

MFC r291932:
Add support for setting the TX moderation mode via a sysctl entry. TX
completion events can be moderated in the same way like RX completion
events. Expose this functionality by a sysctl variable.

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4409

MFC r291931:
The firmware no longer supports setting a port MTU of zero bytes.
Set the port MTU and then query it and report if any problems instead.

Submitted by: Shahar Klein <shahark@mellanox.com>
Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4408

MFC r290650, r290709, r291067, r291068, r291069 and r291070:
Add mlx5 and mlx5en driver(s) for ConnectX-4 and ConnectX-4LX cards
from Mellanox Technologies. The current driver supports ethernet
speeds up to and including 100 GBit/s. Infiniband support will be
done later.

The code added is not compiled by default, which will be done by a
separate commit.

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4177
Differential Revision: https://reviews.freebsd.org/D4178
Differential Revision: https://reviews.freebsd.org/D4179
Differential Revision: https://reviews.freebsd.org/D4180

Add mlx5 and mlx5en driver(s) for ConnectX-4 and ConnectX-4LX cards
from Mellanox Technologies. The current driver supports ethernet
speeds up to and including 100 GBit/s. Infiniband support will be
done later.

The code added is not compiled by default, which will be done by a
separate commit.

Sponsored by: Mellanox Technologies
MFC after: 2 weeks