ena: Upgrade ena-com to freebsd v2.7.0

Merge commit '04cf7cee5ae0c6bb29eb21ce85bab7ca7386a95f'

Approved by: cperciva (mentor)
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

ena: merge ena-com v2.5.0 upgrade

Merge commit '2530eb1fa01bf28fbcfcdda58bd41e055dcb2e4a'

Adjust the driver to the upgraded ena-com part twofold:

First update is related to the driver's NUMA awareness.

Allocate I/O queue memory in NUMA domain local to the CPU bound to the
given queue, improving data access time. Since this can result in
performance hit for unaware users, this is done only when RSS
option is enabled, for other cases the driver relies on kernel to
allocate memory by itself.

Information about first CPU bound is saved in adapter structure, so
the binding persists after bringing the interface down and up again.

If there are more buckets than interface queues, the driver will try to
bind different interfaces to different CPUs using round-robin algorithm
(but it will not bind queues to CPUs which do not have any RSS buckets
associated with them). This is done to better utilize hardware
resources by spreading the load.

Add (read-only) per-queue sysctls in order to provide the following
information:
- queueN.domain: NUMA domain associated with the queue
- queueN.cpu: CPU affinity of the queue

The second change is for the CSUM_OFFLOAD constant, as ENA platform
file has removed its definition. To align to that change, it has been
added to the ena_datapath.h file.

Submitted by: Artur Rojek <ar@semihalf.com>
Submitted by: Dawid Gorecki <dgr@semihalf.com>
Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

ena: Remove redundant declaration of ena_log_level.

GCC6 raises a -Wredundant-decl error due to duplicate declarations
in ena_fbsd_log.h and ena_plat.h.

Sponsored by: Chelsio Communications

Merge tag 'vendor/ena-com/2.4.0'

Update the driver in order not to break its compilation
and make use of the new ENA logging system

Migrate platform code to the new logging system provided by ena_com
layer.

Make ENA_INFO the new default log level.

Remove all explicit use of `device_printf`, all new logs requiring one
of the log macros to be used.

Adjust ENA driver files to latest ena-com changes

* Use the new API of ena_trace_*
* Fix typo syndrom --> syndrome
* Remove validation of the Rx req ID (already performed in the ena-com)
* Remove usage of deprecated ENA_ASSERT macro

Submitted by: Ido Segev <idose@amazon.com>
Submitted by: Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D27115

Fix completion descriptors alignment for the ENA

The latest generation hardware requires IO CQ (completion queue)
descriptors memory to be aligned to a 4K. It needs that feature for
the best performance.

Allocating unaligned descriptors will have a big performance impact as
the packet processing in a HW won't be optimized properly. For that
purpose adjust ena_dma_alloc() to support it.

It's a critical fix, especially for the arm64 EC2 instances.

Submitted by: Ido Segev <idose@amazon.com>
Obtained from: Amazon, Inc
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D27114

Adjust ENA driver to the new HAL

* Removed adaptive interrupt moderation (not suported on FreeBSD).
* Use ena_com_free_q_entries instead of ena_com_free_desc.
* Don't use ENA_MEM_FREE outside of the ena_com.
* Don't use barriers before calling doorbells as it's already done in
the HAL.
* Add function that generates random RSS key, common for all driver's
interfaces.
* Change admin stats sysctls to U64.

Submitted by: Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

Introduce NETMAP support in ENA

Mock implementation of NETMAP routines is located in ena_netmap.c/.h
files. All code is protected under the DEV_NETMAP macro. Makefile was
updated with files and flag.

As ENA driver provide own implementations of (un)likely it must be
undefined before including NETMAP headers.

ena_netmap_attach function is called on the end of NIC attach. It fills
structure with NIC configuration and callbacks. Then provides it to
netmap_attach. Similarly netmap_detach is called during ena_detach.

Three callbacks are used.
nm_register is implemented by ena_netmap_reg. It is called when user
space application open or close NIC in NETMAP mode. Current action is
recognized based on onoff parameter: true means on and false off. As
NICs rings need to be reconfigured ena_down and ena_up are reused.
When user space application wants to receive new packets from NIC
nm_rxsync is called, and when there are new packets ready for Tx
nm_txsync is called.

Differential Revision: https://reviews.freebsd.org/D21934
Submitted by: Rafal Kozik <rk@semihalf.com>
Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

Adjust ENA driver to the new ena-com

Recent HAL change preparing to support ENAv2 required minor driver
modifications.

The ena_com_sq_empty_space() is not available in this ena-com, so it had
to be replaced with ena_com_free_desc().

Moreover, the ena_com_admin_init() is no longer using 3rd argument
indicating if the spin lock should be initialized, so it was removed.

Submitted by: Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

Apply fixes in ena-com

* Change ena-com BIT macro to work on unsigned value.
To make the shifting operations safer, they should be working on
unsigned values.

* Fix a mutex not owned ASSERT panic in ENA control path.
A thread calling cv_broadcast()/cv_signal() must hold the mutex used for
cv_wait(). Fix the ENA control path code that has this problem.

Submitted by: Krishna Yenduri <kyenduri@brkt.com>
Reviewed by: Michal Krawczyk <mk@semihalf.com>
Tested by: Michal Krawczyk <mk@semihalf.com>

Rework printouts and logging level in ENA driver

The driver was printing out a lot of information upon failure, which
does not have to be interested for the user.
Changing logging level required to rebuild driver with proper flags. The
proper sysctl was added, so the level now can be changed dynamically
using bitmask.

Levels of printouts were adjusted to keep on mind end user instead of
debugging purposes.

More verbose messages were added to align the driver with the Linux.

Fix building error introduced by the r325506 by casting csum_flags to
uint64_t.

Submitted by: Michal Krawczyk <mk@semihalf.com>
Reviewed by: byenduri_gmail.com
Obtained from: Semihalf
Sponsored by: Amazon, Inc.
Differential Revision: https://reviews.freebsd.org/D12868

Update ena-com HAL to v1.1.4.3 and update driver accordingly

The newest ena-com HAL supports LLQv2 and introduces
API changes. In order not to break the driver compilation
it was updated/fixed in a following way:

* Change version of the driver to 0.8.0
* Provide reset cause when triggering reset of the device
* Reset device after attach fails
* In the reset task free management irq after calling ena_down. Admin
queue can still be used before ena_down is called, or when it is
being handled
* Do not reset device if ena_reset_task fails
* Move call of the ena_com_dev_reset to the ena_down() routine - it
should be called only if interface was up
* Use different function for checking empty space on the sq ring
(ena-com API change)
* Fix typo on ENA_TX_CLEANUP_THRESHOLD
* Change checking for EPERM with EOPNOTSUPP - change in the ena-com API
* Minor style fixes

Submitted by: Michal Krawczyk <mk@semihalf.com>
Obtained from: Amazon.com, Inc.
Semihalf
Sponsored by: Amazon.com, Inc.
Differential Revision: https://reviews.freebsd.org/D12143

Merge ena-com 1.1.4.2

Update ENA HAL after fixing gcc build in r321861.

Submitted by: rlibby
Reviewed by: cognet (mentor)
Approved by: cognet (mentor)
Differential Revision: https://reviews.freebsd.org/D11480

Import Amazon Elastic Network Adapter (ENA) HAL to sys/contrib/

Import from vendor-sys/ena-com/1.1.4.1
SVN rev.: 317516
Version: 1.1.4.1

Obtained from: Amazon.com, Inc.

ena: merge ena-com v2.5.0 upgrade

Merge commit '2530eb1fa01bf28fbcfcdda58bd41e055dcb2e4a'

Adjust the driver to the upgraded ena-com part twofold:

First update is related to the driver's NUMA awareness.

Allocate I/O queue memory in NUMA domain local to the CPU bound to the
given queue, improving data access time. Since this can result in
performance hit for unaware users, this is done only when RSS
option is enabled, for other cases the driver relies on kernel to
allocate memory by itself.

Information about first CPU bound is saved in adapter structure, so
the binding persists after bringing the interface down and up again.

If there are more buckets than interface queues, the driver will try to
bind different interfaces to different CPUs using round-robin algorithm
(but it will not bind queues to CPUs which do not have any RSS buckets
associated with them). This is done to better utilize hardware
resources by spreading the load.

Add (read-only) per-queue sysctls in order to provide the following
information:
- queueN.domain: NUMA domain associated with the queue
- queueN.cpu: CPU affinity of the queue

The second change is for the CSUM_OFFLOAD constant, as ENA platform
file has removed its definition. To align to that change, it has been
added to the ena_datapath.h file.

Submitted by: Artur Rojek <ar@semihalf.com>
Submitted by: Dawid Gorecki <dgr@semihalf.com>
Obtained from: Semihalf
MFC after: 2 weeks
Sponsored by: Amazon, Inc.

ena: Remove redundant declaration of ena_log_level.

GCC6 raises a -Wredundant-decl error due to duplicate declarations
in ena_fbsd_log.h and ena_plat.h.

Sponsored by: Chelsio Communications

Merge tag 'vendor/ena-com/2.4.0'

Update the driver in order not to break its compilation
and make use of the new ENA logging system

Migrate platform code to the new logging system provided by ena_com
layer.

Make ENA_INFO the new default log level.

Remove all explicit use of `device_printf`, all new logs requiring one
of the log macros to be used.

Adjust ENA driver files to latest ena-com changes

* Use the new API of ena_trace_*
* Fix typo syndrom --> syndrome
* Remove validation of the Rx req ID (already performed in the ena-com)
* Remove usage of deprecated ENA_ASSERT macro

Submitted by: Ido Segev <idose@amazon.com>
Submitted by: Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D27115

Fix completion descriptors alignment for the ENA

The latest generation hardware requires IO CQ (completion queue)
descriptors memory to be aligned to a 4K. It needs that feature for
the best performance.

Allocating unaligned descriptors will have a big performance impact as
the packet processing in a HW won't be optimized properly. For that
purpose adjust ena_dma_alloc() to support it.

It's a critical fix, especially for the arm64 EC2 instances.

Submitted by: Ido Segev <idose@amazon.com>
Obtained from: Amazon, Inc
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D27114

Adjust ENA driver to the new HAL

* Removed adaptive interrupt moderation (not suported on FreeBSD).
* Use ena_com_free_q_entries instead of ena_com_free_desc.
* Don't use ENA_MEM_FREE outside of the ena_com.
* Don't use barriers before calling doorbells as it's already done in
the HAL.
* Add function that generates random RSS key, common for all driver's
interfaces.
* Change admin stats sysctls to U64.

Submitted by: Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

Introduce NETMAP support in ENA

Mock implementation of NETMAP routines is located in ena_netmap.c/.h
files. All code is protected under the DEV_NETMAP macro. Makefile was
updated with files and flag.

As ENA driver provide own implementations of (un)likely it must be
undefined before including NETMAP headers.

ena_netmap_attach function is called on the end of NIC attach. It fills
structure with NIC configuration and callbacks. Then provides it to
netmap_attach. Similarly netmap_detach is called during ena_detach.

Three callbacks are used.
nm_register is implemented by ena_netmap_reg. It is called when user
space application open or close NIC in NETMAP mode. Current action is
recognized based on onoff parameter: true means on and false off. As
NICs rings need to be reconfigured ena_down and ena_up are reused.
When user space application wants to receive new packets from NIC
nm_rxsync is called, and when there are new packets ready for Tx
nm_txsync is called.

Differential Revision: https://reviews.freebsd.org/D21934
Submitted by: Rafal Kozik <rk@semihalf.com>
Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

Adjust ENA driver to the new ena-com

Recent HAL change preparing to support ENAv2 required minor driver
modifications.

The ena_com_sq_empty_space() is not available in this ena-com, so it had
to be replaced with ena_com_free_desc().

Moreover, the ena_com_admin_init() is no longer using 3rd argument
indicating if the spin lock should be initialized, so it was removed.

Submitted by: Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

Apply fixes in ena-com

* Change ena-com BIT macro to work on unsigned value.
To make the shifting operations safer, they should be working on
unsigned values.

* Fix a mutex not owned ASSERT panic in ENA control path.
A thread calling cv_broadcast()/cv_signal() must hold the mutex used for
cv_wait(). Fix the ENA control path code that has this problem.

Submitted by: Krishna Yenduri <kyenduri@brkt.com>
Reviewed by: Michal Krawczyk <mk@semihalf.com>
Tested by: Michal Krawczyk <mk@semihalf.com>

Rework printouts and logging level in ENA driver

The driver was printing out a lot of information upon failure, which
does not have to be interested for the user.
Changing logging level required to rebuild driver with proper flags. The
proper sysctl was added, so the level now can be changed dynamically
using bitmask.

Levels of printouts were adjusted to keep on mind end user instead of
debugging purposes.

More verbose messages were added to align the driver with the Linux.

Fix building error introduced by the r325506 by casting csum_flags to
uint64_t.

Submitted by: Michal Krawczyk <mk@semihalf.com>
Reviewed by: byenduri_gmail.com
Obtained from: Semihalf
Sponsored by: Amazon, Inc.
Differential Revision: https://reviews.freebsd.org/D12868

Update ena-com HAL to v1.1.4.3 and update driver accordingly

The newest ena-com HAL supports LLQv2 and introduces
API changes. In order not to break the driver compilation
it was updated/fixed in a following way:

* Change version of the driver to 0.8.0
* Provide reset cause when triggering reset of the device
* Reset device after attach fails
* In the reset task free management irq after calling ena_down. Admin
queue can still be used before ena_down is called, or when it is
being handled
* Do not reset device if ena_reset_task fails
* Move call of the ena_com_dev_reset to the ena_down() routine - it
should be called only if interface was up
* Use different function for checking empty space on the sq ring
(ena-com API change)
* Fix typo on ENA_TX_CLEANUP_THRESHOLD
* Change checking for EPERM with EOPNOTSUPP - change in the ena-com API
* Minor style fixes

Submitted by: Michal Krawczyk <mk@semihalf.com>
Obtained from: Amazon.com, Inc.
Semihalf
Sponsored by: Amazon.com, Inc.
Differential Revision: https://reviews.freebsd.org/D12143

Merge ena-com 1.1.4.2

Update ENA HAL after fixing gcc build in r321861.

Submitted by: rlibby
Reviewed by: cognet (mentor)
Approved by: cognet (mentor)
Differential Revision: https://reviews.freebsd.org/D11480

Import Amazon Elastic Network Adapter (ENA) HAL to sys/contrib/

Import from vendor-sys/ena-com/1.1.4.1
SVN rev.: 317516
Version: 1.1.4.1

Obtained from: Amazon.com, Inc.

ena: Remove redundant declaration of ena_log_level.

GCC6 raises a -Wredundant-decl error due to duplicate declarations
in ena_fbsd_log.h and ena_plat.h.

Sponsored by: Chelsio Communications

Merge tag 'vendor/ena-com/2.4.0'

Update the driver in order not to break its compilation
and make use of the new ENA logging system

Migrate platform code to the new logging system provided by ena_com
layer.

Make ENA_INFO the new default log level.

Remove all explicit use of `device_printf`, all new logs requiring one
of the log macros to be used.

Adjust ENA driver files to latest ena-com changes

* Use the new API of ena_trace_*
* Fix typo syndrom --> syndrome
* Remove validation of the Rx req ID (already performed in the ena-com)
* Remove usage of deprecated ENA_ASSERT macro

Submitted by: Ido Segev <idose@amazon.com>
Submitted by: Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D27115

Fix completion descriptors alignment for the ENA

The latest generation hardware requires IO CQ (completion queue)
descriptors memory to be aligned to a 4K. It needs that feature for
the best performance.

Allocating unaligned descriptors will have a big performance impact as
the packet processing in a HW won't be optimized properly. For that
purpose adjust ena_dma_alloc() to support it.

It's a critical fix, especially for the arm64 EC2 instances.

Submitted by: Ido Segev <idose@amazon.com>
Obtained from: Amazon, Inc
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D27114

Adjust ENA driver to the new HAL

* Removed adaptive interrupt moderation (not suported on FreeBSD).
* Use ena_com_free_q_entries instead of ena_com_free_desc.
* Don't use ENA_MEM_FREE outside of the ena_com.
* Don't use barriers before calling doorbells as it's already done in
the HAL.
* Add function that generates random RSS key, common for all driver's
interfaces.
* Change admin stats sysctls to U64.

Submitted by: Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

Introduce NETMAP support in ENA

Mock implementation of NETMAP routines is located in ena_netmap.c/.h
files. All code is protected under the DEV_NETMAP macro. Makefile was
updated with files and flag.

As ENA driver provide own implementations of (un)likely it must be
undefined before including NETMAP headers.

ena_netmap_attach function is called on the end of NIC attach. It fills
structure with NIC configuration and callbacks. Then provides it to
netmap_attach. Similarly netmap_detach is called during ena_detach.

Three callbacks are used.
nm_register is implemented by ena_netmap_reg. It is called when user
space application open or close NIC in NETMAP mode. Current action is
recognized based on onoff parameter: true means on and false off. As
NICs rings need to be reconfigured ena_down and ena_up are reused.
When user space application wants to receive new packets from NIC
nm_rxsync is called, and when there are new packets ready for Tx
nm_txsync is called.

Differential Revision: https://reviews.freebsd.org/D21934
Submitted by: Rafal Kozik <rk@semihalf.com>
Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

Adjust ENA driver to the new ena-com

Recent HAL change preparing to support ENAv2 required minor driver
modifications.

The ena_com_sq_empty_space() is not available in this ena-com, so it had
to be replaced with ena_com_free_desc().

Moreover, the ena_com_admin_init() is no longer using 3rd argument
indicating if the spin lock should be initialized, so it was removed.

Submitted by: Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

Apply fixes in ena-com

* Change ena-com BIT macro to work on unsigned value.
To make the shifting operations safer, they should be working on
unsigned values.

* Fix a mutex not owned ASSERT panic in ENA control path.
A thread calling cv_broadcast()/cv_signal() must hold the mutex used for
cv_wait(). Fix the ENA control path code that has this problem.

Submitted by: Krishna Yenduri <kyenduri@brkt.com>
Reviewed by: Michal Krawczyk <mk@semihalf.com>
Tested by: Michal Krawczyk <mk@semihalf.com>

Rework printouts and logging level in ENA driver

The driver was printing out a lot of information upon failure, which
does not have to be interested for the user.
Changing logging level required to rebuild driver with proper flags. The
proper sysctl was added, so the level now can be changed dynamically
using bitmask.

Levels of printouts were adjusted to keep on mind end user instead of
debugging purposes.

More verbose messages were added to align the driver with the Linux.

Fix building error introduced by the r325506 by casting csum_flags to
uint64_t.

Submitted by: Michal Krawczyk <mk@semihalf.com>
Reviewed by: byenduri_gmail.com
Obtained from: Semihalf
Sponsored by: Amazon, Inc.
Differential Revision: https://reviews.freebsd.org/D12868

Update ena-com HAL to v1.1.4.3 and update driver accordingly

The newest ena-com HAL supports LLQv2 and introduces
API changes. In order not to break the driver compilation
it was updated/fixed in a following way:

* Change version of the driver to 0.8.0
* Provide reset cause when triggering reset of the device
* Reset device after attach fails
* In the reset task free management irq after calling ena_down. Admin
queue can still be used before ena_down is called, or when it is
being handled
* Do not reset device if ena_reset_task fails
* Move call of the ena_com_dev_reset to the ena_down() routine - it
should be called only if interface was up
* Use different function for checking empty space on the sq ring
(ena-com API change)
* Fix typo on ENA_TX_CLEANUP_THRESHOLD
* Change checking for EPERM with EOPNOTSUPP - change in the ena-com API
* Minor style fixes

Submitted by: Michal Krawczyk <mk@semihalf.com>
Obtained from: Amazon.com, Inc.
Semihalf
Sponsored by: Amazon.com, Inc.
Differential Revision: https://reviews.freebsd.org/D12143

Merge ena-com 1.1.4.2

Update ENA HAL after fixing gcc build in r321861.

Submitted by: rlibby
Reviewed by: cognet (mentor)
Approved by: cognet (mentor)
Differential Revision: https://reviews.freebsd.org/D11480

Import Amazon Elastic Network Adapter (ENA) HAL to sys/contrib/

Import from vendor-sys/ena-com/1.1.4.1
SVN rev.: 317516
Version: 1.1.4.1

Obtained from: Amazon.com, Inc.

Introduce NETMAP support in ENA

Mock implementation of NETMAP routines is located in ena_netmap.c/.h
files. All code is protected under the DEV_NETMAP macro. Makefile was
updated with files and flag.

As ENA driver provide own implementations of (un)likely it must be
undefined before including NETMAP headers.

ena_netmap_attach function is called on the end of NIC attach. It fills
structure with NIC configuration and callbacks. Then provides it to
netmap_attach. Similarly netmap_detach is called during ena_detach.

Three callbacks are used.
nm_register is implemented by ena_netmap_reg. It is called when user
space application open or close NIC in NETMAP mode. Current action is
recognized based on onoff parameter: true means on and false off. As
NICs rings need to be reconfigured ena_down and ena_up are reused.
When user space application wants to receive new packets from NIC
nm_rxsync is called, and when there are new packets ready for Tx
nm_txsync is called.

Differential Revision: https://reviews.freebsd.org/D21934
Submitted by: Rafal Kozik <rk@semihalf.com>
Michal Krawczyk <mk@semihalf.com>
Obtained from: Semihalf
Sponsored by: Amazon, Inc.

Merge ena-com 1.1.4.2

Update ENA HAL after fixing gcc build in r321861.

Submitted by: rlibby
Reviewed by: cognet (mentor)
Approved by: cognet (mentor)
Differential Revision: https://reviews.freebsd.org/D11480