Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r256689:
Use the same actor key for media types of the same speed.

PR: 176097

MFC r256832:
Add a note that lacp_compose_key() should be updated, when new media
types will be added.

Submitted by: melifaro

MFC r236062

Turn LACP debugging from a compile time option to a sysctl, it is very handy to
be able to turn it on when negotiation to a switch misbehaves.

MFC r232629,r232640

Add the ability to set which packet layers are used for the load balance hash
calculation.

MFC r232008,232010,232080,232089

Using the flowid in the mbuf assumes the network card is giving a good hash for
the traffic flow, this may not be the case giving poor traffic distribution.
Add a sysctl which allows us to fall back to our own flow hash code.

PR: kern/164901

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Use the flowid if its available for selecting the tx port.

Update the interface baudrate taking into account the max speed for the
different aggregation protocols.

Switch the LACP state machine over to its own mutex to protect the internals,
this means that it no longer grabs the lagg rwlock. Use two port table arrays
which list the active ports for Tx and switch between them with an atomic op.
Now the lagg rwlock is only exclusively locked for management (ioctls) and
queuing of lacp control frames isnt needed.

Pass any unmatched slowprotocols frames up the stack instead of dropping them,
there are more subtypes than just LACP.

Use the safer callout_init_rw() to allow the softclock to grab the
rwlock for us.

Allow the LACP state to be queried from userland which at the moment is the
actor and partner peer info. Print out the active aggregator and per port data
in verbose mode from ifconfig.

Approved by: re (mux)

non-functional cleanup
- remove dead code
- use consistent variable names
- gc unused defines
- whitespace cleanup

Compare the partner system priority when choosing the aggregator.

Implement the Marker Protocol. A marker frame is placed on the interface queue
of each port and any further packets are blocked, when the all the marker frames
have been returned to us from the remote network device then we can be sure
that all interface queues are empty.

This is needed when a port is added or removed from the aggregation since it
will affect the hash based distribution, if the queues are not empty then a
packet from an existing connection may be placed on a different interface and
arrive out of order. This was previously achieved by suppressing transmission for
1 second, now that there is an active feedback this timeout as been increased
to 3 seconds and used as a fallback.

Fix locking assert where we should hold the reader lock.

Change from a mutex to a read/write lock. This allows the tx port to be
selected simultaneously by multiple senders and transmit/receive is not
serialised between aggregated interfaces.

Avoid touching various unsafe parts if the interface is disappearing.

Change from using if_delmulti() to if_delmulti_ifma() as it simplifies the code
and is safe to use if the ifp has disappeared.

Suggested by: bms

- Add a disabled state for ports that can not be aggregated
- Refine check for lacp links, set to disabled if not suitable

Test for IFM_FDX rather than IFM_HDX as the half-duplex bit may not be set even
if the link is not full-duplex.

Rename the trunk(4) driver to lagg(4) as it is too similar to vlan trunking.

The name trunk is misused as the networking term trunk means carrying multiple
VLANs over a single connection. The IEEE standard for link aggregation (802.3
section 3) does not talk about 'trunk' at all while it is used throughout IEEE
802.1Q in describing vlans.

The lagg(4) driver provides link aggregation, failover and fault tolerance.

Discussed on: current@

Add the trunk(4) driver for providing link aggregation, failover and fault
tolerance. This driver allows aggregation of multiple network interfaces as
one virtual interface using a number of different protocols/algorithms.

failover - Sends traffic through the secondary port if the master becomes
inactive.
fec - Supports Cisco Fast EtherChannel.
lacp - Supports the IEEE 802.3ad Link Aggregation Control Protocol
(LACP) and the Marker Protocol.
loadbalance - Static loadbalancing using an outgoing hash.
roundrobin - Distributes outgoing traffic using a round-robin scheduler
through all active ports.

This code was obtained from OpenBSD and this also includes 802.3ad LACP support
from agr(4) in NetBSD.