tcp: add counter to track when SACK loss recovery uses TSO

Add a counter to track how frequently SACK has transmitted
more than one MSS using TSO. Instances when this will be
beneficial is the use of PRR, or when ACK thinning due to
GRO/LRO or ACK discards by the network are present.

Reviewed By: tuexen, #transport
Sponsored by: NetApp, Inc.
Differential Revision: https://reviews.freebsd.org/D45070

usr.bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

usr.bin: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

netstat: display c and C flags properly

Restructure c and C flag checks for string length to
work properly. Quickly bypass for non TCP protos too.

Reviewed By: tuexen
Sponsored by: NetApp, Inc.
Differential Revision: https://reviews.freebsd.org/D40480

netstat: fix printing of TCP pcbs with -A

This change touches both kernel and netstat(1), but either of the changes
will fix printing pcb addresses with -A.

The thing is that historically netstat(1) treated TCP differently, and
printed tcpcb address instead of inpcb address. This is not documented
anywhere! With e68b3792440 these two addresses became the same. It is
highly likely they will be the same for a long time, but it might be they
will start to differ again in a far future. My proposal is to stop
treating TCP differently with netstat(1) and right now is a good opportunity
to do that, since there will be no behavior change at all. The kernel
change to tcp_inptoxtp() will go into stable/14 to make it compatible with
netstat(1) binary from stable/13. We can drop it later, probably together
with in_ppcb pointer from inpcb. The in_ppcb in xinpcb will stay for size
compatibility.

Reviewed by: tuexen, rrs
Differential Revision: https://reviews.freebsd.org/D39736

tcp: account sent/received IP ECN markings independently

Have tcpstats (netstat -s) differentiate between received and sent
ECN-marked packets. Also account for IP ECN bits (on TCP packets)
even when the tcp session has not negotiated ECN support.

Event: IETF 115 Hackathon
Reviewed By: glebius, tuexen, #transport
Sponsored by: NetApp, Inc.
Differential Revision: https://reviews.freebsd.org/D37314

Tcp progress timeout

Rack has had the ability to timeout connections that just sit idle automatically. This
feature of course is off by default and requires the user set it on (though the socket option
has been missing in tcp_usrreq.c). Lets get the progress timeout fully supported in
the base stack as well as rack.

Reviewed by: tuexen
Sponsored by: Netflix Inc
Differential Revision: https://reviews.freebsd.org/D36716

divert(4): provide statistics

Instead of incrementing pretty random counters in the IP statistics,
create divert socket statistics structure. Export via netstat(1).

Differential revision: https://reviews.freebsd.org/D36381

divert: declare PF_DIVERT domain and stop abusing PF_INET

The divert(4) is not a protocol of IPv4. It is a socket to
intercept packets from ipfw(4) to userland and re-inject them
back. It can divert and re-inject IPv4 and IPv6 packets today,
but potentially it is not limited to these two protocols. The
IPPROTO_DIVERT does not belong to known IP protocols, it
doesn't even fit into u_char. I guess, the implementation of
divert(4) was done the way it is done basically because it was
easier to do it this way, back when protocols for sockets were
intertwined with IP protocols and domains were statically
compiled in.

Moving divert(4) out of inetsw accomplished two important things:

1) IPDIVERT is getting much closer to be not dependent on INET.
This will be finalized in following changes.
2) Now divert socket no longer aliases with raw IPv4 socket.
Domain/proto selection code won't need a hack for SOCK_RAW and
multiple entries in inetsw implementing different flavors of
raw socket can merge into one without requirement of raw IPv4
being the last member of dom_protosw.

Differential revision: https://reviews.freebsd.org/D36379

sockbuf: remove unused mbuf counter and cluster counter

With M_EXTPG mbufs these two counters already do not represent the
reality. As we are moving towards protocol independent socket buffers,
which may not even use mbufs at all, the counters become less and less
relevant. The only userland seeing them was 'netstat -x'.

PR: 264181 (exp-run)
Reviewed by: markj
Differential revision: https://reviews.freebsd.org/D35334

tcp: Add/update AccECN related statistics and numbers

Reserve couters in the tcps struct in preparation
for AccECN, extend the debugging output for TF2
flags, optimize the syncache flags from individual
bits to a codepoint for the specifc ECN handshake.

This is in preparation of AccECN.

No functional chance except for extended debug
output capabilities.

Reviewed By: #transport, rrs
Sponsored by: NetApp, Inc.
Differential Revision: https://reviews.freebsd.org/D34161

tcptw: count how many times a tcptw was actually useful

This will allow a sysadmin to lower net.inet.tcp.msl and
see how long tcptw are actually useful.

netstat: reduce use of historical Internet classes

When attempting to characterize bound addresses, netstat was checking
for host 0 on a (historical) net using inet_lnaof(). Such addresses
are not normally bound, as they would not work, with the exception
of the unspecified address, INADDR_ANY. Check for that explicitly.
Similarly, don't check bound addresses for a match to a network name.

MFC after: 1 month
Reviewed by: tuexen
Differential Revision: https://reviews.freebsd.org/D32714

tcp: Make dsack stats available in netstat and also make sure its aware of TLP's.

DSACK accounting has been for quite some time under a NETFLIX_STATS ifdef. Statistics
on DSACKs however are very useful in figuring out how much bad retransmissions you
are doing. This is further complicated, however, by stacks that do TLP. A TLP
when discovering a lost ack in the reverse path will cause the generation
of a DSACK. For this situation we introduce a new dsack-tlp-bytes as well
as the more traditional dsack-bytes and dsack-packets. These will now
all display in netstat -p tcp -s. This also updates all stacks that
are currently built to keep track of these stats.

Reviewed by: tuexen
Sponsored by: Netflix Inc.
Differential Revision: https://reviews.freebsd.org/D32158

netstat: Fix typo

Correct spelling of "received packers" to "received packets".

PR: 256926
Reported by: ghuckriede@blackberry.com
MFC after: 3 days

netstat: fix typo from 0471a8c73402

tcp: SACK Lost Retransmission Detection (LRD)

Recover from excessive losses without reverting to a
retransmission timeout (RTO). Disabled by default, enable
with sysctl net.inet.tcp.do_lrd=1

Reviewed By: #transport, rrs, tuexen, #manpages
Sponsored by: Netapp, Inc.
Differential Revision: https://reviews.freebsd.org/D28931

tcp: add support for TCP over UDP

Adding support for TCP over UDP allows communication with
TCP stacks which can be implemented in userspace without
requiring special priviledges or specific support by the OS.
This is joint work with rrs.

Reviewed by: rrs
Sponsored by: Netflix, Inc.
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D29469

Extend netstat to display TCP stack and detailed congestion state

Adding the "-c" option used to show detailed per-connection
congestion control state for TCP sessions.

This is one summary patch, which adds the relevant variables into
xtcpcb. As previous "spare" space is used, these changes are ABI
compatible.

Reviewed by: tuexen
MFC after: 2 weeks
Sponsored by: NetApp, Inc.
Differential Revision: https://reviews.freebsd.org/D26518

Add a -C option to netstat to display the congestion control for
TCP connections.

Reviewed by: rscheff
MFC after: 1 week
Sponsored by: Netflix, Inc.
Differential Revision: https://reviews.freebsd.org/D26414

Fix an issue of net.inet.igmp.stats handler.
The header of (struct igmpstat) could be cleared by sysctl(3).
This can be reproduced by "netstat -s -z -p igmp".

PR: 244584
MFC after: 1 week

netstat: igmp stats, error on unexpected information, not only warn

The igmp stats tend to print two lines of warning for an unexpected
version and length. Despite an invalid version and struct size it
continues to try to do something with the data. Do not try to parse
the remainder of the struct and error on warning.

Note the underlying issue of the data not being available properly
is still there and needs to be fixed seperately.

Reported by: test cases, lwhsu
MFC after: 3 weeks

Restore description of packets dropped due to full reassembly queue.

r265408 renamed tcps_rcvmemdrop to tcps_rcvreassfull and gave it a more
specific description. r279122 (libxo-ification) reverted that change.
This commit brings it back, but with a small tweak to the description.

MFC after: 2 weeks

Improve ARP logging.

r344504 added an extra ARP_LOG() call in case of an if_output() failure.
It turns out IPv4 can be noisy. In order to not spam the console by default:
(a) add a counter for these events so people can keep better track of how
often it happens, and
(b) add a sysctl to select the default ARP_LOG log level and set it to
INFO avoiding the one (the new) DEBUG level by default.

Claim a spare (1st one after 10 years since the stats were added) in order
to not break netstat from FreeBSD 12->13 updates in the future.

Reviewed by: karels
Differential Revision: https://reviews.freebsd.org/D19490

Fix compilation of world with WITHOUT_{INET,INET6}_SUPPORT or both set.

Buildworld failed when both WITHOUT_INET6_SUPPORT and INET equivalent were set.
Fix netstat and syslogd by applying appropriate #ifdef INET/INET6 to make world
compile again.

Reviewed by: ngie, hrs, ume
Welcomed by: Michael Dexter (D17040)
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D19418

Use uintptr_t alone when assigning to kvaddr_t variables.

Suggested by: jhb

Work around lame warnings in ancient gcc on 32-bit platforms.

Fixes r335979.

Make struct xinpcb and friends word-size independent.

Replace size_t members with ksize_t (uint64_t) and pointer members
(never used as pointers in userspace, but instead as unique
idenitifiers) with kvaddr_t (uint64_t). This makes the structs
identical between 32-bit and 64-bit ABIs.

On 64-bit bit systems, the ABI is maintained. On 32-bit systems,
this is an ABI breaking change. The ABI of most of these structs
was previously broken in r315662. This also imposes a small API
change on userspace consumers who must handle kernel pointers
becoming virtual addresses.

PR: 228301 (exp-run by antoine)
Reviewed by: jtl, kib, rwatson (various versions)
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D15386

Add the "TCP Blackbox Recorder" which we discussed at the developer
summits at BSDCan and BSDCam in 2017.

The TCP Blackbox Recorder allows you to capture events on a TCP connection
in a ring buffer. It stores metadata with the event. It optionally stores
the TCP header associated with an event (if the event is associated with a
packet) and also optionally stores information on the sockets.

It supports setting a log ID on a TCP connection and using this to correlate
multiple connections that share a common log ID.

You can log connections in different modes. If you are doing a coordinated
test with a particular connection, you may tell the system to put it in
mode 4 (continuous dump). Or, if you just want to monitor for errors, you
can put it in mode 1 (ring buffer) and dump all the ring buffers associated
with the connection ID when we receive an error signal for that connection
ID. You can set a default mode that will be applied to a particular ratio
of incoming connections. You can also manually set a mode using a socket
option.

This commit includes only basic probes. rrs@ has added quite an abundance
of probes in his TCP development work. He plans to commit those soon.

There are user-space programs which we plan to commit as ports. These read
the data from the log device and output pcapng files, and then let you
analyze the data (and metadata) in the pcapng files.

Reviewed by: gnn (previous version)
Obtained from: Netflix, Inc.
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D11085

Hide struct socket and struct unpcb from the userland.

Violators may define _WANT_SOCKET and _WANT_UNPCB respectively and
are not guaranteed for stability of the structures. The violators
list is the the usual one: libprocstat(3) and netstat(1) internally
and lsof in ports.

In struct xunpcb remove the inclusion of kernel structure and add
a bunch of spare fields. The xsocket already has socket not included,
but add there spares as well. Embed xsockbuf into xsocket.

Sort declarations in sys/socketvar.h to separate kernel only from
userland available ones.

PR: 221820 (exp-run)

Use counter(9) for PLPMTUD counters.

Remove unused PLPMTUD sysctl counters.

Bump UPDATING and FreeBSD Version to indicate a rebuild is required.

Submitted by: kevin.bowling@kev009.com
Reviewed by: jtl
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D12003

Listening sockets improvements.

o Separate fields of struct socket that belong to listening from
fields that belong to normal dataflow, and unionize them. This
shrinks the structure a bit.
- Take out selinfo's from the socket buffers into the socket. The
first reason is to support braindamaged scenario when a socket is
added to kevent(2) and then listen(2) is cast on it. The second
reason is that there is future plan to make socket buffers pluggable,
so that for a dataflow socket a socket buffer can be changed, and
in this case we also want to keep same selinfos through the lifetime
of a socket.
- Remove struct struct so_accf. Since now listening stuff no longer
affects struct socket size, just move its fields into listening part
of the union.
- Provide sol_upcall field and enforce that so_upcall_set() may be called
only on a dataflow socket, which has buffers, and for listening sockets
provide solisten_upcall_set().

o Remove ACCEPT_LOCK() global.
- Add a mutex to socket, to be used instead of socket buffer lock to lock
fields of struct socket that don't belong to a socket buffer.
- Allow to acquire two socket locks, but the first one must belong to a
listening socket.
- Make soref()/sorele() to use atomic(9). This allows in some situations
to do soref() without owning socket lock. There is place for improvement
here, it is possible to make sorele() also to lock optionally.
- Most protocols aren't touched by this change, except UNIX local sockets.
See below for more information.

o Reduce copy-and-paste in kernel modules that accept connections from
listening sockets: provide function solisten_dequeue(), and use it in
the following modules: ctl(4), iscsi(4), ng_btsocket(4), ng_ksocket(4),
infiniband, rpc.

o UNIX local sockets.
- Removal of ACCEPT_LOCK() global uncovered several races in the UNIX
local sockets. Most races exist around spawning a new socket, when we
are connecting to a local listening socket. To cover them, we need to
hold locks on both PCBs when spawning a third one. This means holding
them across sonewconn(). This creates a LOR between pcb locks and
unp_list_lock.
- To fix the new LOR, abandon the global unp_list_lock in favor of global
unp_link_lock. Indeed, separating these two locks didn't provide us any
extra parralelism in the UNIX sockets.
- Now call into uipc_attach() may happen with unp_link_lock hold if, we
are accepting, or without unp_link_lock in case if we are just creating
a socket.
- Another problem in UNIX sockets is that uipc_close() basicly did nothing
for a listening socket. The vnode remained opened for connections. This
is fixed by removing vnode in uipc_close(). Maybe the right way would be
to do it for all sockets (not only listening), simply move the vnode
teardown from uipc_detach() to uipc_close()?

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D9770

Hide struct inpcb, struct tcpcb from the userland.

This is a painful change, but it is needed. On the one hand, we avoid
modifying them, and this slows down some ideas, on the other hand we still
eventually modify them and tools like netstat(1) never work on next version of
FreeBSD. We maintain a ton of spares in them, and we already got some ifdef
hell at the end of tcpcb.

Details:
- Hide struct inpcb, struct tcpcb under _KERNEL || _WANT_FOO.
- Make struct xinpcb, struct xtcpcb pure API structures, not including
kernel structures inpcb and tcpcb inside. Export into these structures
the fields from inpcb and tcpcb that are known to be used, and put there
a ton of spare space.
- Make kernel and userland utilities compilable after these changes.
- Bump __FreeBSD_version.

Reviewed by: rrs, gnn
Differential Revision: D10018

Typo.

Renumber copyright clause 4

Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.

Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96

Merge projects/ipsec into head/.

Small summary
-------------

o Almost all IPsec releated code was moved into sys/netipsec.
o New kernel modules added: ipsec.ko and tcpmd5.ko. New kernel
option IPSEC_SUPPORT added. It enables support for loading
and unloading of ipsec.ko and tcpmd5.ko kernel modules.
o IPSEC_NAT_T option was removed. Now NAT-T support is enabled by
default. The UDP_ENCAP_ESPINUDP_NON_IKE encapsulation type
support was removed. Added TCP/UDP checksum handling for
inbound packets that were decapsulated by transport mode SAs.
setkey(8) modified to show run-time NAT-T configuration of SA.
o New network pseudo interface if_ipsec(4) added. For now it is
build as part of ipsec.ko module (or with IPSEC kernel).
It implements IPsec virtual tunnels to create route-based VPNs.
o The network stack now invokes IPsec functions using special
methods. The only one header file <netipsec/ipsec_support.h>
should be included to declare all the needed things to work
with IPsec.
o All IPsec protocols handlers (ESP/AH/IPCOMP protosw) were removed.
Now these protocols are handled directly via IPsec methods.
o TCP_SIGNATURE support was reworked to be more close to RFC.
o PF_KEY SADB was reworked:
- now all security associations stored in the single SPI namespace,
and all SAs MUST have unique SPI.
- several hash tables added to speed up lookups in SADB.
- SADB now uses rmlock to protect access, and concurrent threads
can do SA lookups in the same time.
- many PF_KEY message handlers were reworked to reflect changes
in SADB.
- SADB_UPDATE message was extended to support new PF_KEY headers:
SADB_X_EXT_NEW_ADDRESS_SRC and SADB_X_EXT_NEW_ADDRESS_DST. They
can be used by IKE daemon to change SA addresses.
o ipsecrequest and secpolicy structures were cardinally changed to
avoid locking protection for ipsecrequest. Now we support
only limited number (4) of bundled SAs, but they are supported
for both INET and INET6.
o INPCB security policy cache was introduced. Each PCB now caches
used security policies to avoid SP lookup for each packet.
o For inbound security policies added the mode, when the kernel does
check for full history of applied IPsec transforms.
o References counting rules for security policies and security
associations were changed. The proper SA locking added into xform
code.
o xform code was also changed. Now it is possible to unregister xforms.
tdb_xxx structures were changed and renamed to reflect changes in
SADB/SPDB, and changed rules for locking and refcounting.

Reviewed by: gnn, wblock
Obtained from: Yandex LLC
Relnotes: yes
Sponsored by: Yandex LLC
Differential Revision: https://reviews.freebsd.org/D9352

Use strlcpy and snprintf in netstat(1).

Expand inet6name() line buffer to NI_MAXHOST and use strlcpy/snprintf
in various places.

Reported by: Anton Yuzhaninov <citrin citrin ru>
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D8916

Print hostcache usage counts with TCP statistics.

PR: 196252
Submitted by: Anton Yuzhaninov <citrin+pr@citrin.ru>
MFC after: 3 weeks.

When calling netstat -Laptcp the local address values are not aligned
with the corresponding entry in the table header.
r295136 increased the value width from 14 to 32 without the corresponding
change to the table header. This commit adds the change to the table
header width.

MFC after: 3 days

Use NULL instead of 0 for pointers.

Also malloc will return NULL if it cannot allocate memory.

MFC after: 2 weeks.

Print running TCP connection counts with TCP statistics.

Increase max allowed backlog for listen sockets
from short to int.

PR: 203922
Submitted by: White Knight <white_knight@2ch.net>
MFC After: 4 weeks

Fix udp entry of `netstat -TW'.

Correct alignment of the addresses in the `netstat -aW' output.

Use a common subroutine to fetch and zero protocol stats instead of
duplicating roughly similar code for each protocol.

MFC after: 2 weeks

- Add -W flag support for network column in intpr() (-i flag) and
routepr() (-r flag). It is too narrow to show an IPv6 prefix
in most cases.

- Accept "local" as a synonym of "unix" in protocol family name.

- Show a prefix length in CIDR notation when name resolution failed in
netname().

- Make routename() and netname() AF-independent and remove
unnecessary typecasting from struct sockaddr.

- Use getnameinfo(3) to format L2 addr in intpr().

- Fix a bug which showed "Address" when -A flag is specfied in pr_rthdr().

- Replace cryptic GETSA() macro with SA_SIZE().

- Fix declarations shadowing local variables with the same names.

- Add more static, remove unused header files and variables.

MFC after: 1 week

Convert netstat to use libxo.

Obtained from: Phil Shafer <phil@juniper.net>
Ported to -current by: alfred@ (mostly), Kim Shrier
Formatting: marcel@
Sponsored by: Juniper Networks, Inc.

Burn bridges to FreeBSD 7.x IGMP stats.

Merge from projects/sendfile:

o Introduce a notion of "not ready" mbufs in socket buffers. These
mbufs are now being populated by some I/O in background and are
referenced outside. This forces following implications:
- An mbuf which is "not ready" can't be taken out of the buffer.
- An mbuf that is behind a "not ready" in the queue neither.
- If sockbet buffer is flushed, then "not ready" mbufs shouln't be
freed.

o In struct sockbuf the sb_cc field is split into sb_ccc and sb_acc.
The sb_ccc stands for ""claimed character count", or "committed
character count". And the sb_acc is "available character count".
Consumers of socket buffer API shouldn't already access them directly,
but use sbused() and sbavail() respectively.
o Not ready mbufs are marked with M_NOTREADY, and ready but blocked ones
with M_BLOCKED.
o New field sb_fnrdy points to the first not ready mbuf, to avoid linear
search.
o New function sbready() is provided to activate certain amount of mbufs
in a socket buffer.

A special note on SCTP:
SCTP has its own sockbufs. Unfortunately, FreeBSD stack doesn't yet
allow protocol specific sockbufs. Thus, SCTP does some hacks to make
itself compatible with FreeBSD: it manages sockbufs on its own, but keeps
sb_cc updated to inform the stack of amount of data in them. The new
notion of "not ready" data isn't supported by SCTP. Instead, only a
mechanical substitute is done: s/sb_cc/sb_ccc/.
A proper solution would be to take away struct sockbuf from struct
socket and allow protocols to implement their own socket buffers, like
SCTP already does. This was discussed with rrs@.

Sponsored by: Netflix
Sponsored by: Nginx, Inc.

Add -R to netstat to dump RSS/flow information.

This is intended to help in diagnostics and debugging of NIC and stack
flowid support.

Eventually this will grow another column (RSS CPU ID) but
that currently isn't cached in the inpcb.

There's also no clean flowtype -> flowtype identifier string. This is
the mbuf M_HASHTYPE_* values for RSS.

Here's some example output:

adrian@adrian-hackbox:~/work/freebsd/head/src % netstat -Rn | more
Active Internet connections
Proto Recv-Q Send-Q Local Address Foreign Address flowid ftype
tcp4 0 0 10.11.1.65.22 10.11.1.64.12409 29041942 2
udp4 0 0 127.0.0.1.123 *.* 00000000 0
udp6 0 0 fe80::1%lo0.123 *.* 00000000 0
udp6 0 0 ::1.123 *.* 00000000 0
udp4 0 0 10.11.1.65.123 *.* 00000000 0

Tested:

* amd64 system w/ igb NIC; local driver changes to expose RSS flowid in if_igb.

- Remove net.inet.tcp.reass.overflows sysctl. It counts exactly
same events that tcpstat's tcps_rcvmemdrop counter counts.
- Rename tcps_rcvmemdrop to tcps_rcvreassfull and improve its
description in netstat(1) output.

Sponsored by: Netflix
Sponsored by: Nginx, Inc.

Print the MD5 signature information introduced in r221023 in the
TCP statistics output.

MFC after: 3 weeks

Migrate struct carpstats to PCPU counters.

Migrate structs arpstat, icmpstat, mrtstat, pimstat and udpstat to PCPU
counters.

Use new macros to implement ipstat and tcpstat using PCPU counters.
Change interface of kread_counters() similar ot kread() in the netstat(1).

Prepare network statistics structures for migration to PCPU counters.
Use uint64_t as type for all fields of structures.

Changed structures: ahstat, arpstat, espstat, icmp6_ifstat, icmp6stat,
in6_ifstat, ip6stat, ipcompstat, ipipstat, ipsecstat, mrt6stat, mrtstat,
pfkeystat, pim6stat, pimstat, rip6stat, udpstat.

Discussed with: arch@

Use kvm_counter_u64_fetch() to fix obtaining ipstat and tcpstat from
kernel core files.

Sponsored by: Nginx, Inc.

Merge from projects/counters: TCP/IP stats.

Convert 'struct ipstat' and 'struct tcpstat' to counter(9).

This speeds up IP forwarding at extreme packet rates, and
makes accounting more precise.

Sponsored by: Nginx, Inc.

WARNS=6 compliance

Use pluralies() for "entry"/"entries".

- Updated TOE support in the kernel.

- Stateful TCP offload drivers for Terminator 3 and 4 (T3 and T4) ASICs.
These are available as t3_tom and t4_tom modules that augment cxgb(4)
and cxgbe(4) respectively. The cxgb/cxgbe drivers continue to work as
usual with or without these extra features.

- iWARP driver for Terminator 3 ASIC (kernel verbs). T4 iWARP in the
works and will follow soon.

Build-tested with make universe.

30s overview
============
What interfaces support TCP offload? Look for TOE4 and/or TOE6 in the
capabilities of an interface:
# ifconfig -m | grep TOE

Enable/disable TCP offload on an interface (just like any other ifnet
capability):
# ifconfig cxgbe0 toe
# ifconfig cxgbe0 -toe

Which connections are offloaded? Look for toe4 and/or toe6 in the
output of netstat and sockstat:
# netstat -np tcp | grep toe
# sockstat -46c | grep toe

Reviewed by: bz, gnn
Sponsored by: Chelsio communications.
MFC after: ~3 months (after 9.1, and after ensuring MFC is feasible)

Eliminate an unused parameter of static method igmp_stats_live_old().

MFC after: 1 month

Replace index() and rindex() calls with strchr() and strrchr().

The index() and rindex() functions were marked LEGACY in the 2001
revision of POSIX and were subsequently removed from the 2008 revision.
The strchr() and strrchr() functions are part of the C standard.

This makes the source code a lot more consistent, as most of these C
files also call into other str*() routines. In fact, about a dozen
already perform strchr() calls.

Fixed sockets display somewhat (-L, -T, -x, -Lx, with and without -A).
(I didn't try to fix negative TCP timers with -x.)

MFC after: 3 days

- Merge in OFED 1.5.3 from projects/ofed/head

Remove the advertising clause from UCB copyrighted files in usr.bin. This
is in accordance with the information provided at
ftp://ftp.cs.berkeley.edu/pub/4bsd/README.Impt.License.Change

Also add $FreeBSD$ to a few files to keep svn happy.

Discussed with: imp, rwatson

Restore the (state) and \n printout when not using -T.

Pointed out by: brucec@
MFC after: 3 weeks

Add new, per connection, statistics for TCP, including:
Retransmitted Packets
Zero Window Advertisements
Out of Order Receives

These statistics are available via the -T argument to
netstat(1).
MFC after: 2 weeks

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Show hostcache statistics.

Submitted by: Maxim Dounin

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

MFC r196797:

Add ARP statistics to the kernel and netstat.

In netstat -x, do not try to print out tcp timer status for udp sockets.

Add the ability to see TCP timers via netstat -x. This can be a useful
feature when you have a seemingly stuck socket and want to figure
out why it has not been closed yet.

No plans to MFC this, as it changes the netstat sysctl ABI.

Reviewed by: andre, rwatson, Eric Van Gyzen

Add ARP statistics to the kernel and netstat.

New counters now exist for:
requests sent
replies sent
requests received
replies received
packets received
total packets dropped due to no ARP entry
entrys timed out
Duplicate IPs seen

The new statistics are seen in the netstat command
when it is given the -s command line switch.

MFC after: 2 weeks
In collaboration with: bz

Correct a number of evolved problems with inp_vflag and inp_flags:
certain flags that should have been in inp_flags ended up in inp_vflag,
meaning that they were inconsistently locked, and in one case,
interpreted. Move the following flags from inp_vflag to gaps in the
inp_flags space (and clean up the inp_flags constants to make gaps
more obvious to future takers):

INP_TIMEWAIT
INP_SOCKREF
INP_ONESBCAST
INP_DROPPED

Some aspects of this change have no effect on kernel ABI at all, as these
are UDP/TCP/IP-internal uses; however, netstat and sockstat detect
INP_TIMEWAIT when listing TCP sockets, so any MFC will need to take this
into account.

MFC after: 1 week (or after dependencies are MFC'd)
Reviewed by: bz

Merge IGMPv3 and Source-Specific Multicast (SSM) to the FreeBSD
IPv4 stack.

Diffs are minimized against p4.
PCS has been used for some protocol verification, more widespread
testing of recorded sources in Group-and-Source queries is needed.
sizeof(struct igmpstat) has changed.

__FreeBSD_version is bumped to 800070.

Fix a printing problem when using the -L flag to netstat caused
by adding the -x flag earlier.

Submitted by: Anton Yuzhaninov
MFC after: 3 days

- Import the HEAD csup code which is the basis for the cvsmode work.

Use strlcpy() when we mean it.

Minimize changes CURRENT<->releng7.

Add ECN stats.

Fix build

Update the kernel to count the number of mbufs and clusters
(all types) used per socket buffer.

Add support to netstat to print out all of the socket buffer
statistics.

Update the netstat manual page to describe the new -x flag
which gives the extended output.

Reviewed by: rwatson, julian

Make a few messages more consistant with the others.

more style(9)

style(9)
+ kread is not a boolean, so check it as such
+ fix $FreeBSD$ Ids
+ denote copyrights with /*-
+ misc whitespace changes.

Fix printing of the number of syncache entries added.

Restore netstat -M functionality for most statistics on core dumps. In
general, when support was added to netstat for fetching data using sysctl,
no provision was left for fetching equivalent data from a core dump, and
in fact, netstat would _always_ fetch data from the live kernel using
sysctl even when -M was specified resulting in the user believing they
were getting data from coredumps when they actually weren't. Some specific
changes:
- Add a global 'live' variable that is true if netstat is running against
the live kernel and false if -M has been specified.
- Stop abusing the sysctl flag in the protocol tables to hold the protocol
number. Instead, the protocol is now its own field in the tables, and
it is passed as a separate parameter to the PCB and stat routines rather
than overloading the KVM offset parameter.
- Don't run PCB or stats functions who don't have a namelist offset if we
are being run against a crash dump (!live).
- For the inet and unix PCB routines, we generate the same buffer from KVM
that the sysctl usually generates complete with the header and trailer.
- Don't run bpf stats for !live (before it would just silently always run
live).
- kread() no longer trashes memory when opening the buffer if there is an
error on open and the passed in buffer is smaller than _POSIX2_LINE_MAX.
- The multicast routing code doesn't fallback to kvm on live kernels if
the sysctl fails. Keeping this made the code rather hairy, and netstat
is already tied to the kernel ABI anyway (even when using sysctl's since
things like xinpcb contain an inpcb) so any kernels this is run against
that have the multicast routing stuff should have the sysctls.
- Don't try to dig around in the kernel linker in the netgraph PCB routine
for core dumps.

Other notes:
- sctp's PCB routine only works on live kernels, it looked rather
complicated to generate all the same stuff via KVM. Someone can always
add it later if desired though.
- Fix the ipsec removal bug where N_xxx for IPSEC stats weren't renumbered.
- Use sysctlbyname() everywhere rather than hardcoded mib values.

MFC after: 1 week
Approved by: re (rwatson)

Import rewrite of IPv4 socket multicast layer to support source-specific
and protocol-independent host mode multicast. The code is written to
accomodate IPv6, IGMPv3 and MLDv2 with only a little additional work.

This change only pertains to FreeBSD's use as a multicast end-station and
does not concern multicast routing; for an IGMPv3/MLDv2 router
implementation, consider the XORP project.

The work is based on Wilbert de Graaf's IGMPv3 code drop for FreeBSD 4.6,
which is available at: http://www.kloosterhof.com/wilbert/igmpv3.html

Summary
* IPv4 multicast socket processing is now moved out of ip_output.c
into a new module, in_mcast.c.
* The in_mcast.c module implements the IPv4 legacy any-source API in
terms of the protocol-independent source-specific API.
* Source filters are lazy allocated as the common case does not use them.
They are part of per inpcb state and are covered by the inpcb lock.
* struct ip_mreqn is now supported to allow applications to specify
multicast joins by interface index in the legacy IPv4 any-source API.
* In UDP, an incoming multicast datagram only requires that the source
port matches the 4-tuple if the socket was already bound by source port.
An unbound socket SHOULD be able to receive multicasts sent from an
ephemeral source port.
* The UDP socket multicast filter mode defaults to exclusive, that is,
sources present in the per-socket list will be blocked from delivery.
* The RFC 3678 userland functions have been added to libc: setsourcefilter,
getsourcefilter, setipv4sourcefilter, getipv4sourcefilter.
* Definitions for IGMPv3 are merged but not yet used.
* struct sockaddr_storage is now referenced from <netinet/in.h>. It
is therefore defined there if not already declared in the same way
as for the C99 types.
* The RFC 1724 hack (specify 0.0.0.0/8 addresses to IP_MULTICAST_IF
which are then interpreted as interface indexes) is now deprecated.
* A patch for the Rhyolite.com routed in the FreeBSD base system
is available in the -net archives. This only affects individuals
running RIPv1 or RIPv2 via point-to-point and/or unnumbered interfaces.
* Make IPv6 detach path similar to IPv4's in code flow; functionally same.
* Bump __FreeBSD_version to 700048; see UPDATING.

This work was financially supported by another FreeBSD committer.

Obtained from: p4://bms_netdev
Submitted by: Wilbert de Graaf (original work)
Reviewed by: rwatson (locking), silence from fenner,
net@ (but with encouragement)

'netstat -A -p tcp' doesn't print the Socket but the Tcpcb pointer in the
first column.

o Fill the list of icmp types; make its size depend on ICMP_MAXTYPE.
o Print "unknown ICMP" instead of "(null)" if we don't have a description for a icmp type.

Based on code

Submitted by: Christoph Weber-Fahr
PR: misc/112126
MFC after: 2 weeks

Reap FIN_WAIT_2 connections marked SOCANTRCVMORE faster. This mitigate
potential issues where the peer does not close, potentially leaving
thousands of connections in FIN_WAIT_2. This is controlled by a new sysctl
fast_finwait2_recycle, which is disabled by default.

Reviewed by: gnn, silby.

Export tcps_rcvmemdrop available in 'struct tcpstat' with netstat(1).

Requested by: Tomasz Pilat <tomasz.pilat (at) axelspringer.pl>
Approved by: andre

- Achieve WARNS=3 by using sparse initializers or avoiding initializers at all.
- Fix a nlist initialization: it should be terminated by a NULL entry.
- Constify.
- Catch an unused parameter.

Tested on: i386 amd64 ia64

Achieve WARNS=2 by using uintmax_t to pass around 64-bit quantities,
including to printf(). Using uintmax_t is also robust to further
extensions in both the C language and the bitwidth of kernel counters.

Tested on: i386 amd64 ia64

Silence a warning on systems without carp(4).

Add limits on the number of elements in the sack scoreboard both
per-connection and globally. This eliminates potential DoS attacks
where SACK scoreboard elements tie up too much memory.

Submitted by: Raja Mukerji (raja at moselle dot com).
Reviewed by: Mohan Srinivasan (mohans at yahoo-inc dot com).

Add CARP (Common Address Redundancy Protocol), which allows multiple
hosts to share an IP address, providing high availability and load
balancing.

Original work on CARP done by Michael Shalayeff, with many
additions by Marco Pfatschbacher and Ryan McBride.

FreeBSD port done solely by Max Laier.

Patch by: mlaier
Obtained from: OpenBSD (mickey, mcbride)

Add __FBSDID. Replace local variable sin by sockin to not conflict with sin(3).
Use warnx() instead of warn() when error message is not of any interest. Add
prototypes.

Add SACK statistics to netstat.

Make netstat(1) more closely follow documented behaviour. If a TCP
socket in LISTEN state happens to be bound to an interface, it will
show up in netstat(1) output even without the -a switch.

As the definition of "sockets used by server processes" is a
difficult one to qualify with regards to UDP, do not change the
output behaviour for UDP sockets.

PR: bin/26359

Add missing dot and newline in a message.

Submitted by: Igor Sysoev <is (at) rambler-co.ru>
MFC after: 3 days

Output style nit.

Add the bad reset statistic (corresponds to tcp_input.c rev 1.235)

Do not print a warning about net.inet.pim.stats if errno is
ENOENT, because that means we do not have PIM in the kernel.

Submitted by: hmp
MFC after: 1 week

Print the correct ICMP statistics for "no return routes".

1. Add support for printing PIM-related statistics with
netstat -s -p pim

2. Print information about the bandwidth meters installed in the kernel with
netstat -g

Submitted by: Pavlin Radoslavov <pavlin@icir.org>

Back out support for RFC3514.

RFC3514 poses an unacceptale risk to compliant systems.

Implement support for RFC 3514 (The Security Flag in the IPv4 Header).
(See: ftp://ftp.rfc-editor.org/in-notes/rfc3514.txt)

This fulfills the host requirements for userland support by
way of the setsockopt() IP_EVIL_INTENT message.

There are three sysctl tunables provided to govern system behavior.

net.inet.ip.rfc3514:

Enables support for rfc3514. As this is an
Informational RFC and support is not yet widespread
this option is disabled by default.

net.inet.ip.hear_no_evil

If set the host will discard all received evil packets.

net.inet.ip.speak_no_evil

If set the host will discard all transmitted evil packets.

The IP statistics counter 'ips_evil' (available via 'netstat') provides
information on the number of 'evil' packets recieved.

For reference, the '-E' option to 'ping' has been provided to demonstrate
and test the implementation.

o Typo/Grammar fixes
o Added mini-function to correctly handle singular/plural
of words ending in 'ly'

Approved by: markm (mentor)
Not objected to by: -audit

Warns cleanups for netstat:
1) Include arpa/inet.h for ntohs.
2) Constness fixes.
3) Fix shadowing except for "sin" which shouldn't be in scope.
4) Remove register keyword.
5) Add missing initialsers to user defined structs.
5) Make prototype of netname6 globally visable.
6) Use right macros for printing syncache stats (even though entrie isn't
a word).

Widen struct sockbuf's sb_timeo member to int from short. With
non-default but reasonable values of hz this member overflowed,
breaking NFS over UDP.

Also, as long as I'm plowing up struct sockbuf ... Change certain
members from u_long/long to u_int/int in order to reduce wasted
space on 64-bit machines. This change was requested by Andrew
Gallatin.

Netstat and systat need to be rebuilt. I am incrementing
__FreeBSD_version in case any ports need to change.

Add the tcps_sndrexmitbad statistic, keep track of late acks that caused
unnecessary retransmissions.

- Make ip_rtaddr() global, and use it to look up the correct source
address in icmp_reflect().
- Two new "struct icmpstat" members: icps_badaddr and icps_noroute.

PR: kern/31575
Obtained from: BSD/OS
MFC after: 1 week

Add syncache statistics to netstat.

Don't print stray socket addresses (-A) with the listen queue display (-L).
Also, print socket's protocol with the -L.

Add netstat(1) knob to reset net.inet.{ip|icmp|tcp|udp|igmp}.stats.
For example, ``netstat -s -p ip -z'' will show and reset IP stats.

PR: bin/17338

remove warnings
remove superfluous declarations
make things more consistent

remove K&R support

revert removal of warning and K&R support

Requested by: bde

remove most of the warnings

add the option -S for printing port numbers symbolically but addresses
numerically. clean up the CFLAGS in Makefile.

Sync with recent KAME.
This work was based on kame-20010528-freebsd43-snap.tgz and some
critical problem after the snap was out were fixed.
There are many many changes since last KAME merge.

TODO:
- The definitions of SADB_* in sys/net/pfkeyv2.h are still different
from RFC2407/IANA assignment because of binary compatibility
issue. It should be fixed under 5-CURRENT.
- ip6po_m member of struct ip6_pktopts is no longer used. But, it
is still there because of binary compatibility issue. It should
be removed under 5-CURRENT.

Reviewed by: itojun
Obtained from: KAME
MFC after: 3 weeks

Add a -W flag that tells netstat not to truncate addresses even if they are
too long for the column they're printed in.

Move variable definitions out of netstat.h and into main.c.

Clean up some warnings.

MAXHOSTNAMELEN includes space for a NUL.
Don't roll our own version of trimdomain(), use the one in libutil.

Not objected to by: freebsd-audit

Count and show incoming UDP datagrams with no checksum.

sync with latest kame netstat. basically, more statistics

-permit protocol specific statistics at iflag,
only when either of sflag and "-f inet6" is specified.
-fix the indentation of default output

Specified by: Stephen McKay <syssgm@detir.qld.gov.au>

Reviewed and Confirmed by: Stephen McKay <syssgm@detir.qld.gov.au>

Add new option, -L that will listen the various listen queue lengths.

Reviewed by: Garrett Wollman <wollman@khavrinen.lcs.mit.edu>,
Clive Lin <clive@GnatS.CirX.ORG>

udp IPv6 support, IPv6/IPv4 tunneling support in kernel,
packet divert at kernel for IPv6/IPv4 translater daemon

This includes queue related patch submitted by jburkhol@home.com.

Submitted by: queue related patch from jburkhol@home.com
Reviewed by: freebsd-arch, cvs-committers
Obtained from: KAME project

Cleanup towards -Wall

$Id$ -> $FreeBSD$

Don't truncate port numbers.

Submitted by: ru

Fix fstat compatibility by printing the address of the tcppcb instead of
that of the regular socket pcb for TCP sockets.

PR: bin/9963
Submitted by: Ruslan Ermilov <ru@ucb.crimea.ua>

Fixed printf format errors.

o Use snprintf over sprintf.
o Use strncpy correctly.
o Use enough buffer for line.

Inspired by or Obtained from: Similar changes in OpenBSD

Make "netstat -s" display stats on fast-forwarded packets.
Obtained from: NetBSD code

mbuf, inet, and unix modules no longer read kvm.

Print out the two new fields in the ICMP stats. While we're at it, also
convert icmp_stats() to use sysctl(3) to retrieve the information
rather than kvm. This makes it easy to also print whether ICMP address
mask responses are enabled, so do so.

Revert $FreeBSD$ to $Id$

Print out the new uninteresting-multicast-packets statistic.

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

TCP statistics are unsigned counters.

Tor Egge reports counter wrap and requests and update to quad_t sized
counters, which is also a good thing to do, but I'm unhappy about adding
two more instructions into the code path every time we doink a counter.

Maybe with or after the Lite2 merge...
Submitted by: Tor Egge <Tor.Egge@idt.ntnu.no>

Eliminate unnecessary include of <sys/mbuf.h>. PCBs aren't stored in mbufs
any more.

Try to make netstat more smart about port numbers.
Feedback most welcome.

Print PCB hash miss statistic.

Code cleanup: remove unused variables, use correct *printf format
specifiers (some unsigned values were printed as signed, some longs
were printed as ints), and place parentheses around assignments in
if statements.

Move #include of queue.h before #include of socketvar.h in preparation for
struct socket changes.

Add a counter for the number of times the listen queue was overflowed to
the tcpstat structure. (netstat -s)
Reviewed by: wollman
Obtained from: Steves, TCP/IP Ill. vol.3, page 189

tidy up the domain name trimming code, and move it to a single place
rather than having the same bit of code duplicated in three places,
each with their own static copy of the host's local name.

Merge in Lite-2 changes.

Import netstat from 4.4Lite-2 - this version eliminates some of the
severely bogus mask guessing in the netname code, and also reports the
mask length of the route (like linux's netstat, but in a much more compact
form)

The current code is doing bogus things like shifting c-class subnets by
4 bits and looking them up, causing a DNS lookup of 0x0C0CBxxx which
takes an eternity to fail.

Change .8s port name restriction to .15s, modern tcp port
have longer names, check /etc/services.
kerberos_master triggers this bug: it is undistiguishable from
simple kerberos for .8s

- #if defined(TF_NEEDSYN) && defined(TF_NEEDSYN)
+ #if defined(TF_NEEDSYN) && defined(TF_NEEDFIN)

In TCP statistics, display new statistic about how many resends were
initiated as a result of the Path MTU discovery process.

Output statistics as unsigned in the -w section.
Look at error return of kread() and stop on error.
Fix warning in kread() to not output "kvm_read:" twice.
Killed PCB cache misses stat as we no longer have it.

Add "connections dropped by persist timeout" statistic.

Correct spelling mistake: upcated -> updated.

Print out the statistics for cache updates of rtt, rttvar, and ssthresh.

That's what I get for doing things in a hurry...applied up-to-date patch
to this file rather than old one (before I renamed the structure element).

Oops, remove some chaff from previous commit.

Updated netstat to grok 4.4 QUEUE macros now being used to like the PCBs
together.

Make netstat report the correct state for T/TCP.

Submitted by: Andras Olah <olah@cs.utwente.nl>

Document router advertisement and solicitation messages in ICMP input
histogram.

BSD 4.4 Lite Usr.bin Sources