Copy stable/10@r272459 to releng/10.1 as part of
the 10.1-RELEASE process.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Mechanically substitute flags from historic mbuf allocator with
malloc(9) flags within sys.

Exceptions:

- sys/contrib not touched
- sys/mbuf.h edited manually

Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs.

This means that their use is restricted to a single C file.

Node constructor methods are supposed to be called in syscall
context always. Convert nodes to consistently use M_WAITOK flag
for memory allocation.

Reviewed by: julian

Remove some more alignment constraints.

Make ng_ppp fulfill upper protocol stack layers alignment requirements
on platforms with strict alignment constraints.
This fixes kernel panics on arm and probably other architectures.

PR: sparc64/80410

Remove strict limitation on minimal multilink MRRU. RFC claims that MRRU
of 1500 must be supported, but allows smaller values to be negotiated.
Enforce specified MRRU for outgoing frames.

MFC after: 2 weeks

Retire the MALLOC and FREE macros. They are an abomination unto style(9).

MFC after: 3 months

Do not use bcmp() to compare two bytes with constants.

Improve multilink receive performance by netgraph item reuse.

Improve multilink xmit performance by netgraph item reuse.

Improve multilink receive performance with fragment headers preallocation.

Fix bundle xmit octets stats for packet-split operation mode.

Fix typo which brokes VJ decompression
when VJC negotiated in only one direction.

Approved by: re (bmah), glebius (mentor)

Add 64bit statistic counters to the ng_ppp node.
64bit counters are needed to simplify traffic accounting and
reduce system load at the big PPP concentrators.

Approved by: re (rwatson), glebius (mentor)

This patch improves fine-grained locking for the ng_ppp node.
Till now node's transmit path was completely unprotected
and so wasn't thread safe in multilink mode. It's receive path was
declared as WRITER as the simpliest protection method but it
reduces performance when compression or encryption enabled.

Approved by: re (rwatson), glebius (mentor)

No need to update link queue stats when round-robin algorithm enabled.

Approved by: glebius (mentor)

Added m_tag_copy_chain() call to copy original outgoing packet tags to all of
it's fragments.

Reviewed by: archie
Approved by: glebius (mentor)

Optimized packet distribution plan for the equal links case. Do not
split packet on fragments smaller then MP_MIN_FRAG_LEN to reduce total
overhead.

Reviewed by: archie
Approved by: glebius (mentor)

- Changed sequence numbers processing to avoid incorrect timeout waiting
when one of links is inactive and have stale sequence number. To avoid
this sequence numbers of all links are getting updated on every
successful packet reassembling.
- ng_ppp_bump_mseq function created to simplify code.
- ng_ppp_frag_drop function separated from ng_ppp_frag_process to
simplify code.

Reviewed by: archie
Approved by: glebius (mentor)

- Fixed mistakes in latency and xmitBytes calculation math
which lead to ineffective multilink packet distribution plans.
- Changed bytesInQueue calculation math to have more precise information
about links utilization.
- Taken rough account of the link overhead. Better way to do it could be to
get exact overhead from user-level, but I have not done it to keep
binary compatibility.

Reviewed by: archie
Approved by: glebius (mentor)

- Create ng_ppp_bypass() function, that prepares a packet
with bypass header, to send it out to userland.
- Use ng_ppp_bypass() in ng_ppp_proto_recv().
- Use ng_ppp_bypass() in ng_ppp_comp_recv() and in
ng_ppp_crypt_recv() if compression or encryption is
disabled, respectively.
- Any LCP packet goes directly to ng_ppp_bypass(), instead
of passing through PPP stack.
- Any non-LCP packet on disabled link is discarded. This
is behavior defined in RFC.

Submitted by: Alexander Motin <mav alkar.net>

A less draconian fix to the build.

Temporarily comment out the KASSERT that broke the kernel build.

Revise the ng_ppp(4) node, so that code flow is more clear. All non-link
hooks get their per hook rcvdata methods, and all functions are organized
corresponding to protocol stack model.

Submitted by: Alexander Motin <mav alkar.net>
Reviewed by: archie, julian

Before this commit, if the compression is enabled the, ng_ppp(4)
node would send every outgoing frame to the "compress" hook.
Packets received on the "compress" hook were expected to be
compressed and PROT_COMPD tag was put on them unconditionally.

After this commit an alternative compression mode can be set.
In this mode the node doesn't put the PROT_COMPD, the compressor
should put it itself. This is important for such kind of
compressors, that can submit uncompressed frames.

Before this commit, if the decompression is enabled, the ng_ppp(4)
node would send and incoming frame to the "decompress" hook
only if it has the PROT_COMPD proto tag on it.

After this commit an alternative decompression mode can be set.
In this mode the node sends all the incoming packets to the
decompression hook. This is important for such kind of compressors
that need uncompressed packets too, to keep their library in sync.

These new features will be used in new version of mpd4, and in new
compressor nodes.

Submitted by: Alexander Motin <mav alkar.net>

Refactor places where ng_package_data() failed.

Suggested & reviewed by: archie

- Catch up with new interface to ng_package_data().
- Handle errors from ng_package_data().

Utilize callout_pending() macro

/* -> /*- for license, minor formatting changes

- Use ng_callout() instead of timeout(9).
- Do not put/remove node references, since this no longer
needed.
- Remove timerActive flag, use callout flags.
- Schedule next callout after doing current one.

Reviewed by: archie
Approved by: julian (mentor)

Disable protocol field compression on the inner PPP frame when also doing
normal PPP compression, as a workaround for certain (arguably) broken
Linux PPP implementations that can't handle this particular case.

MFC after: 1 week

Do not change link[n].conf.latency for internal usage but have
link[n].latency calculated from user supplied value.
This prevents repeated NGM_PPP_SET_CONFIG/NGM_PPP_GET_CONFIG
from failing because of link[n].conf.latency being out of range.

Reviewed by: archie
Approved by: pjd (mentor)

Use qsort_r() instead of qsort() when sorting links by latency
This helps us to remove a global variable and a mutex protecting it.

Reviewed by: rwatson
Approved by: julian (mentor)

Add ng_ppp_latencies_mtx, a global mutex to protect the latency list.
Note that the table is a hack, and so is this mutex.

Reviewed by: glebius

Having moved metadata usage to mbuf tags, remove code that supports
the old way of doing it.

Submitted by: Gleb Smirnoff <glebius@cell.sick.ru>

Switch to using C99 sparse initialisers for the type methods array.
Should make no binary difference.

Submitted by: Gleb Smirnoff <glebius@cell.sick.ru>
Reviewed by: Harti Brandt <harti@freebsd.org>
MFC after: 1 week

Deprecate machine/limits.h in favor of new sys/limits.h.
Change all in-tree consumers to include <sys/limits.h>

Discussed on: standards@
Partially submitted by: Craig Rodrigues <rodrigc@attbi.com>

Fix a use-after-free bug that could cause multi-link fragment reassembly to
fail for a long time (until the incoming sequence numbers wrapped around).

Reported by: Matthew Impett <mimpett@Glue.umd.edu>
MFC after: 3 days

Back out M_* changes, per decision of the TRB.

Approved by: trb

Remove M_TRYWAIT/M_WAITOK/M_WAIT. Callers should use 0.
Merge M_NOWAIT/M_DONTWAIT into a single flag M_NOWAIT.

o Untangle the confusion with the malloc flags {M_WAITOK, M_NOWAIT} and
the mbuf allocator flags {M_TRYWAIT, M_DONTWAIT}.
o Fix a bpf_compat issue where malloc() was defined to just call
bpf_alloc() and pass the 'canwait' flag(s) along. It's been changed
to call bpf_alloc() but pass the corresponding M_TRYWAIT or M_DONTWAIT
flag (and only one of those two).

Submitted by: Hiten Pandya <hiten@unixdaemons.com> (hiten->commit_count++)

Use intptr_t to fix various sizeof(int) != sizeof(void *) warnings.

if you have taken the mbuf out of the message object, then if you pass
the object to someone else, you need to put the mbuf back into it first..

Fix GCC warnings caused by initializing a zero length array. In the process,
simply things a bit by getting rid of 'struct ng_parse_struct_info' which
was useless because it only contained one field.

MFC after: 2 weeks

Fix bugs where the ng_ppp node could transmit PPP frames whose length
exceeded the peer's configured MRU or MRRU.

MFC after: 1 week

Forced commit; the previous bug-fix commit was..

Submitted by: John Wiersema <J.Wiersema@flarion.com>

Fix another bug in handling of multi-link sequence numbers.

MFC after: 1 week

Update to C99, s/__FUNCTION__/__func__/,
also don't use ANSI string concatenation.

Fix some memory leaks
Add memory leak detection assitance.

Part 2 of the netgraph rewrite.
This is mostly cosmetic changes, (though I caught a bug or two while
makeing them)
Reviewed by: archie@freebsd.org

Rewrite of netgraph to start getting ready for SMP.
This version is functional and is aproaching solid..
notice I said APROACHING. There are many node types I cannot test
I have tested: echo hole ppp socket vjc iface tee bpf async tty
The rest compile and "Look" right. More changes to follow.
DEBUGGING is enabled in this code to help if people have problems.

Divorce the kernel binary ABI version number from the message
format version number. (userland programs should not need to be
recompiled when the netgraph kernel internal ABI is changed.

Also fix modules that don;t handle the fact that a caller may not supply
a return message pointer. (benign at the moment because the calling code
checks, but that will change)

Use "node->ID" for the node's ID, instead of "(long)node".

Reported by: julian

Reviewed by: Archie@freebsd.org
This clears out my outstanding netgraph changes.
There is a netgraph change of design in the offing and this is to some
extent a superset of soem of the new functionality and some of the old
functionality that may be removed.

This code works as before, but allows some new features that I want to
work with and evaluate. It is the basis for a version of netgraph
with integral locking for SMP use.

This is running on my test machine with no new problems :-)

Add the use of M_ZERO to netgraph.

Submitted by: josh@zipperup.org
Submitted by: Robert Drehmel <robd@gmx.net>
Submitted by: archie
Approved by: archie

In preparation for deprecating CIRCLEQ macros in favor of TAILQ
macros which provide the same functionality and are a bit more
efficient, convert use of CIRCLEQ's in netgraph PPP code to TAILQ's.

Reviewed by: Archie Cobbs <archie@dellroad.org>

More complete fix for multi-link sequence number handling bugs.
Add a new control message for querying the sequence number state.

Fix bug in handling of multi-link sequence numbers.

Reported by: Becca Anderson <becca@worldint.com>

Remove unnecessary #include's as reported by phk's script.

Allocate all memory (including within node constructors) with M_NOWAIT
instead of M_WAITOK, to allow for maximum flexibility.

Oops, previous commit fixed a bug that was already fixed before.
Back it out.

Fix bug where bundle-level receive statistics were not getting updated.

Fix bug where receive statistics for the bundle were not getting updated.

Back out the previous change to the queue(3) interface.
It was not discussed and should probably not happen.

Requested by: msmith and others

Change the way that the queue(3) structures are declared; don't assume that
the type argument to *_HEAD and *_ENTRY is a struct.

Suggested by: phk
Reviewed by: phk
Approved by: mdodd

Fix broken multi-link fragment reassembly algorithm.
Add hook for IPv6. Misc cleanups.

PR: kern/16335

Two simple changes to the kernel internal API for netgraph modules,
to support future work in flow-control and 'packet reject/replace'
processing modes.

reviewed by: phk, archie

Fix a few obscure memory leaks.

Fix bugs in the MP fragment reassembly code that can cause a panic.

Remove a bunch of un-needed includes.
Submitted by: phk@freebsd.org

Add two new generic control messages, NGM_ASCII2BINARY and
NGM_BINARY2ASCII, which convert control messages to ASCII and back.
This allows control messages to be sent and received in ASCII form
using ngctl(8), which makes ngctl a lot more useful.

This also allows all the type-specific debugging code in libnetgraph
to go away -- instead, we just ask the node itself to do the ASCII
translation for us.

Currently, all generic control messages are supported, as well as
messages associated with the following node types: async, cisco,
ksocket, and ppp.

See /usr/share/examples/netgraph/ngctl for an example of using this.

Also give ngctl(8) the ability to print out incoming data and
control messages at any time. Eventually nghook(8) may be subsumed.

Several other misc. bug fixes.

Reviewed by: julian

Change the prototype of the strto* routines to make the second
parameter a char ** instead of a const char **. This make these
kernel routines consistent with the corresponding libc userland
routines.

Which is actually 'correct' is debatable, but consistency and
following the spec was deemed more important in this case.

Reviewed by (in concept): phk, bde

Fix bug where hook pointers were not getting updated on disconnection.

More bug fixes.

Move handling of the address and control fields into the PPP node;
they belong there because they are device independent.
Also some other misc. fixes.

Fix several bugs found in the first bit of testing.

Move isfoo() and friends to the newly created sys/ctype.h.

Urged by: bde

Simplify checking/parsing of strings using strtoul(), isdigit(), etc.

Fix some bugs in MP allocation routine when links are non-equivalent.

Add typedefs for node methods
Suggested by phk.

Revamped and more useful PPP node type, supporting multi-link PPP directly.
This is one piece of the grand unified PPP daemon concept, whereby using
netgraph nodes enables PPP data to be handled completely in kernel land,
while leaving negotiation/control to be handled by a single user land
daemon, no matter what the link type(s).

This is a safety checkin only; it compiles, but is utterly untested.

Concept reviewed by: julian, brian

Whistle's Netgraph link-layer (sometimes more) networking infrastructure.
Been in production for 3 years now. Gives Instant Frame relay to if_sr
and if_ar drivers, and PPPOE support soon. See:
ftp://ftp.whistle.com/pub/archie/netgraph/index.html
for on-line manual pages.

Reviewed by: Doug Rabson (dfr@freebsd.org)
Obtained from: Whistle CVS tree