libfetch, fetch: Stop recommending the use of ca_root_nss.

MFC after: 3 days
Reviewed by: kevans, emaste
Differential Revision: https://reviews.freebsd.org/D42119

Remove $FreeBSD$: two-line nroff pattern

Remove /^\.\\"\n\.\\"\s*\$FreeBSD\$$\n/

There's no PEM(3) anywhere around; delete reference.

Remove support for SSLv3 from fetch(3).

Support for SSLv3 was already removed from OpenSSL (r361392).

Differential Revision: https://reviews.freebsd.org/D24947

fetch(3): Add SOCKS5 support

This change adds SOCKS5 support to the library fetch(3) and updates the man
page.

Details: Within the fetch_connect() function, fetch(3) checks if the
SOCKS5_PROXY environment variable is set. If so, it connects to this host
rather than the end-host. It then initializes the SOCKS5 connection in
accordance with RFC 1928 and returns the resulting conn_t (file descriptor)
for usage by the regular FTP/HTTP handlers.

Design Decision: This change defaults all DNS resolutions through the proxy
by sending all IPs as hostnames. Going forward, another feature might be to
create another environmental variable to toggle resolutions through the
proxy or not..

One may set the SOCKS5_PROXY environment variable in any of the formats:

SOCKS5_PROXY=proxy.example.com
SOCKS5_PROXY=proxy.example.com:1080
SOCKS5_PROXY=192.0.2.0
SOCKS5_PROXY=198.51.100.0:1080
SOCKS5_PROXY=[2001:db8::1]
SOCKS5_PROXY=[2001:db8::2]:1080

Then perform a request with fetch(1).

(note by kevans)
I've since been informed that Void Linux/xbps has a fork of libfetch that
also implements SOCKS5. I may compare/contrast the two in the mid-to-near
future.

Submitted by: Farhan Khan <farhan farhan codes>
Differential Revision: https://reviews.freebsd.org/D18908

Document fetchReqHTTP().

Submitted by: Farhan Khan <khanzf@gmail.com>
Reviewed by: 0mp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D18788

Fix bunch of .Xrs.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Update fetch.1 and fetch.3 to reflect libfetch's actual use of CA bundles

Reviewed by: wblock
Approved by: wblock
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D5558

Use .netrc for HTTP sites and proxies, not just FTP.

PR: 193740
Submitted by: TEUBEL György <tgyurci@gmail.com>
MFC after: 1 week

Remove defunct SSLv2 support from fetch(1) and fetch(3).

As pointed out by several people, r273114 was incorrect: it unconditionally
disabled everything except TLS 1.0. Replace it with a more carefully
wrought patch:

- Switch the default for SSLv3 from on to off
- Add environment variables to control TLS 1.1 and 1.2
- In verbose mode, report which version is used
- Update the man page to reflect these changes.

MFC after: 1 week

Use Mt macro to properly format mailto links (patch from Franco Fichtner <franco@lastsummer.de> for Dragonfly)

Reviewed by: des
Approved by: des
Obtained from: Dragonfly
MFC after: 1 week

If HTTP_USER_AGENT is defined but empty, don't send User-Agent at all.

PR: 184507
Submitted by: jbeich@tormail.org (with modifications)
MFC after: 1 week

Include an Accept header in requests.

PR: kern/180917
MFC after: 1 week

Implement certificate verification, and many other SSL-related
imrovements; complete details in the PR.

PR: kern/175514
Submitted by: Michael Gmelin <freebsd@grem.de>
MFC after: 1 week

Remove trailing whitespace per mdoc lint warning

Disussed with: gavin
No objection from: doc
Approved by: joel
MFC after: 3 days

latin1 -> utf8

Update copyright dates and strip my middle name.

Bump date.

Think first, commit second.

1. Allow the caller to select active mode.
2. Fix the envar logic so it *always* overrides the caller's flags.
3. Document the change from active to passive.

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

insta-mfh r203028 (doc update)

Document HTTP digest authentication support.

Mention some prominent past contributors: Hajimu Umemoto (ipv6), Henry
Whincup (https), Jukka Ukkonen (if-modified-since) and Jean-François
Dockes (digest auth)

Add support for HTTP 1.1 If-Modified-Since behavior.

fetch(1) accepts a new argument -i <file> that if specified will cause
the file to be downloaded only if it is more recent than the mtime of
<file>.

libfetch(3) accepts the mtime in the url structure and a flag to
indicate when this behavior is desired.

PR: bin/87841
Submitted by: Jukka A. Ukkonen <jau@iki.fi> (partially)
Reviewed by: des, ru
MFC after: 3 weeks

- Import the HEAD csup code which is the basis for the cvsmode work.

s/MAXPATHLEN/PATH_MAX/ to reflect five-year old change to the code :)

Submitted by: Joerg Sonnenberger <joerg@britannica.bec.de>
MFC after: 2 weeks

Document NO_PROXY / no_proxy.

MFC after: 3 weeks

Enable FTP_COMBINE_CWDS.

Add back the original behavior of changing the entire directory path at
once (CWD a/b/c vs. 3 CWDs). If an error occurs, we fall back to the default
method of a single CWD per directory element. Since this is technically
a violation of the basic FTP RFC, this behavior is under a compile-time
option FTP_COMBINE_CWDS and is off by default. It should work with most
Unix-based FTP daemons and can save latency.

MFC after: 2 weeks

Fix misspelling in Poul-Henning Kamp's email address under AUTHORS, from
pkh@ to phk@.

Approved by: ade

Expand contractions.

Expand *n't contractions.

Update copyright years.

Mechanically kill hard sentence breaks.

I don't normally use my middle name, so remove it from attributions in
man pages (though not from copyright notices). While I'm here, add email
addresses where appropriate.

mdoc(7): Use the new feature of the .In macro.

Add an EXAMPLES section.

PR: 43899
Reviewed by: des, ru

Assorted mdoc(7) fixes.

Approved by: re (blanket)

Add and document support for a FETCH_BIND_ADDRESS environment variable
specifying a local address to bind sockets to. Caveat: lightly tested.

PR: bin/37572

Document .netrc support.

english(4) police.

Implement and document support for an HTTP_REFERER environment variable.

PR: 28171
Submitted by: Andre Albsmeier <andre.albsmeier@mchp.siemens.de>
Approved by: re (bmah)
MFC after: 1 week

Fix two typos.

mdoc(7) police: Use the new .In macro for #include statements.

Remove whitespace at EOL.

mdoc(7) police: remove extraneous .Pp before and/or after .Sh.

mdoc(7) police: fixed markup and typo.

Implement a HTTP_USER_AGENT environment variable.

PR: 27669
Submitted by: Eoin Lawless <eoin@maths.tcd.ie>

Document the authentication callback interface.
Update RFC references (should have done that ages ago...)

Apply 'const' liberally.

Fix some other minor glitches.

Grammar police: "its", not "it's", is the possessive form of "it".

Use high port range by default, and replace the 'h' option with an 'l' option
that forces the ftp code to use the low (default) port range instead.

Support lower-case versions of the proxy environment variables.

PR: bin/25494

mdoc(7) police: fixed broken references.

Prepare for mdoc(7)NG.

The user name for anonymous ftp is now "anonymous".
Remove the period after the last man page reference.
Add a reference to RFC1635, and sort the RFC references by number.

Document FTP_LOGIN.

Somewhere along the line, I misunderstood the whole FTP_PASSIVE_MODE debate
and had libfetch selecting passive mode even when FTP_PASSIVE_MODE was not
set at all, which is really quite surprising unless you know about it. So
change it to the agreed default behaviour of selecting passive mode if
FTP_PASSIVE_MODE is set, but not "no".

Remove some obsolete comments.

log

Spell the des's name correctly.

Use Fx macro wherever possible.

Document FTP_PASSIVE_MODE and FTP_PASSWORD.

PR: docs/20626
Submitted by: nik

Document recent changes in URL parsing and proxy handling

Make the ENVIRONMENT section more legible, and mention that HTTP_AUTH is
colon-separated.

Document the FTP_PROXY and HTTP_PROXY variables better.

Document the proxy support better.

Introduce fetchXGet*(), which combine the functionalities of fetchGet*() and
fetchStat*(). In most cases, either fetchGet*() or fetchXGet*() is a wrapper
around the other; in all cases, calling fetchGet*() is identical to calling
fetchXGet*() with the second argument set to NULL.

Fully document the HTTP authorization feature.
Update the BUGS section to reflect the HTTP rewrite.

Document that basic authorization now kinda works.

Add (and document) fetchMakeURL()

Dump com_err, it's a stinking crock of shit.

Document struct url.
Document the default values for fetchStat*().

Fix hard sentence break.

Submitted by: sheldonh

Reorganize some of the http code and split it into more functions.
Implement fetchStatHTTP().
Unbungle struct url, and add fetchFreeURL().
Document it.

Supply only one author name per instance of %A, as per mdoc.samples(7).

PR: 18465
Submitted by: Kazu TAKAMUNE <takamune@avrl.mei.co.jp>

Introduce .Lb macro to libfetch manpage
Sort .Nm values
Change first column width in errors list (table look much better now)
Remove redundant comma

Remove single-space hard sentence breaks. These degrade the quality
of the typeset output, tend to make diffs harder to read and provide
bad examples for new-comers to mdoc.

Patches I've had lying around for several months:
* Add the 'h' ftp flag (allocate local port in high range)
* Add the 'd' flag (use direct connection even if proxy is defined)
* Make sure flags != NULL before calling strchr().
* Minor changes to some comments.

$Id$ -> $FreeBSD$

Fixed prototype for fetchParseURL().

Implement and document file list retrieval.

Implement and document fetchStatFTP.
Update description of struct url_stat in the man page.
Clean up error handling in ftp.c.

Add verbose flag, and support functions.
Brucify the Makefile.
Differentiate atime and mtime in fetch*Stat().
Fix a few pointer bugs.
Tweak some error messages.
Don't #include sys/param.h and stdio.h in fetch.h.
Document that sys/param.h and stdio.h must be #included before fetch.h.

Second of a series of cleanups to libfetch.

This commit introduces the following features:

a) the fetchStat*() functions, which return meta-information for a
document, such as size, modification time, etc.

b) the use of the com_err(3) facilities to report errors.

It also fixes a bunch of style bugs and a few logic bugs and somewhat
improves the man page.

Changed files, in alphabetical order:

Makefile:
Don't generate macros in {ftp,http}err.c.

Generate category fields for the error message lists.

Compile the error table.

Install fetch_err.h along with fetch.h.

common.c:
Remove the _netdb_errstring() macro, and add FETCH_ERR_NETDB to the
error code in the _netdb_seterr() macro.

Add categories to the _netdb_errlist table.

Report errors through the Common Error library.

common.h:
Add the DEBUG macros.

Add prototype for fetchConnect().

Remove the prototype for _fetch_errstring(), which is local to common.c

Add a categroy field to struct fetcherr, and define constants for
error categories.

Define macros for _{url,netdb,ftp,http}_seterr().

errors.et: (new file)
List error categories.

fetch.3:
Document the fetchStat*() functions.

Move the "unimplemented functionality" comments from NOTES to BUGS.

Document that applications which use libfetch must also use
libcom_err, and list existing error codes.

Undocument fetchLastErr{Code,String}.

Remove the (empty) DIAGNOSTICS section.

Mention Eugene Skepner in the AUTHORS section.

fetch.c:
Move the DEBUG macros to common.c

Add fetchStat() and fetchStatURL().

Generate error messages for URL parser errors, and fix a minor bug
in the parser.

Use 'struct url' instead of 'url_t'.

Remove fetchLastErr{Code,String}.

fetch.h:
Use 'struct url' instead of 'url_t', and remove the typedef.

Define struct url_stat (used by fetchStat()).

Add prototypes for fetchStat*().

Remove the declarations for fetchLastErr{Code,String}.

Include fetch_err.h.

fetch_err.et: (new file)
Error table for libfetch.

file.c:
Add fetchStatFile().

Use 'struct url' instead of 'url_t'.

ftp.c:
Add fetchStatFTP().

Use 'struct url' instead of 'url_t'.

Don't use fetchLastErrCode.

ftp.errors:
Add categories to all error messages.

http.c:
Add fetchStatHTTP().

Use 'struct url' instead of 'url_t'.

Don't use fetchLastErr{Code,Text}.

http.errors:
Add categories to all error messages.

Prompted by: jkh and Eugene Skepner
Numerous sugestions from: Garett Wollman and Eugene Skepner

First of a series of cleanups to libfetch. Changed files, in
alphabetical order:

Makefile:
Add common.c to SRCS.

Make debugging easier by making 'CFLAGS += -DNDEBUG' conditional on DEBUG

Don't declare struct {ftp,http}err in {ftp,http}err.c; use struct fetcherr
instead.

README:
Remove the todo list, which is out of date anyway.

common.c: (new file)
Gather utility functions in this file.

Merge the error reporting functions intp _fetch_errstring(),
_fetch_seterr() and _fetch_syserr().

Set fetchLastErrCode and fetchLastErrText appropriately when fetchConnect
fails.

common.h: (new file)
Gather internal prototypes and structures in this files.

fetch.3:
Undocument fetchFreeURL().

Document a few more known bugs.

Document fetchLastErrCode and fetchLastErrText.

fetch.c:
Add descriptive comments to all functions that lacked them.

Move fetchConnect() to common.c.

Obviate the need for fetchFreeURL(), and remove it.

fetch.h:
Modify struct url_t so the document part is at the end.

ftp.c:
Remove code that is duplicated elsewhere.

http.c:
Remove code that is duplicated elsewhere.

Prompted by: jkh

Bring the man page more or less up to date with the code.

Imported libfetch into the tree. It compiles, but there's still some
work to do. I especially need help with the man page.