Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Fix libfetch out of bounds read.

Approved by: so
Security: SA-21:15.libfetch
Security: CVE-2021-36159

Improve URL parsing. In particular, convert scheme and host to lowercase.

MFC after: 1 week

Support proxying FTP over HTTPS, not just HTTP.

There is probably a PR for this, but I can't find this, or remember who
submitted it. The patch got lost in the noise of another that wasn't
ready to commit.

MFC after: 3 days

Fix an inverted conditional in the netrc code, which would ignore the
value of $HOME and always use the home directory from the passwd
database, unless $HOME was unset, in which case it would use (null).

While there, clean up handling of netrcfd and add debugging aids.

MFC after: 3 weeks

Use __VA_ARGS__ to simplify the DEBUG macro.

MFC after: 3 weeks

SPDX: use the Beerware identifier.

lib: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Refactor fetch_connect() and fetch_bind() to improve readability and avoid
repeating the same DNS lookups.

MFC after: 3 weeks

libfetch: replace 0 with NULL for pointers.

Found with devel/coccinelle.

Reviewed by: des

latin1 -> utf8

Update copyright dates and strip my middle name.

Think first, commit second.

1. Allow the caller to select active mode.
2. Fix the envar logic so it *always* overrides the caller's flags.
3. Document the change from active to passive.

Long overdue: make passive mode the default for ftp.

Increase WARNS to 4.

Fix a bug related to connection caching which could cause a crash after
a STAT command.

PR: kern/153748 (different patch)
Submitted by: Mark Johnston <markjdb@gmail.com>
MFC after: 2 weeks

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

MFC r199801:
Fix a socket leak.

Sponsored by: Sandvine Incorporated

Fix a socket leak in ftp_request() after that a connection is established.

Submitted by: Sandvine Incorporated
Reviewed by: des, emaste
Sponsored by: Sandvine Incorporated
MFC: 1 week

- Import the HEAD csup code which is the basis for the cvsmode work.

Use memcpy(3) instead of the BSD-specific bcopy(3).

Submitted by: Joerg Sonnenberger <joerg@britannica.bec.de>
MFC after: 2 weeks

Fix a regression introduced in rev 1.99: replace fclose(f) with a comment
explaining why f cannot possibly be a valid FILE * at this point.

MFC after: 1 day

As several people pointed out, I did all the ctype casts the wrong
way (not for the first time...)

Noticed by: bde, ru ++
MFC after: 1 week

Add support for the NO_PROXY / no_proxy environment variable as used by
lynx, curl etc. Note that this patch differs significantly from that
in the PR, as the submitter refined it after submitting the PR.

PR: 110388
Submitted by: Alexander Pohoyda <alexander.pohoyda@gmx.net>
MFC after: 3 weeks

Old patch I had lying around: correctly cast the argument to is*().
IWBNI gcc could warn about this the way it warns about printf() abuse.

MFC after: 1 week

Clean up namespace violations.

MFC after: 1 week

Add back the original behavior of changing the entire directory path at
once (CWD a/b/c vs. 3 CWDs). If an error occurs, we fall back to the default
method of a single CWD per directory element. Since this is technically
a violation of the basic FTP RFC, this behavior is under a compile-time
option FTP_COMBINE_CWDS and is off by default. It should work with most
Unix-based FTP daemons and can save latency.

MFC after: 2 weeks

Respect FETCH_BIND_ADDRESS when opening the data connection.

PR: misc/98872

Fix a bug introduced in rev 1.92, where, when changing from one directory
to another, the first CWD after a string of CDUPs would incorrectly include
a slash before the directory name.

Reported by: obrien
PR: bin/83278

In order to maintain interoperability with certain broken FTP servers,
ignore a MODE failure if and only if the mode we attempted to set was S
(which is supposed to be the default).

PR: bin/91973

Change directory one level at a time, and use CDUP to back out. This is a
work in progress; it partially fixed bin/83278 and is a prerequisite to
fixing bin/83277.

PR: bin/83277, bin/83278

Update copyright years.

Don't close a FILE * which we know is bogus.

Some servers respond to RETR in active mode with 125 (connection already
open) rather than 150 (opening connection). There's no reason why we
shouldn't accept that.

PR: misc/42172
MFC in: 3 days

The flags passed in to _ftp_get_proxy may be null

Approved by: des, markm (mentor)(implicit)

Don't parse the proxy URL unless we're actually going to use it. No real
functional difference, but debugging output will be less confusing.

style(9): add parentheses to sizeof even when not strictly required.

MFC after: 3 days

Hook in the .netrc code + don't use pointers as if they were booleans.

The FTP connection caching needs a better interface -- connections are
closed through _fetch_close() which is the only one who knows the connection
REALLY was closed (since ref -> 0). However, FTP keeps its own local
cached_connection and checks if it is valid by comparing it to NULL. This
is bogus since it may have been freed elsewhere by _fetch_close().

This change checks if we are closing the cached_connection and the ref is 1
(soon to be 0). If so, set cached_connection to NULL so we don't
accidentally reuse it. The REAL fix should be to move connection caching
to the common.c level (_fetch_* functions) and NULL the cache(s) in
_fetch_close(). Then all layers could benefit from caching.

When recycling a cached connection, increment the reference count so that
the heap block does not get freed and reused. This should fix the
pkg_add -r crashes that have been happening for months.

Make _fetch_connect() always set the error code.
Tell ftp that _fetch_connect() always sets the error code (http already knew)

Cope with 2292bis-01 getaddrinfo (no NI_WITHSCOPEID, always attach
scope identifier).

Approved by: des
MFC after: 3 weeks

Add a reference count to struct fetchconn so we don't prematurely close and
free a cached FTP connection.

Wrap everything in struct connection, and enforce timeouts everywhere
(except for DNS operations). Always use funopen() for HTTP, to support
both timeouts and SSL.

First step towards SSL support: wrap connections in a 'struct connection'
which contains the socket descriptor, the input buffer and (yet unused)
SSL state variables. This has the neat side effect of greatly improving
reentrance (though we're not *quite* there yet) and opening the door to
HTTP connection caching.

This commit is inspired by email conversations with and patches from
Henry Whincup <henry@techiebod.com> last fall.

Modernize my email address

Reindent, and add parentheses to return statements. Some functions in
ftp.c and http.c now have exceedingly long lines due to deep nesting;
this will be corrected by reorganizing the code in a later revision.

Mark uploads as O_WRONLY, not O_RDONLY.

PR: misc/34043
MFC after: 2 weeks

Remove VT100 escapes from debugging messages now that they're enabled by
default.

PR: 32988
MFC after: 3 days

Reorganize to reduce code duplication.

Tons of type, style and warning fixes that have been rotting in my tree for
ages - some of which wouldn't be necessary if gcc wasn't broken or TPTB were
willing to do something (-fno-builtin) about it.

Add __FBSDID()s to libfetch

Handle snprintf() returning < 0 (not just -1)

MFC after: 2 weeks

Handle snrintf overflows.

Spotted by: bde

Handle snprintf() returning -1.

MFC after: 2 weeks

Don't call _ftp_closefn() upon EOF in _ftp_readfn(); just return 0. This
fixes a bug in pkg_add(1) (which nobody noticed because of another bug).

Add rudimentary support for an authentication callback function.

Plug memory leak.

PR: 27506

Apply 'const' liberally.

Fix some other minor glitches.

If the server's reply to the SIZE command is unparseable, reset us->size
to -1 so the caller will know it's invalid. This is an MFC candidate.

Use high port range by default, and replace the 'h' option with an 'l' option
that forces the ftp code to use the low (default) port range instead.

Don't remember an EINTR, since the caller may want to restart the call.

Support lower-case versions of the proxy environment variables.

PR: bin/25494

Use "anonymous" rather than "ftp" as login name for anonymous ftp.

Rather than have a separate (misnamed) FTP_ANONYMOUS_PASSWORD constant, use
FTP_ANONYMOUS_USER (i.e. "anonymous") to construct the anonymous ftp password
if getlogin() fails.

Check the FTP_LOGIN environment variable before falling back on
FTP_ANONYMOUS_USER.

Somewhere along the line, I misunderstood the whole FTP_PASSIVE_MODE debate
and had libfetch selecting passive mode even when FTP_PASSIVE_MODE was not
set at all, which is really quite surprising unless you know about it. So
change it to the agreed default behaviour of selecting passive mode if
FTP_PASSIVE_MODE is set, but not "no".

Fix old-style proxy specs: default to FTP if FTP_PROXY was set; only default
to HTTP if HTTP_PROXY was used instead.

Don't go haywire if the server closes the connection in the middle of a
multiline response (proper fix this time).

Revert previous commit, it was somewhat hasty.

Don't go haywire if the server closes the connection during a multiline
response.

Use the documented (and historical) defaults. Centralize the decision logic
in order to avoid this bug in the future.

Submitted by: se

Use CHECK_FLAG

Stricter error checking in the I/O functions.

Fix this my way. David had absolutely no call overriding MAINTAINER without
even giving me 24 hours to read his mail and find the bug.

Repeat after me, "check to see that a pointer isn't NULL before
dereferenceing it". This fixes ``pkg_add -r''.

MFS: don't expect result code until you close the data connection

Use funopen() instead of fdopen(). This fixes three problems:

- ftpTimeout was not honored when reading actual data, as opposed to
talking protocol

- connection caching was broken because _ftp_cached_connect() would see
the result of the transfer instead of the result of the NOOP.

- if the RETR succeeded, but an error occurred later (as can happen
when talking to a proxy), the error would not be detected.

There still remains to register an atexit(3) callback to close the cached
connection gracefully instead of just dropping it on the floor.

Argh! Fix passive mode selection (again)

Use the right user name

Rework proxy handling so that proxies can be specified as URLs.
As a side effect, remove a lot of duplicate and now redundant code.

Grok 125 replies to passive RETR. This fixes an interoperability bug with
Microsoft FTP Service.

Reported by: asmodai, eivind

Don't fail if _ftp_stat() reports a protocol error, since that most likely
means that the server doesn't understand SIZE or MDTM, which should not be a
hard failure.

Submitted by: ume

If the server reports the size as 0, treat it as unknown. This works around
a bug in some ftp servers (most notably ftp.vmunix.com) which report the
size of a file correctly in ascii mode, but report it as 0 in binary mode.

Reported by: asmodai

Also remove an unneeded initialization.

Centralize the default port finding code.
Work around YA Apache bug: don't send port in Host: header if it's the
default port.

The decision to use an HTTP proxy for FTP documents belongs in the FTP code

Don't send port number to proxy unless it's not the standard FTP port

Fix the case where the file name is preceded by a lone slash.

Don't take a failure to stat a file to mean that it doesn't exist, as it may
be a dynamically created file, e.g. a tarball on servers that support sending
tarballs of directories.

Introduce fetchXGet*(), which combine the functionalities of fetchGet*() and
fetchStat*(). In most cases, either fetchGet*() or fetchXGet*() is a wrapper
around the other; in all cases, calling fetchGet*() is identical to calling
fetchXGet*() with the second argument set to NULL.

Use a symbolic constant instead of hardcoding 999

Use fetch_putln()

Clean up the debugging output

Make EPSV work again. Separate parsing code of 229 replies from
the code for 227 and 228.

Submitted by: des

Don't assume the remote address in a 227 reply is enclosed in parentheses.

Make restart work in active mode, too.

PR: bin/18688
Submitted by: Rudolf Cejka <cejkar@dcse.fee.vutbr.cz>

Handle multiline replies properly, instead of kinda-right.

Back out wes' commit with extreme prejudice.

Add implementation of fetchListFTP.

Reviewed by: silence

Honor FTP_PASSIVE_MODE so caller doesn't need to.

Use $FTP_PASSWORD for FTP password. If $FTP_PASSWORD is not
found, `yourname@yourhost' is used.

IPv6 support.

Better handling of some boundary conditions.

Submitted by: ume

Initialize the struct url_stat at the beginning of fetchStatFTP().

Add workaround for ftpds with the Y2K MDTM bug

Implement restart.
Use sizeof more consistently.
Outline sanity checking of server port spec.

Fix timestamp handling.

Submitted by: bde

More old uncommitted patches: implement timeouts at the protocol level.
Currently only supported for ftp connections.

Patches I've had lying around for several months:
* Add the 'h' ftp flag (allocate local port in high range)
* Add the 'd' flag (use direct connection even if proxy is defined)
* Make sure flags != NULL before calling strchr().
* Minor changes to some comments.

$Id$ -> $FreeBSD$

Implement and document file list retrieval.

Don't specify a language to compile_et.
Rename fetchConnect to _fetch_connect since it's internal.

Implement and document fetchStatFTP.
Update description of struct url_stat in the man page.
Clean up error handling in ftp.c.

Remove unused header files. Fix a few shadowing bugs. Improve error reporting.

Add verbose flag, and support functions.
Brucify the Makefile.
Differentiate atime and mtime in fetch*Stat().
Fix a few pointer bugs.
Tweak some error messages.
Don't #include sys/param.h and stdio.h in fetch.h.
Document that sys/param.h and stdio.h must be #included before fetch.h.

Second of a series of cleanups to libfetch.

This commit introduces the following features:

a) the fetchStat*() functions, which return meta-information for a
document, such as size, modification time, etc.

b) the use of the com_err(3) facilities to report errors.

It also fixes a bunch of style bugs and a few logic bugs and somewhat
improves the man page.

Changed files, in alphabetical order:

Makefile:
Don't generate macros in {ftp,http}err.c.

Generate category fields for the error message lists.

Compile the error table.

Install fetch_err.h along with fetch.h.

common.c:
Remove the _netdb_errstring() macro, and add FETCH_ERR_NETDB to the
error code in the _netdb_seterr() macro.

Add categories to the _netdb_errlist table.

Report errors through the Common Error library.

common.h:
Add the DEBUG macros.

Add prototype for fetchConnect().

Remove the prototype for _fetch_errstring(), which is local to common.c

Add a categroy field to struct fetcherr, and define constants for
error categories.

Define macros for _{url,netdb,ftp,http}_seterr().

errors.et: (new file)
List error categories.

fetch.3:
Document the fetchStat*() functions.

Move the "unimplemented functionality" comments from NOTES to BUGS.

Document that applications which use libfetch must also use
libcom_err, and list existing error codes.

Undocument fetchLastErr{Code,String}.

Remove the (empty) DIAGNOSTICS section.

Mention Eugene Skepner in the AUTHORS section.

fetch.c:
Move the DEBUG macros to common.c

Add fetchStat() and fetchStatURL().

Generate error messages for URL parser errors, and fix a minor bug
in the parser.

Use 'struct url' instead of 'url_t'.

Remove fetchLastErr{Code,String}.

fetch.h:
Use 'struct url' instead of 'url_t', and remove the typedef.

Define struct url_stat (used by fetchStat()).

Add prototypes for fetchStat*().

Remove the declarations for fetchLastErr{Code,String}.

Include fetch_err.h.

fetch_err.et: (new file)
Error table for libfetch.

file.c:
Add fetchStatFile().

Use 'struct url' instead of 'url_t'.

ftp.c:
Add fetchStatFTP().

Use 'struct url' instead of 'url_t'.

Don't use fetchLastErrCode.

ftp.errors:
Add categories to all error messages.

http.c:
Add fetchStatHTTP().

Use 'struct url' instead of 'url_t'.

Don't use fetchLastErr{Code,Text}.

http.errors:
Add categories to all error messages.

Prompted by: jkh and Eugene Skepner
Numerous sugestions from: Garett Wollman and Eugene Skepner

First of a series of cleanups to libfetch. Changed files, in
alphabetical order:

Makefile:
Add common.c to SRCS.

Make debugging easier by making 'CFLAGS += -DNDEBUG' conditional on DEBUG

Don't declare struct {ftp,http}err in {ftp,http}err.c; use struct fetcherr
instead.

README:
Remove the todo list, which is out of date anyway.

common.c: (new file)
Gather utility functions in this file.

Merge the error reporting functions intp _fetch_errstring(),
_fetch_seterr() and _fetch_syserr().

Set fetchLastErrCode and fetchLastErrText appropriately when fetchConnect
fails.

common.h: (new file)
Gather internal prototypes and structures in this files.

fetch.3:
Undocument fetchFreeURL().

Document a few more known bugs.

Document fetchLastErrCode and fetchLastErrText.

fetch.c:
Add descriptive comments to all functions that lacked them.

Move fetchConnect() to common.c.

Obviate the need for fetchFreeURL(), and remove it.

fetch.h:
Modify struct url_t so the document part is at the end.

ftp.c:
Remove code that is duplicated elsewhere.

http.c:
Remove code that is duplicated elsewhere.

Prompted by: jkh

Commit a bunch of patches that have been accumulating:

- Fix the README to reflect the new status of the ftp code.
- Change tons of 'if (xxx < 0)' to 'if (xxx == -1)'
- Add two new interface functions
- Fix the Makefile so it actually works (yay!)

Now the manpage is lagging even further behind... :( Next on the todo
list is to clean up the http code.

Base64 code (and the MIT copyright) moved to http.c
FTP STORe and APPEnd added.
FTP proxy support added (untested).

FTP retrieve works.

Too many changes to list. Basically, FTP is nearly there and error
reporting is kinda sorted out. Now HTTP needs to catch up...

Imported libfetch into the tree. It compiles, but there's still some
work to do. I especially need help with the man page.