ftpd: stop using -g flag for /bin/ls

In 3bfbb521 the behaviour of ls was changed such that -g was no longer
a noop for compatibility with BSD 4.3, but instead changed the output
of long mode to exclude the owner of the file and display only the
group.

Update how FTPd invokes ls to restore the previous behaviour

Reported-by: Andrew Fengler <andrew.fengler@scaleengine.com>
Reviewed-by: jrtc27, des, imp
MFC after: 3 days
Sponsored-by: ScaleEngine Inc.
Fixes: 3bfbb521fef5 ("ls: Improve POSIX compatibility for -g and -n.")

libexec: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

libexec: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Fix build with recent byacc.

General further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

Rename getline with get_line to avoid collision with getline(3)

When getline(3) in 2009 was added a _WITH_GETLINE guard has also been added.
This rename is made in preparation for the removal of this guard

Remove the 3rd clause ("advertising clause") of the BSD license as
permitted by the University of Berkeley on July 22, 1999.

Reviewed by: imp
MFC after: 1 week

Spelling fixes for libexec/

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

Move variable externs into extern.h so they are checked against the definitions.

Prevent cross-site forgery attacks on ftpd(8) due to splitting
long commands into multiple requests. [08:12]

Avoid calling uninitialized function pointers in protocol switch
code. [08:13]

Merry Christmas everybody...

Approved by: so (cperciva)
Approved by: re (kensmith)
Security: FreeBSD-SA-08:12.ftpd, FreeBSD-SA-08:13.protosw

- Import the HEAD csup code which is the basis for the cvsmode work.

Add support for RFC 2389 (FEAT) and RFC 2640 (UTF8) to ftpd(8).

The support for RFC 2640 (UTF8) is optional and rudimentary.
The server just advertises its capability to handle UTF-8 file
names and relies on its own 8-bit cleanness, as well as on
the backward compatibility of UTF-8 with ASCII. So uploaded
files will have UTF-8 names, but the initial server contents
should be prepared in UTF-8 by hand, no on-the-fly conversion
of file names will be done.

PR: bin/111714
Submitted by: Zhang Weiwu <see email in the PR>
MFC after: 1 week

Fix compilation of ftpcmd.y without -DINET6.
Respect MK_INET6_SUPPORT in Makefile.

Requested by: Attila Nagy <bra at fsn dot hu>
MFC after: 1 week

Use __FBSDID.

Use uniform punctuation, capitalization, and language style
in server messages wherever this doesn't contradict to a particular
message format.

Fix perror_reply() vs. reply() usage.

Kill more unneeded casts found.

Noticed by: Nick Leuta <skynick -at- mail.sc.ru> (some of them)

Kill ancient casts to integral types left from the K&R era.
They're unneeded and sometimes erroneous now.

Change ``(foo *)0'' to ``NULL'' where it's possible
(and it appears possible throughout ftpd(8) source.)

It is not a mere issue of style: Null pointers in C
seem to have been mistaken one way or another quite often.

Kill a small herd of casts to off_t where they were not needed.
Thank Fortune, the C compiler can figure out by itself the proper
conversion for assignments, comparisons, and prototyped function
arguments.

Printf(3) off_t values through conversion to intmax_t since
we've got <stdint.h> et al now. (This makes ftpd(8) WARNS=2 clean.)

Convert a couple of bogus null statements to the right form.
(Heading to WARNS=2.)

Pacify gcc warning with a Douglas Adams reference.

Block SIGURG while reading from the control channel.

Rationale:

SIGURG is configured by ftpd to interrupt system calls, which is useful
during data transfers. However, SIGURG could interrupt I/O on the
control channel as well, which was mistaken for the end of the session.

A practical example could be aborting the download of a tiny file,
when the abort sequence reached ftpd after ftpd had passed the file
data to the system and returned to its command loop.

Reported by: ceri
MFC after: 1 week

Improve error handling in getline():
- always check the return value from getc(3) for EOF;
- if the attempt to read the TELNET command byte has
returned EOF, exit from the loop instead of using
the EOF value as a normal character.

MFC after: 1 week

Don't declare unneeded extern variables,
leave alone specifying a wrong type for one of them.

If ftpd is run with an -h option (hide host-specific info,)
don't reveal the info in reply to the SYST command.

Get rid of using the "unix" macro at the same time. It was a rather
poor way to check if the system was Unix since there were quite a
few Unix clones out there whose cc didn't define "unix" (e.g.,
NetBSD.) It was also sensitive to the C standard used, which caused
unnecessary trouble: With -std=c99, it should have been "__unix__",
and so on.

PR: bin/50690
Submitted by: Alex Semenyaka <alexs _at_ snark.ratmir.ru>
MFC after: 1 week

Allow "~/" in pathnames to work for a chrooted user.

Let tilde expansion be done even if a file/directory doesn't exist yet.
This makes such natural commands as "MKD ~user/newdir" or "STOR ~/newfile"
do what they are supposed to instead of failing miserably with the
"File not found" error.

This involves a bit of code reorganization. Namely, the code doing
glob(3) expansion has been separated to a function; a new function
has been introduced to do tilde expansion; the latter function is
invoked on a pathname before the former one. Thus behaviour mimicing
that of the Bourne shell has been achieved.

Add a new option to ftpd(8), "-h", to disable printing any
host-specific information in FTP server messages (so paranoid
admins can sleep at night :-)

PR: bin/16705
MFC after: 1 week

Give the code around chroot(2)/chdir(2) a major overhaul by
separating its part around chroot(2) from that around initial
chdir(2). This makes the below changes really easy.

Move seteuid(to user's uid) to before calling chdir(2). There are
two goals to achieve by that. First, NFS mounted home directories
with restrictive permissions become accessible (local superuser
can't access them if not mapped to uid 0 on the remote side
explicitly.) Second, all the permissions to the home directory
pathname components become effective; previously a user could be
carried to any local directory despite its permissions since the
chdir(2) was done with euid 0. This reduces possible impact from
FTP server misconfiguration, e.g., assigning a wrong home directory
to a user.

Implement the "/./" feature. Now a guest or user subject to chrooting
may have "/./" in his login directory, which separates his chroot
directory from his home directory inside the chrooted environment.
This works for ftpchroot(5) as well.

PR: bin/17843 bin/23944

GLOB_MAXPATH has been deprecated in favour of GLOB_LIMIT.

Prevent server-side glob(3) patterns from expanding
to a pathname that contains '\r' or '\n'.

Together with the earlier STAT bugfix, this must solve
the problem of such pathnames appearing in the FTP control
stream.

Replace the instances of literal "/bin/ls"
with the _PATH_LS macro to be consistent
with the rest of the ftpd(8) source.

Use the standardized CHAR_BIT constant instead of NBBY in userland.

Fix lexer jam on unimplemented commands.

Submitted by: maxim
MFC after: 5 days

Fix command help lines:
o PORT takes six byte values, not five.
o TYPE argument is mandatory.

Submitted by: demon (the 1st part)
MFC after: 3 days

1) Use "pathstring" instead of "STRING" consistently.
2) Remove unneeded "if not NULL" props from "pathstring",
which will never be NULL by the lexer design.

Inspired by: OpenBSD
MFC after: 1 week

Since GLOB_NOCHECK is set in the glob(3) call,
glob(3) will return at least one pathname unless
a system error has occured. It's not a "not found"
error otherwise.

MFC after: 3 days

Spot places where "pathname" hasn't been checked
for NULL. The "pathname" rule may return NULL
on a glob(3) error.

Obtained from: OpenBSD
MFC after: 1 week

Disallow invalid numeric mode values for SITE CHMOD.
Earlier, a decimal number (e.g., 890) could be passed
for mode, leading to dangerous permissions set:
-1, that is, 07777.

Obtained from: OpenBSD
MFC after: 1 week

Deny the SIZE command on large files when in ASCII mode.
This eliminates an opportunity for DoS attack.

Pointed out by: maxim
Inspired by: lukemftpd, OpenBSD
MFC after: 2 weeks

Use <arpa/ftp.h> stuff cleanly, without introducing
non-portable constants (in this case, hidden as offsets
to the "?AEIL" string.)

MFC after: 1 week

GLOB_QUOTE has been retired.

Teach REST how to restart a file transfer after 2^31 bytes: now yylex()
returns off_t in yylval.u.o. REST is the only user of yylval.u.o at the
moment.

NB: seems lukemftpd has the same bug.

PR: misc/28629
Reviewed by: ru
Approved by: ru
MFC after: 1 month

Remove duplicated yacc nonterminals declarations, sort includes.
No functional changes from rev. 1.31.

Reviewed by: ru
Approved by: ru
MFC after: 1 week

o Eliminate __P
o Use new-style function definitions
o remove some !__STDC__ code
o eliminate register

Remove the setjmp/longjmp stuff completely. Use signal
handlers to set flags only (with exception for sigquit(),
which still seems to call some non-reentrant functions on
its way to _exit(2).) That must eliminate the possibility
of catching SIGSEGV from following non-reentrant paths from
signal handlers.

PR: bin/32740 bin/33846
Submitted by: Maxim Konovalov <maxim@macomnet.ru>
Obtained from: OpenBSD

Be more careful about freeing memory after parsing commands.
Hiroyuki YAMAMORI gave a patch for the EPRT command in the
PR below. Problems with the rest of the patch are my fault.

PR: 33268
Reviewed by: iedowse, sheldonh

Extend the functionality offered by the -o option into a new option
-O, which limits the impact of the write-only restriction to guest
users.

*) The existing manual page's SYNOPSIS and option listing in the
DESCRIPTION are already horribly disordered. No attempt has been
made to fix this.

*) The existing source's getopt() optstring and option handling switch
are already horribly disordered. No attempt has been made to fix
this.

Discussed with: nik, -audit

Add a new option, '-o', for "Write-only". Disables the RETR command,
preventing anyone from downloading files. In conjunction with -A, and some
appropriate file permissions, this lets you create an anonymous FTP drop
box for people to upload files to.

The more obvious "-w" flag is already taken by NetBSD's ftpd. "-o" was
available as an option letter in all three BSDs.

Change names of functions and variables with global scope that are
in conflict with library values of the same name. This allows static
linking.

Previous clobbered a work-in-progress. Here is the merged result:

Limit the "pathname" glob to one item, as that is what all users of it
are expecting, except for LIST.

Always glob, instead of when the first character is a ~. For example,
if you had directories ~/x1, and ~/x2, then "cwd x[1]" would fail, but
"cwd ~/x[1]" would work since it was globbed due to the ~ character.
Also, "cwd ~/x[12]" used to arbitarily work as it used the first
expansion (ie: x1) without an error. Make it return '550 ambiguous'
instead of '550 not found' so that the user can see the difference.

For LIST, just use the user supplied string as the popen does the glob.

Problem noticed by: Ajay Mittal <amittal@iprg.nokia.com>

Limit number of paths returned via glob() for authorized users
using tilde expansion.

Support the empty "PASS\r\n" command.

Add the "SITE MD5 filename" facility.

This allows you to determine if the file on the other side is the same
as the one you have without transferring the entire file to compare.

Needless to say, if the server end lies to you this check doesn't work,
but on the other hand, if it lies to you about the files checksum,
what can you trust from it ?

Change the read-only reply to "550 Permission denied.".

Limit commands that can be issued when not logged in:
TYPE, STRU, MODE, ALLO, STAT, ABOR, SITE IDLE, SYST, REST

Reviewed by: kris, sheldon

Add option -E to disable EPSV which throws certain stateful firewalls
into confusion.

Add option -r to make ftpd support only read-only operations.

Submitted by: Flemming (F3) Jacobsen <fj@batmule.dk>
Reviewed by: phk

Prevent leakage of information about anonymous user's homedir
via 'QUOTE CWD'.

Reviewed by: des

Don't reply "not a plain file" when the requested file doesn't exist.

another tcp apps IPv6 updates.(should be make world safe)
ftp, telnet, ftpd, faithd
also telnet related sync with crypto, secure, kerberosIV

Obtained from: KAME project

sync with netbsd PR 8534, fix undefined C code.
Pointed out by: David A. Holland

$Id$ -> $FreeBSD$

Cosmetics in man page. Exit(-1) -> exit(1).

Many places in the code NULL is used in integer context, where
plain 0 should be used. This happens to work because we #define
NULL to 0, but is stylistically wrong and can cause problems
for people trying to port bits of code to other environments.

PR: 2752
Submitted by: Arne Henrik Juul <arnej@imf.unit.no>

Make useage of hostname global variable consistent.
PR: 4135
Based on submitted patch by: blank@fox.uni-trier.de

Revert $FreeBSD$ to $Id$

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

add forgotten $Id$

Convert STATS and PARANOID to run-time options.
Document the new -R (relax paranoia) option.

From NetBSD/Lite2: code and man page cleanups, Kerberos IV hooks
(relax, we're still exportable), and /etc/ftpchroot feature for
semi-anonymous accounts

If PARANOID is set, do not allow PORT commands to remote ports less than 1024
or addresses other than the requestor's address. This violates the FTP
protocol (hmm...as I write this, I'm going to change this to a run-time var.)

Require login before PASV and RNTO commands.

Close unused PASV ports so they don't hang around forever.

Do not allow file overwrites via rename or STOR when anonymous
(suspenders).

Clean up buffer utilization.

My code, but heavily inspired by Hobbit's changes to wu-ftpd as pointed out
by Mike Prettejohn and Kit Knox.

Make ftpd use setproctitle() from libutil

I've left the old code in there under #ifdef OLD_SETPROCTITLE in case
somebody wants to try to compile out ftpd on some other machine.

Figured it out, misapplied a patch, ftpd now works again.

Fix broken command parser (fall back 10 yards and scratch head).

Include most of the logdaemon v4.4 S/key changes

BSD 4.4 Lite Libexec Sources