MFC r320163:

Fix usr.sbin/makefs/makefs_ffs_tests when /etc/fstab does not exist

dumpfs prints a harmless warning message (via ufs_disk_fillout(3) and
getfsfile(3)), when /etc/fstab does not exist. We can ignore it.

PR: 220165
Reported by: gjb
Sponsored by: Spectra Logic Corp

MFC r316593:

Point out that -F probably does not do what the user expects.

Users attempting to create images from mtree METALOG files created by
installworld often use -F when they should be passing the METALOG file
in place of a directory. This is often produces difficult to debug
error reports.

Reviewed by: emaste
Sponsored by: DARPA, AFRL

MFC r309192

Fix null dereference when running out of nodes during rename.

PR: 212370
Submitted by: Sascha Wildner <swildner AT gmail>
Reviewed by: bcr (mentor)
Approved by: bapt
Obtained from: NetBSD
Differential Revision: https://reviews.freebsd.org/D8643

MFC r299461:
r299461 (by cem):

ffs_bswap: Copy one UFS dinode member at a time

No functional change.

CIDs: 974635, 974636, 977396, 977397, 977398, 977399

MFC r299953

Fix an off by one error to avoid overflowing rp[].

Reported by: Coverity
CID: 1007579

MFC r294891:

Require /sbin/mount_cd9660 when running the cd9660 tests

In some cases the test system might not have mount_cd9660(8). Don't
implicitly rely on it while testing cd9660 support; explicitly rely
on it

MFC r289677:

Fix a ton of speelling errors

MFC r292884:

Fix getopt(3) argument after r290180; I forgot to change -r to -R
by accident

Pointyhat to: ngie

Bump .Dd

MFC r290259,r290601:

r290259:

Sync makefs(8) content a bit with src/usr.sbin/makefs/makefs.8@1.53

Sections involving unimplemented filesystems (chfs, msdosfs, udf, v7fs) and
options have been omitted.

Obtained from: NetBSD
Sponsored by: EMC / Isilon Storage Division

r290601:

Follow up to r290259 dealing with makefs(8)

- Don't use contractions (don't -> do not)
- Change "throw away" to "discard" when describing the -o keep-bad-images option
- Revert author e-mail split I brought over from NetBSD, effectively reverting
the change bapt made in r267668

Submitted by: bjk
Sponsored by: EMC / Isilon Storage Division

MFC r290260,r290262:

r290260:

Document undocumented long options for -t cd9660

Note which options have been implemented and which options haven't
been implemented

Submitted as the following NetBSD PRs: bin/50390 and bin/50392

Sponsored by: EMC / Isilon Storage Division

r290262:

Fix spelling of `isolevel` cd9660 option

Sponsored by: EMC / Isilon Storage Division

MFC r266930,r289225:

r266930 (by jmg):

convert to using the _daddr_t types like newfs was...

Put the superblock in the correct possition for UFS2... There is a bug
in FFS that if we don't put it here (for UFS2), it will forcefully
relocate the superblock, and I believe cause data loss..

I have a fix for that, but w/ how many releases are broken, we won't be
able to switch to the better _FLOPPY (block 0) for this for a while..

r289225 (by sbruno):

makefs(8) leaves sblock.fs_providersize uninitialized (zero) that can be easily
checked with dumpfs(8). This may lead to other problems, f.e. geom_label kernel
module sanity checks do not like zero fs_old_size value and skips such UFS1
file system while tasting (fs_old_size derives from sblock.fs_providersize).

PR: 203704
Submitted by: eugen@grosbein.net
Reviewed by: marcel

MFC r290268:

Sync minor whitespace / type changes in ffs_csum_swap and ffs_sb_swap with
src/sys/ufs/ffs/ffs_bswap.c@1.39

Obtained from: NetBSD
Sponsored by: EMC / Isilon Storage Division

MFC r290264:

Limit isoLevel to 1 and 2 to avoid segfaulting when isoLevel is set to 3 by
dereferencing a NULL function pointer

Add some asserts to ensure that isolevel is always either 1 or 2.

PR: 203645
Reported by: Thomas Schmitt <scdbackup@gmx.net>
Sponsored by: EMC / Isilon Storage Division

MFC r290265,r290267,r290270:

r290265:

Add testcases for -t cd9660 -o isolevel=[1-3]

-- -o isolevel=1 currently fails because of path comparison issues,
so mark it as an expected failure.
-- -o isolevel=3 is not implemented, so expect it to fail as an out
of bounds value [*].

PR: 203645
Sponsored by: EMC / Isilon Storage Division

r290267:

Clean up mtree keyword support a slight bit and add a few more default keywords

- Parameterize the mtree keywords as $DEFAULT_MTREE_KEYWORDS
- Test with the extra mtree keywords, `mode,gid,uid`.
- Add a note about mtrees with time support not working with makefs right now

Sponsored by: EMC / Isilon Storage Division

r290270:

Add testcases for -t ffs -o version=[12]

Verify the filesystem type using dumpfs. Add preliminary support
for NetBSD (needs to be validated)

Sponsored by: EMC / Isilon Storage Division

MFC r289902:

Add a regression test for r289899 to validate rockridge encoding
of device types

Sponsored by: EMC / Isilon Storage Division

MFC r289899:

Import the fix from NetBSD kern/48852 (sic) to fix rockridge encoding of
device nodes

In particular, use st_rdev (the device type), not st_dev (the device inode),
and fix the comparison to be correct with the st_rdev field

Bug 203648
Submitted by: Thomas Schmitt <scdbackup@gmx.net>
Coverity CID: 1008927
Sponsored by: EMC / Isilon Storage Division

MFC r289203,r290180:

r289203 (by adrian):

makefs: introduce a new option to specify what to round the resulting
image up to.

From ticket:

While trying to run FreeBSD/mips on some device having very small flash media,
one is forced to compress file system with mkulzma(8) utility. It is desirable
to specify small UFS block/fragment sizes like 4096/512 bytes for makefs(8)
and big compression block size like 65535 bytes to mkulzma at the same time.
Then one obtains very good comression ratios (like 75% and more) but faces
the following problem.

geom_uncompress kernel module reports GEOM provider size rounded up to its
compression block size. Generally, this changes original media size and now
it fails to match the size of embedded UFS file system that leads to other
problems, f.e. geom_label kernel module does not like this and skips the
file system while tasting the GEOM and looking for UFS label.

This makes it impossible to refer to the file system using known UFS label
instead of something like /dev/map/rootfs.uncompress.

The following patch introduces new command line option "-r roundup" for makefs
that makes it round up the image to specified block size. Hence, geom_uncompress
does not change GEOM media size for images rounded that way and geom_label
accepts such GEOMs just fine.

With the patch applied, one can use following commands:

$ makefs -t ffs -r 65536 -o bsize=4096,fsize=512,label=flash optimization=space fs.img fs
$ mkulzma -s 65536 -o fs.img.ulzma fs.img

PR: bin/203707
Submitted by: <eugen@grosbein.net>

r290180:

Follow up to roundup feature addition in r289203

- Rename -r to -R to avoid the clash with makefs -r in NetBSD
- Note that -R is an FFS-specific option because it's not implemented
in cd9660 today
- Rename the roundup variable to "roundup-size" in the manpage and help
text for consistency with other variables.
- Bump .Dd (missed in r289203)

PR: 203707
Differential Revision: https://reviews.freebsd.org/D3959
Reviewed by: adrian (earlier patch), emaste
Sponsored by: EMC / Isilon Storage Division

MFC r289687,r289693:

r289687:

Free buffer before returning from cd9660_write_path_table to avoid
leaking it after returning from the function

PR: 203647
Submitted by: Thomas Schmitt <scdbackup@gmx.net>
Coverity CID: 978431
Sponsored by: EMC / Isilon Storage Division

r289693:

Unbreak makefs -t cd9660 after r289687

buffer_head needs to be freed -- not buffer

Detected by jemalloc, i.e. running makefs failed the arena assert
because my copy of malloc on CURRENT is compiled with the default
!MALLOC_PRODUCTION asserts on

Pointyhat to: ngie
PR: 203647
Sponsored by: EMC / Isilon Storage Division

MFC r289739,r289743,r289897,r289901:

r289739:

Correctly reintroduce the rudimentary smoke tests I botched up
in r289684

Sponsored by: EMC / Isilon Storage Division

r289743:

Revise "create_test_inputs" to simplify the file structure as
these testcases don't need to be nested as much as bin/ls/ls_tests.sh
do when verifying ls -a, ls -A, etc. This allows the tests to make
all paths relative to the top of the temporary directory instead of
always tacking on $ATF_TMPDIR, thus complicating things unnecessarily

Create non-empty files in create_test_inputs as well now, similar to
create_test_inputs2 in bin/ls/ls_tests.sh

Compare the input files to the output file contents using diff where
possible:
- Skip over the fifo comparison for now because it always fails
- Skip over the symlink comparison on cd9660 because it always fails
today

Sponsored by: EMC / Isilon Storage Division

r289897:

Add more cd9660/FFS makefs testcases

General changes:
- Parameterize out the mount command.
- Use mtree to verify the contents of an image (check_image_contents) instead
of using diff (diff verifies content, but not file metadata).
- Move common logic out to functions (common_cleanup, mount_image,
check_image_contents)
- Add stub testcases for makefs -D (crashes with SIGBUS, similar to bug # 192839)
- Add a note about the ISO-9660 and rockridge specs
- Add testcases that exercise:
-- Creating disk images from an mtree and multiple directories.
-- -F flag use (not really an extensive testcase right now)

cd9660-specific test changes:

- Remove an XXX comment about symlinks; I forgot that non-rockridge images turn
symlinks into hardlinks.
- Add testcases that exercise:
-- -o allow-deep-trees
-- -o allow-max-name stub testcase (doesn't seem to be implemented in makefs)
-- -o preparer (existence in image; not conformance to spec)
-- -o publisher (existence in image; not conformance to spec)
-- -o rockridge (basic)

Sponsored by: EMC / Isilon Storage Division

r289901:

Remove an ls -l I was using for debugging

Sponsored by: EMC / Isilon Storage Division

MFC r289601:

Don't check if `val` is NULL before calling free; free(3) already
handles this

PR: 203649
Submitted by: Thomas Schmitt <scdbackup@gmx.net>
Coverity CID: 1305659
Sponsored by: EMC / Isilon Storage Division

MFH: r277085

Fix a typo in the FFS maxbpg option, it was erroneously spelled maxbpf.

The error has been reported to and fixed in the NetBSD upstream version as well.

PR: 196598
Submitted by: Dan McGregor

MFC: r260041

Record the IEEE P1282 Rock Ridge version 1.12 POSIX File Serial Number,
i. e. the POSIX:5.6.1 st_ino field, which can be used to detect hard links
in the file system. This is also the default in mkisofs(8) and according to
its man page, no system only being able to cope with Rock Ridge version 1.10
is known to exist.

PR: 185138
Submitted by: Kurt Lidl

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Mark the makefs(8) '-p' flag as deprecated in preference for
the '-Z' flag for compatibility with NetBSD.

Submitted by: Eric van Gyzen (via stable@)
MFC after: 3 days

- Set the System Identifier in the Primary Volume Descriptor to FreeBSD
rather than NetBSD.
- Correctly set the Expiration Time in the Primary Volume Descriptor;
according to ISO 9660 8.4.26.1 unspecified date and time are denoted
by the digit 0 in RBP 1 to 16 but the number 0 in RBP 17. [1]
- Merge iso9660_rrip.c rev. 1.11 from NetBSD: name_len should be read
as unsigned byte. [2]
Note: This is according to ISO 9660 9.1.10.
- Rock Ridge TF entries should use a length of 5, because after the 4
bytes of generic SUSP header there is one byte of flags. See typedef
of ISO_RRIP_TF in iso9660_rrip.h. [1]

Submitted by: Thomas Schmitt [1]
Obtained from: NetBSD [2]
MFC after: 3 days

Ensure that makefs.h is included when using ufs_bswap.h so the FFS_EI macro
is picked up when defined. Previously, ffs_subr.c was always built without
support for opposite endianess as it doesn't include makefs.h on its own.

MFC after: 3 days

Set st_nlink in the stat structure within the inode to 1 as well.
The cd9660 file system uses that field for the link count and it
was 0. This impacts pwd_mkdb(8) as it checks for st_nlink not being
0 as part of closing a race.

Remove duplicated copies of various NetBSD compatibility shims used by
makefs and use libnetbsd, contrib/mtree, and contrib/mknod instead.

Sponsored by: DARPA, AFRLo
MFC after: 1 month

Remove contractions.

Support hardlinks in manifest files by the same logic as the treewalk
code.

Reviewed by: marcel
Sponsored by: DARPA, AFRL

Allow '.' components in manifest paths. They are always the first
component of mtree -C and install -M output and are easily skipped.

Reviewed by: marcel
Sponsored by: DARPA, AFRL

Fix the -N option in manifest mode by using pwcache(3). This also
speeds up image creation appreciably.

Reviewed by: marcel
Sponsored by: DARPA, AFRL

Add a -D flag that causes duplicate entries in an mtree manifest to be
treated as warnings rather than errors.

Reviewed by: marcel
Sponsored by: DARPA, AFRL

If no contents keyword is specified, the default for files is
the named file.

Approved by: marcel (mentor)

Fix usr.bin/ and usr.sbin/ build with a 64-bit ino_t.

Original code by: Gleb Kurtsou

Remove trailing whitespace.

Sort options.

Add -p flag to create the image as a sparse file.

Submitted by: Shesha Sreenivasamurthy
PR: bin/167779

Remove unused member of struct indir (in_exists) from UFS and EXT2 code.

Reviewed by: mckusick
Approved by: trasz (mentor)
MFC after: 1 week

- Write the ISO9660 descriptor after the apm partition entries.
- Fill the needed pmPartStatus flags. At least the OpenBIOS
implementation relies on these flags.

This commit fixes the panic seen on OS-X when inserting a FreeBSD/ppc disc.
Additionally OpenBIOS recognizes the partition where the boot code is located.
This lets us load a FreeBSD/ppc PowerMac kernel inside qemu.

PR: powerpc/162091
MFC after: 1 week

Allow contents of multiple directories to be merged to the current image.
Note this patch was submitted to NetBSD and they already adopted it.

http://mail-index.netbsd.org/source-changes/2012/01/28/msg031078.html

MFC after: 1 week

Fix a SIGSEGV problem in directory entry renaming.

Fix warning when compiling with gcc46:
error: variable 'parentrecnum' set but not used

Approved by: dim, cperciva (mentor, blanket for pre-mentorship already-approved commits)
MFC after: 3 days

Fix a variety of warnings when compiling with gcc46

Approved by: dim, cperciva (mentor, blanket for pre-mentorship already-approved commits)
MFC after: 3 days

Fix warning when compiling with gcc46:
error: variable 'temp' set but not used

Approved by: dim
Approved by: cperciva (mentor, blanket for pre-mentorship already-approved commits)
MFC after: 3 days

Spelling fixes for usr.sbin/

Small cleanups to panic() prototype.

- Let panic() use _Noreturn and __printflike() instead of GCC-specific
attributes.
- Remove prototype from ffs_subr.c and let it include ffs_extern.h.
- Forward declare struct inode to make ffs_extern.h not depend on it.

- remove device keyword from makefs manpage

PR: docs/144408
Submitted by: gcooper
Approved by: gjb
Approved by: sahil (mentor)
MFC after: 1 week

Add support to makefs(8) to add UFS labels to images.

Submitted by: avg
MFC after: 3 days

Fix buffer overflow and possible ISO image corruption in wrong
handling of "." character case in makefs ISO level 1 and 2 filename
conversion.

Filed as NetBSD PR #45285
http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=45285

Reviewed by: Christos Zoulas <christos@netbsd.org>
Approved by: re (kib)
MFC after: 3 days

Sync makefs(8) ISO 9660 support with NetBSD:
o cd9960 -> cd9660
o Move inclusion of sys/endian.h from cd9660_eltorito.c to cd9660.h
since actual user is not cd9660_eltorito.c but iso.h and
cd9660_eltorito.h.

Actually, include order/place of sys/endian.h doesn't matter on
netbsd since it is always included by sys/types.h but it's not
true on other system. This should fix cross build breakage on
freebsd introduced by rev. 1.16 of cd9660_eltorito.c.
Problem reported and fix suggested on twitter.
o Fix fd leaks in error cases. Found by cppcheck.
o RRIP RE length should be 4, not 0
o Apply fixes for PR bin/44114 (makefs(8) -t cd9660 -o rockridge creates
corrupted cd9660fs), iso9660_rrip.c part:
- cd9660_rrip_finalize_node() should check rr_real_parent in node->parent,
not in node itself in RRIP_PL case
- cd9660_rrip_initialize_node() should update only node passed as arg
so handle RRIP_PL in DOTDOT case

Fixes malformed dotdot entries in deep (more than 8 level) directories
moved into .rr_moved dir.

Should be pulled up to netbsd-5.
(no official ISO has such deep dirs, but cobalt restorecd is affected)

Reviewed by: mm
Approved by: re (kib)
Obtained from: NetBSD
MFC after: 3 days

Fix NetBSD PR bin/44114:
makefs with -t cd9660 -o rockridge against directories with
deep structure creates a corrupted cd9660 image.

http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=44114

Fix NetBSD PR bin/45217:
makefs creates ISO9660 images that violate the ECMA-119 (ISO9660)
specification. This is caused by erroneously writing 32 bytes
with value 0x20 to the volume_set_id field and 128 bytes with value 0x20
to the the following 37-byte fields in the PVD:
copyright_file_id, abstract_file_id, bibliographic_file_id

This causes, among other unwanted results the reserved4 field to be
overwritten with the value 0x20. To comply with the specification,
this field muse be zero. As a result, all FreeBSD distribution
images created with makefs have not been 100% valid ISO9660 files.

http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=45217

Reviewed by: kientzle
Approved by: re (kib)
Obtained from: NetBSD
MFC after: 3 days

Add support for using mtree(5) manifest files to define the image
to be created. The support is based on mtree version 2.0, as used
in libarchive, but adds new features on top of it.

The current implementation is fully functional, but is envisioned
to grow at least the following additional features over time:
o Add support for the /include special command so that manifest
files can be constructed using includable fragments.
o Add support specifying a search path to locate content files.
o Content file filters: commands that provide file contents on
stdout.

The manifest file eliminates the need to first construct a tree
as root in order to create an image and allows images (releases)
to be created directly from object trees and/or source trees.

Reviewed by: deo
Sponsored by: Juniper Networks, Inc

Drop the include of <ufs/ffs/ffs_extern.h> from usr.sbin/makefs/ffs/ffs_bswap.c
and usr.sbin/makefs/ffs/ffs_subr.c as they have no need of anything in that
file. No other programs or libraries include <ufs/ffs/ffs_extern.h> (nor
should they as it is totally in-kernel interfaces). For added protection
I enclosed the entire contents of <ufs/ffs/ffs_extern.h> in ifdef _KERNEL.

Feedback from: Bruce Evans and Tai-hwa Liang

Use alternative, less messy solution to avoid breakage after r223020:
put the snapdata structure between #ifdef _KERNEL guards.

Suggested by: kib

Apparently makefs needs a few more system headers to compile during
buildworld.

Unbreak usr.sbin/makefs after r223020.

Pointy hat to: mckusick

Add analogs to the -chrp-boot and -prep-boot options to mkisofs.

Simplify this code somewhat by only writing a 512-byte-sector APM map.
All firmwares can read it, we need to write one anyway for old OF
versions, and if a 512-byte one is present, OS X ignores the 2048-byte one
in favor of it.

Fix boot on old machines (e.g. blue and white G3s) by synthesizing a
512-byte sector map instead unused space in the first 2048-byte sector.

Add support for synthesizing an APM partition map to map Mac PowerPC
bootstrap partitions from the ISO9660 boot catalog. This preserves OS X's
ability to mount the CD, while allowing us a way to provide HFS-ified
bootstrap code for Open Firmware.

Really fix the confusion, sorry for noise

Submitted by: avg
Approved by: cognet

Fix confusion between a-characters and d-characters

Submitted by: avg
Approved by: cognet

The dump, fsck_ffs, fsdb, fsirand, newfs, makefs, and quot utilities
include sys/time.h instead of time.h. This include is incorrect as
per the manpages for the APIs and the POSIX definitions. This commit
replaces sys/time.h where necessary with time.h.

The commit also includes some minor style(9) header fixup in newfs.

This commit is part of a larger effort by Garrett Cooper started in
//depot/user/gcooper/posix-conformance-work/ -- to make FreeBSD more
POSIX compliant.

Submitted by: Garrett Cooper yanegomi at gmail dot com

Sync with the latest version from NetBSD. It notably addds ISO9660 support.

Submitted by: bapt

Fix typos and spelling mistakes.

- Merge soft-updates journaling from projects/suj/head into head. This
brings in support for an optional intent log which eliminates the need
for background fsck on unclean shutdown.

Sponsored by: iXsystems, Yahoo!, and Juniper.
With help from: McKusick and Peter Holm

The NetBSD Foundation has given permission to remove clause 3 and 4
from their liceense.

Obtained from: NetBSD

Correct copy-paste typo from previous option description.

Delete some empty mergeinfo.

Fix LLVM compiler errors related to K&R declarations with ANSI prototypes.

Submitted by: Pawel Worach <pawel.worach@gmail.com>

fix pathnames to work w/ buildworld

fix 64-bit build

import netbsd makefs tool

add properties and FreeBSD id's

turn warnx into a debug msg

enable support for st_flags and st_gen

another nbtool turd

fix stuff i broke

If the filesystem size rounded up to a multiple of the block size
is larger than what a user specified then round down to get something
that works but wastes a little space.

This happens reliably for me when building filesystems for CF parts
>1G; not sure why noone else is complaining.

unbreak cg_chkmagic_swap

Apply the big hammer:
o remove all of compat except for pwcache and strstuftoll; these might
end up in libutil or similar so keep them in the subdir
o mv getid.c up to the top level; this looks like something that'll be
makefs-specific
o eliminate private versions of .h files in sys; use system files instead
o eliminate private ffs_tables.c; use the system version directly (might
want to adopt const'ification at some point but that's the only diff I
can see)
o mv remaining code from sys to ffs and strip out unused bits; this now
becomes part of makefs
o add compat defs and shims to makefs.h
o strip all vestiges of nbtool_config.h, compat_defs.h, etc.
o fixup includes after file shuffling
o rename system #defines that do implicit byte swapping to have an _swap
suffix; e.g. DIRSIZ -> DIRSIZ_SWAP, cg_inosused -> cg_inosused_swap; if
we ever add endian-agnostic support to the kernel these can go back to
their original names
o strip some netbsd'isms that aren't worth shim'ing (e.g. _DIAGASSERT)

Code compiles w/o complaints but is untested.

gone

fparseln is in libutil

getmode+setmode are in libc, remove

First step toward proper mtree integration:
o use mtree code directly from ../mtree; s/spec(fp)/mtree_readspec(fp)/
(assume this will eventually go in usr.sbin)
o nuke st_rdev use; there's no reason to create filesystems with device nodes
o shim missing compat bits for inotype and nodetoino
o remove now unneeded compat/mtree bits, pack_dev.?, and stat_flags.?
o temporarily move getid.c code up to compat to keep this building

split stanzas in preparation for better integration

This directory is now empty

This isn't needed...

Remove the dump_node and vispath routines from compat/mtree/spec.c.
They aren't needed, and were the only users of the compat/vis.[ch]
files. As such, remove them from the Makefile.

prepare makefs for import to base