#
d0b2dbfa |
|
16-Aug-2023 |
Warner Losh <imp@FreeBSD.org> |
Remove $FreeBSD$: one-line sh pattern Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/
|
#
a71ea7be |
|
22-Jul-2022 |
Emmanuel Vadot <manu@FreeBSD.org> |
pkgbase: Put zfs utilities and lib in their own package It is useful to have zfs utilities and lib in a separate package as it allow users to create image that can support ZFS (i.e. not with WITHOUT_ZFS in src.conf set) without bloating the default image with all zfs tools (for example for jails). Differential Revision: https://reviews.freebsd.org/D36225
|
#
716fd348 |
|
18-May-2022 |
Martin Matuska <mm@FreeBSD.org> |
zfs: merge openzfs/zfs@c0cf6ed67 Notable upstream pull request merges: #10662 zvol_wait: Ignore locked zvols #12789 Improve log spacemap load time #12812 Improved zpool status output, list all affected datasets #13277 FreeBSD: Use NDFREE_PNBUF if available #13302 Make zfs_max_recordsize default to 16M #13311 Fix error handling in FreeBSD's get/putpages VOPs #13345 FreeBSD: Fix translation from ABD to physical pages #13373 zfs: holds: dequadratify #13375 Corrected edge case in uncompressed ARC->L2ARC handling #13388 Improve mg_aliquot math #13405 Reduce dbuf_find() lock contention #13406 FreeBSD: use zero_region instead of allocating a dedicated page Obtained from: OpenZFS OpenZFS commit: c0cf6ed6792e545fd614c2a88cb53756db7e03f8
|
#
73734d6e |
|
16-Nov-2020 |
Brooks Davis <brooks@FreeBSD.org> |
Add missing includes of src.opts.mk Without this "SUBDIR.${MK_TESTS}=tests" would always expand to "SUBDIR.=tests" resulting in the tests not being built. Sponsored by: DARPA |
#
9e5787d2 |
|
24-Aug-2020 |
Matt Macy <mmacy@FreeBSD.org> |
Merge OpenZFS support in to HEAD. The primary benefit is maintaining a completely shared code base with the community allowing FreeBSD to receive new features sooner and with less effort. I would advise against doing 'zpool upgrade' or creating indispensable pools using new features until this change has had a month+ to soak. Work on merging FreeBSD support in to what was at the time "ZFS on Linux" began in August 2018. I first publicly proposed transitioning FreeBSD to (new) OpenZFS on December 18th, 2018. FreeBSD support in OpenZFS was finally completed in December 2019. A CFT for downstreaming OpenZFS support in to FreeBSD was first issued on July 8th. All issues that were reported have been addressed or, for a couple of less critical matters there are pull requests in progress with OpenZFS. iXsystems has tested and dogfooded extensively internally. The TrueNAS 12 release is based on OpenZFS with some additional features that have not yet made it upstream. Improvements include: project quotas, encrypted datasets, allocation classes, vectorized raidz, vectorized checksums, various command line improvements, zstd compression. Thanks to those who have helped along the way: Ryan Moeller, Allan Jude, Zack Welch, and many others. Sponsored by: iXsystems, Inc. Differential Revision: https://reviews.freebsd.org/D25872 |
#
a8c08e00 |
|
18-Nov-2019 |
Andriy Gapon <avg@FreeBSD.org> |
MFV r354378,r354379,r354386: 10499 Multi-modifier protection (MMP) 10499 Multi-modifier protection (MMP) illumos/illumos-gate@e0f1c0afa46cc84d4b1e40124032a9a87310386e https://github.com/illumos/illumos-gate/commit/e0f1c0afa46cc84d4b1e40124032a9a87310386e https://www.illumos.org/issues/10499 Port the following ZFS commits from ZoL to illumos. 379ca9cf2 Multi-modifier protection (MMP) bbffb59ef Fix multihost stale cache file import 0d398b256 Do not initiate MMP writes while pool is suspended 10701 Correct lock ASSERTs in vdev_label_read/write illumos/illumos-gate@58447f688d5e308373ab16a3b129bc0ba0fbc154 https://github.com/illumos/illumos-gate/commit/58447f688d5e308373ab16a3b129bc0ba0fbc154 https://www.illumos.org/issues/10701 Port of ZoL commit: 0091d66f4e Correct lock ASSERTs in vdev_label_read/write At a minimum, this fixes a blown assert during an MMP test run when running on a DEBUG build. 11770 additional mmp fixes illumos/illumos-gate@4348eb901228d2f8fa50bb132a34248e8662074e https://github.com/illumos/illumos-gate/commit/4348eb901228d2f8fa50bb132a34248e8662074e https://www.illumos.org/issues/11770 Port a few additional MMP fixes from ZoL that came in after our initial MMP port. 4ca457b065 ZTS: Fix mmp_interval failure ca95f70dff zpool import progress kstat (only minimal changes from above can be pulled in right now) 060f0226e6 MMP interval and fail_intervals in uberblock Note from the committer (me). I do not have any use for this feature and I have not tested it. I only did smoke testing with multihost=off. Please be aware. I merged the code only to make future merges easier. Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com> Portions contributed by: Tim Chase <tim@chase2k.com> Portions contributed by: sanjeevbagewadi <sanjeev.bagewadi@gmail.com> Portions contributed by: John L. Hammond <john.hammond@intel.com> Portions contributed by: Giuseppe Di Natale <dinatale2@llnl.gov> Portions contributed by: Prakash Surya <surya1@llnl.gov> Portions contributed by: Brian Behlendorf <behlendorf1@llnl.gov> Author: Olaf Faaland <faaland1@llnl.gov> MFC after: 4 weeks
|
#
d12e91d5 |
|
20-Aug-2018 |
Matt Macy <mmacy@FreeBSD.org> |
Make dnode definition uniform on !x86 gcc4 requires -fms-extensions to accept anonymous union members |
#
450e5a43 |
|
15-Aug-2018 |
Will Andrews <will@FreeBSD.org> |
zfs: add ztest to the kyua test suite. This program is currently failing, and has been for >6 months on HEAD. Ideally, this should be run 24x7 in CI, to discover hard-to-find bugs that only manifest with concurrent i/o. Requested by: lwhsu, mmacy |
#
cc0fbbb9 |
|
11-Aug-2018 |
Matt Macy <mmacy@FreeBSD.org> |
MFV/ZoL: Implement large_dnode pool feature commit 50c957f702ea6d08a634e42f73e8a49931dd8055 Author: Ned Bass <bass6@llnl.gov> Date: Wed Mar 16 18:25:34 2016 -0700 Implement large_dnode pool feature Justification ------------- This feature adds support for variable length dnodes. Our motivation is to eliminate the overhead associated with using spill blocks. Spill blocks are used to store system attribute data (i.e. file metadata) that does not fit in the dnode's bonus buffer. By allowing a larger bonus buffer area the use of a spill block can be avoided. Spill blocks potentially incur an additional read I/O for every dnode in a dnode block. As a worst case example, reading 32 dnodes from a 16k dnode block and all of the spill blocks could issue 33 separate reads. Now suppose those dnodes have size 1024 and therefore don't need spill blocks. Then the worst case number of blocks read is reduced to from 33 to two--one per dnode block. In practice spill blocks may tend to be co-located on disk with the dnode blocks so the reduction in I/O would not be this drastic. In a badly fragmented pool, however, the improvement could be significant. ZFS-on-Linux systems that make heavy use of extended attributes would benefit from this feature. In particular, ZFS-on-Linux supports the xattr=sa dataset property which allows file extended attribute data to be stored in the dnode bonus buffer as an alternative to the traditional directory-based format. Workloads such as SELinux and the Lustre distributed filesystem often store enough xattr data to force spill bocks when xattr=sa is in effect. Large dnodes may therefore provide a performance benefit to such systems. Other use cases that may benefit from this feature include files with large ACLs and symbolic links with long target names. Furthermore, this feature may be desirable on other platforms in case future applications or features are developed that could make use of a larger bonus buffer area. Implementation -------------- The size of a dnode may be a multiple of 512 bytes up to the size of a dnode block (currently 16384 bytes). A dn_extra_slots field was added to the current on-disk dnode_phys_t structure to describe the size of the physical dnode on disk. The 8 bits for this field were taken from the zero filled dn_pad2 field. The field represents how many "extra" dnode_phys_t slots a dnode consumes in its dnode block. This convention results in a value of 0 for 512 byte dnodes which preserves on-disk format compatibility with older software. Similarly, the in-memory dnode_t structure has a new dn_num_slots field to represent the total number of dnode_phys_t slots consumed on disk. Thus dn->dn_num_slots is 1 greater than the corresponding dnp->dn_extra_slots. This difference in convention was adopted because, unlike on-disk structures, backward compatibility is not a concern for in-memory objects, so we used a more natural way to represent size for a dnode_t. The default size for newly created dnodes is determined by the value of a new "dnodesize" dataset property. By default the property is set to "legacy" which is compatible with older software. Setting the property to "auto" will allow the filesystem to choose the most suitable dnode size. Currently this just sets the default dnode size to 1k, but future code improvements could dynamically choose a size based on observed workload patterns. Dnodes of varying sizes can coexist within the same dataset and even within the same dnode block. For example, to enable automatically-sized dnodes, run # zfs set dnodesize=auto tank/fish The user can also specify literal values for the dnodesize property. These are currently limited to powers of two from 1k to 16k. The power-of-2 limitation is only for simplicity of the user interface. Internally the implementation can handle any multiple of 512 up to 16k, and consumers of the DMU API can specify any legal dnode value. The size of a new dnode is determined at object allocation time and stored as a new field in the znode in-memory structure. New DMU interfaces are added to allow the consumer to specify the dnode size that a newly allocated object should use. Existing interfaces are unchanged to avoid having to update every call site and to preserve compatibility with external consumers such as Lustre. The new interfaces names are given below. The versions of these functions that don't take a dnodesize parameter now just call the _dnsize() versions with a dnodesize of 0, which means use the legacy dnode size. New DMU interfaces: dmu_object_alloc_dnsize() dmu_object_claim_dnsize() dmu_object_reclaim_dnsize() New ZAP interfaces: zap_create_dnsize() zap_create_norm_dnsize() zap_create_flags_dnsize() zap_create_claim_norm_dnsize() zap_create_link_dnsize() The constant DN_MAX_BONUSLEN is renamed to DN_OLD_MAX_BONUSLEN. The spa_maxdnodesize() function should be used to determine the maximum bonus length for a pool. These are a few noteworthy changes to key functions: * The prototype for dnode_hold_impl() now takes a "slots" parameter. When the DNODE_MUST_BE_FREE flag is set, this parameter is used to ensure the hole at the specified object offset is large enough to hold the dnode being created. The slots parameter is also used to ensure a dnode does not span multiple dnode blocks. In both of these cases, if a failure occurs, ENOSPC is returned. Keep in mind, these failure cases are only possible when using DNODE_MUST_BE_FREE. If the DNODE_MUST_BE_ALLOCATED flag is set, "slots" must be 0. dnode_hold_impl() will check if the requested dnode is already consumed as an extra dnode slot by an large dnode, in which case it returns ENOENT. * The function dmu_object_alloc() advances to the next dnode block if dnode_hold_impl() returns an error for a requested object. This is because the beginning of the next dnode block is the only location it can safely assume to either be a hole or a valid starting point for a dnode. * dnode_next_offset_level() and other functions that iterate through dnode blocks may no longer use a simple array indexing scheme. These now use the current dnode's dn_num_slots field to advance to the next dnode in the block. This is to ensure we properly skip the current dnode's bonus area and don't interpret it as a valid dnode. zdb --- The zdb command was updated to display a dnode's size under the "dnsize" column when the object is dumped. For ZIL create log records, zdb will now display the slot count for the object. ztest ----- Ztest chooses a random dnodesize for every newly created object. The random distribution is more heavily weighted toward small dnodes to better simulate real-world datasets. Unused bonus buffer space is filled with non-zero values computed from the object number, dataset id, offset, and generation number. This helps ensure that the dnode traversal code properly skips the interior regions of large dnodes, and that these interior regions are not overwritten by data belonging to other dnodes. A new test visits each object in a dataset. It verifies that the actual dnode size matches what was stored in the ztest block tag when it was created. It also verifies that the unused bonus buffer space is filled with the expected data patterns. ZFS Test Suite -------------- Added six new large dnode-specific tests, and integrated the dnodesize property into existing tests for zfs allow and send/recv. Send/Receive ------------ ZFS send streams for datasets containing large dnodes cannot be received on pools that don't support the large_dnode feature. A send stream with large dnodes sets a DMU_BACKUP_FEATURE_LARGE_DNODE flag which will be unrecognized by an incompatible receiving pool so that the zfs receive will fail gracefully. While not implemented here, it may be possible to generate a backward-compatible send stream from a dataset containing large dnodes. The implementation may be tricky, however, because the send object record for a large dnode would need to be resized to a 512 byte dnode, possibly kicking in a spill block in the process. This means we would need to construct a new SA layout and possibly register it in the SA layout object. The SA layout is normally just sent as an ordinary object record. But if we are constructing new layouts while generating the send stream we'd have to build the SA layout object dynamically and send it at the end of the stream. For sending and receiving between pools that do support large dnodes, the drr_object send record type is extended with a new field to store the dnode slot count. This field was repurposed from unused padding in the structure. ZIL Replay ---------- The dnode slot count is stored in the uppermost 8 bits of the lr_foid field. The bits were unused as the object id is currently capped at 48 bits. Resizing Dnodes --------------- It should be possible to resize a dnode when it is dirtied if the current dnodesize dataset property differs from the dnode's size, but this functionality is not currently implemented. Clearly a dnode can only grow if there are sufficient contiguous unused slots in the dnode block, but it should always be possible to shrink a dnode. Growing dnodes may be useful to reduce fragmentation in a pool with many spill blocks in use. Shrinking dnodes may be useful to allow sending a dataset to a pool that doesn't support the large_dnode feature. Feature Reference Counting -------------------------- The reference count for the large_dnode pool feature tracks the number of datasets that have ever contained a dnode of size larger than 512 bytes. The first time a large dnode is created in a dataset the dataset is converted to an extensible dataset. This is a one-way operation and the only way to decrement the feature count is to destroy the dataset, even if the dataset no longer contains any large dnodes. The complexity of reference counting on a per-dnode basis was too high, so we chose to track it on a per-dataset basis similarly to the large_block feature. Signed-off-by: Ned Bass <bass6@llnl.gov> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes #3542 |
#
7fca1b93 |
|
02-Aug-2018 |
Alexander Motin <mav@FreeBSD.org> |
Do not blindly include illumos kernel headers instead of user-space. It is not needed now, and I doubt it much helped at all, creating more confusions then good. |
#
1c05a6ea |
|
26-Oct-2017 |
Andriy Gapon <avg@FreeBSD.org> |
MFV r325013,r325034: 640 number_to_scaled_string is duplicated in several commands illumos/illumos-gate@0a0551200ecbcd4f1b17560acaeeb7b6c8b0090e https://github.com/illumos/illumos-gate/commit/0a0551200ecbcd4f1b17560acaeeb7b6c8b0090e https://www.illumos.org/issues/640 du(1), df(1m), ls(1), and swap(1m) all include a copy (it appears literally copied) of the 'number_to_scaled_string' function in their source. This should be moved to a shared library and all 4 commands should use this instead. FreeBSD note: of all libcmdutils functionality ZFS (and other illumos contrib code) currently uses only nicenum() function (which is similar to humanize_number but has some formatting differences). For this reason I decided to not port the whole library. As a result, nicenum.c from libcmdutils is compiled into libzfs and libzpool. This is a bit ugly, but works. If one day we are forced to create libillumos, then the file should be moved to that library. Reviewed by: Sebastian Wiedenroth <wiedi@frubar.net> Reviewed by: Robert Mustacchi <rm@joyent.com> Reviewed by: Yuri Pankov <yuripv@gmx.com> Approved by: Dan McDonald <danmcd@joyent.com> Author: Jason King <jason.brian.king@gmail.com> MFC after: 2 weeks
|
#
acc37ca1 |
|
04-Mar-2017 |
Enji Cooper <ngie@FreeBSD.org> |
cddl: normalize paths using SRCTOP-relative paths or :H when possible This simplifies make logic/output While here, remove bogus CFLAGS which look for headers in cddl/lib/libumem. There aren't any source files there (just Makefiles) MFC after: 1 month Sponsored by: Dell EMC Isilon |
#
eacae6dc |
|
03-Dec-2015 |
Bryan Drewery <bdrewery@FreeBSD.org> |
Fix LDADD/DPADD that should be LIBADD. Sponsored by: EMC / Isilon Storage Division |
#
3bdf7758 |
|
12-Apr-2014 |
Warner Losh <imp@FreeBSD.org> |
NO_MAN= has been deprecated in favor of MAN= for some time, go ahead and finish the job. ncurses is now the only Makefile in the tree that uses it since it wasn't a simple mechanical change, and will be addressed in a future commit. |
#
5a5347c3 |
|
26-Aug-2013 |
Will Andrews <will@FreeBSD.org> |
Build all ZFS testing & debugging tools with -g. These programs and everything using libzpool rely on the embedded asserts to verify the correctness of operations. Given that, the core dumps would be useless without debug symbols. |
#
dce1a726 |
|
05-Mar-2013 |
Martin Matuska <mm@FreeBSD.org> |
WiP merge of libzfs_core (MFV r238590, r238592) not yet working, ioctl handling needs to be changed
|
#
4c5238d5 |
|
12-Sep-2012 |
Martin Matuska <mm@FreeBSD.org> |
Merge recent zfs vendor changes, sync code and adjust userland DEBUG. Illumos issued covered: 1884 Empty "used" field for zfs *space commands 3006 VERIFY[S,U,P] and ASSERT[S,U,P] frequently check if first argument is zero 3028 zfs {group,user}space -n prints (null) instead of numeric GID/UID 3048 zfs {user,group}space [-s|-S] is broken 3049 zfs {user,group}space -t doesn't really filter the results 3060 zfs {user,group}space -H output isn't tab-delimited 3061 zfs {user,group}space -o doesn't use specified fields order 3064 usr/src/cmd/zpool/zpool_main.c misspells "successful" 3093 zfs {user,group}space's -i is noop 3098 zfs userspace/groupspace fail without saying why when run as non-root References: https://www.illumos.org/issues/ + [issue_id] Obtained from: illumos (vendor/illumos, vendor/illumos-sys) MFC after: 2 weeks
|
#
2d9cf57e |
|
11-Jun-2012 |
Martin Matuska <mm@FreeBSD.org> |
Introduce "feature flags" for ZFS pools (bump SPA version to 5000). Add first feature "com.delphix:async_destroy" (asynchronous destroy of ZFS datasets). Implement features support in ZFS boot code. Illumos revisions merged: 13700:2889e2596bd6 13701:1949b688d5fb 2619 asynchronous destruction of ZFS file systems 2747 SPA versioning with zfs feature flags References: https://www.illumos.org/issues/2619 https://www.illumos.org/issues/2747 Obtained from: illumos (issue #2619, #2747) MFC after: 1 month |
#
35362067 |
|
30-Oct-2011 |
Robert Millan <rmh@FreeBSD.org> |
Fix a few gratuitous library dependencies. Some of the ZFS utilities are linked with libraries they don't use: - zinject doesn't use libavl - ztest doesn't use libz - zdb uses neither libavl nor libz - zfs uses neither libbsdxml nor libm, nor libsbuf - zpool uses neither libbsdxml nor libm, nor libsbuf In addition, libzfs needs libm because it uses pow(), however it isn't linked with -lm. This went unnoticed because all its users had -lm before. Reviewed by: pjd, mm Approved by: kib (mentor) MFC after: 1 week |
#
10b9d77b |
|
27-Feb-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Finally... Import the latest open-source ZFS version - (SPA) 28. Few new things available from now on: - Data deduplication. - Triple parity RAIDZ (RAIDZ3). - zfs diff. - zpool split. - Snapshot holds. - zpool import -F. Allows to rewind corrupted pool to earlier transaction group. - Possibility to import pool in read-only mode. MFC after: 1 month |
#
4ef20db2 |
|
02-Mar-2010 |
Ulrich Spörlein <uqs@FreeBSD.org> |
Remove manual .includes in cddl Makefiles - Break the dependency on ../Makefile.inc for .PATH, and include ../Makefile.inc implicitly. This is required to ... - Set WARNS?=6 in top-level Makefile.inc - Remove now redundant WARNS settings, add WARNS?=0 where appropriate - Remove redundant SHLIB_MAJOR overrides - Use NO_MAN, not MK_MAN=no - Remove redundant inclusion of bsd.own.mk - Order Makefiles more according to style.Makefile(9) - Reduce diff of cddl Makefiles against each other No objection: pjd Approved by: ed (co-mentor) |
#
300d03a8 |
|
14-Mar-2009 |
Roman Divacky <rdivacky@FreeBSD.org> |
Switch over to gnu99 compilation on default for userland. Tested by: make universe Tested by: ports exp build (done by pav) Reviewed by: ru Reviewed by: silence on arch Approved by: ed (mentor) |
#
1ba4a712 |
|
17-Nov-2008 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Update ZFS from version 6 to 13 and bring some FreeBSD-specific changes. This bring huge amount of changes, I'll enumerate only user-visible changes: - Delegated Administration Allows regular users to perform ZFS operations, like file system creation, snapshot creation, etc. - L2ARC Level 2 cache for ZFS - allows to use additional disks for cache. Huge performance improvements mostly for random read of mostly static content. - slog Allow to use additional disks for ZFS Intent Log to speed up operations like fsync(2). - vfs.zfs.super_owner Allows regular users to perform privileged operations on files stored on ZFS file systems owned by him. Very careful with this one. - chflags(2) Not all the flags are supported. This still needs work. - ZFSBoot Support to boot off of ZFS pool. Not finished, AFAIK. Submitted by: dfr - Snapshot properties - New failure modes Before if write requested failed, system paniced. Now one can select from one of three failure modes: - panic - panic on write error - wait - wait for disk to reappear - continue - serve read requests if possible, block write requests - Refquota, refreservation properties Just quota and reservation properties, but don't count space consumed by children file systems, clones and snapshots. - Sparse volumes ZVOLs that don't reserve space in the pool. - External attributes Compatible with extattr(2). - NFSv4-ACLs Not sure about the status, might not be complete yet. Submitted by: trasz - Creation-time properties - Regression tests for zpool(8) command. Obtained from: OpenSolaris |
#
e327f524 |
|
27-Mar-2008 |
John Birrell <jb@FreeBSD.org> |
The sources covered by Sun's CDDL have been repo copied below the src/cddl and src/sys/cddl directories per the core@ decision following the license review. This change modifies the affected Makefiles to reference the sources in their new location. |
#
03768697 |
|
01-Oct-2007 |
Ruslan Ermilov <ru@FreeBSD.org> |
Fixed static linkage (build with -DNO_SHARED). Approved by: re (kensmith) |
#
1c22c47c |
|
17-Apr-2007 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Link libzpool, ztest and zdb against libpthread. Requested by: ru |
#
5941f035 |
|
16-Apr-2007 |
Ruslan Ermilov <ru@FreeBSD.org> |
Apply a set of style.Makefile(5) changes to src/cddl/ makefiles. There are some insignificant non-style changes as well. Not fixed: makefiles use ${LIBTHR} that doesn't exist, thus breaking "make checkdpadd" and not tracking dependencies properly. Approved by: pjd |
#
f0a75d27 |
|
05-Apr-2007 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Please welcome ZFS - The last word in file systems. ZFS file system was ported from OpenSolaris operating system. The code in under CDDL license. I'd like to thank all SUN developers that created this great piece of software. Supported by: Wheel LTD (http://www.wheel.pl/) Supported by: The FreeBSD Foundation (http://www.freebsdfoundation.org/) Supported by: Sentex (http://www.sentex.net/) |
#
73734d6e |
|
16-Nov-2020 |
Brooks Davis <brooks@FreeBSD.org> |
Add missing includes of src.opts.mk Without this "SUBDIR.${MK_TESTS}=tests" would always expand to "SUBDIR.=tests" resulting in the tests not being built. Sponsored by: DARPA
|
#
9e5787d2 |
|
24-Aug-2020 |
Matt Macy <mmacy@FreeBSD.org> |
Merge OpenZFS support in to HEAD. The primary benefit is maintaining a completely shared code base with the community allowing FreeBSD to receive new features sooner and with less effort. I would advise against doing 'zpool upgrade' or creating indispensable pools using new features until this change has had a month+ to soak. Work on merging FreeBSD support in to what was at the time "ZFS on Linux" began in August 2018. I first publicly proposed transitioning FreeBSD to (new) OpenZFS on December 18th, 2018. FreeBSD support in OpenZFS was finally completed in December 2019. A CFT for downstreaming OpenZFS support in to FreeBSD was first issued on July 8th. All issues that were reported have been addressed or, for a couple of less critical matters there are pull requests in progress with OpenZFS. iXsystems has tested and dogfooded extensively internally. The TrueNAS 12 release is based on OpenZFS with some additional features that have not yet made it upstream. Improvements include: project quotas, encrypted datasets, allocation classes, vectorized raidz, vectorized checksums, various command line improvements, zstd compression. Thanks to those who have helped along the way: Ryan Moeller, Allan Jude, Zack Welch, and many others. Sponsored by: iXsystems, Inc. Differential Revision: https://reviews.freebsd.org/D25872
|
#
d12e91d5 |
|
20-Aug-2018 |
Matt Macy <mmacy@FreeBSD.org> |
Make dnode definition uniform on !x86 gcc4 requires -fms-extensions to accept anonymous union members
|
#
450e5a43 |
|
15-Aug-2018 |
Will Andrews <will@FreeBSD.org> |
zfs: add ztest to the kyua test suite. This program is currently failing, and has been for >6 months on HEAD. Ideally, this should be run 24x7 in CI, to discover hard-to-find bugs that only manifest with concurrent i/o. Requested by: lwhsu, mmacy
|
#
cc0fbbb9 |
|
11-Aug-2018 |
Matt Macy <mmacy@FreeBSD.org> |
MFV/ZoL: Implement large_dnode pool feature commit 50c957f702ea6d08a634e42f73e8a49931dd8055 Author: Ned Bass <bass6@llnl.gov> Date: Wed Mar 16 18:25:34 2016 -0700 Implement large_dnode pool feature Justification ------------- This feature adds support for variable length dnodes. Our motivation is to eliminate the overhead associated with using spill blocks. Spill blocks are used to store system attribute data (i.e. file metadata) that does not fit in the dnode's bonus buffer. By allowing a larger bonus buffer area the use of a spill block can be avoided. Spill blocks potentially incur an additional read I/O for every dnode in a dnode block. As a worst case example, reading 32 dnodes from a 16k dnode block and all of the spill blocks could issue 33 separate reads. Now suppose those dnodes have size 1024 and therefore don't need spill blocks. Then the worst case number of blocks read is reduced to from 33 to two--one per dnode block. In practice spill blocks may tend to be co-located on disk with the dnode blocks so the reduction in I/O would not be this drastic. In a badly fragmented pool, however, the improvement could be significant. ZFS-on-Linux systems that make heavy use of extended attributes would benefit from this feature. In particular, ZFS-on-Linux supports the xattr=sa dataset property which allows file extended attribute data to be stored in the dnode bonus buffer as an alternative to the traditional directory-based format. Workloads such as SELinux and the Lustre distributed filesystem often store enough xattr data to force spill bocks when xattr=sa is in effect. Large dnodes may therefore provide a performance benefit to such systems. Other use cases that may benefit from this feature include files with large ACLs and symbolic links with long target names. Furthermore, this feature may be desirable on other platforms in case future applications or features are developed that could make use of a larger bonus buffer area. Implementation -------------- The size of a dnode may be a multiple of 512 bytes up to the size of a dnode block (currently 16384 bytes). A dn_extra_slots field was added to the current on-disk dnode_phys_t structure to describe the size of the physical dnode on disk. The 8 bits for this field were taken from the zero filled dn_pad2 field. The field represents how many "extra" dnode_phys_t slots a dnode consumes in its dnode block. This convention results in a value of 0 for 512 byte dnodes which preserves on-disk format compatibility with older software. Similarly, the in-memory dnode_t structure has a new dn_num_slots field to represent the total number of dnode_phys_t slots consumed on disk. Thus dn->dn_num_slots is 1 greater than the corresponding dnp->dn_extra_slots. This difference in convention was adopted because, unlike on-disk structures, backward compatibility is not a concern for in-memory objects, so we used a more natural way to represent size for a dnode_t. The default size for newly created dnodes is determined by the value of a new "dnodesize" dataset property. By default the property is set to "legacy" which is compatible with older software. Setting the property to "auto" will allow the filesystem to choose the most suitable dnode size. Currently this just sets the default dnode size to 1k, but future code improvements could dynamically choose a size based on observed workload patterns. Dnodes of varying sizes can coexist within the same dataset and even within the same dnode block. For example, to enable automatically-sized dnodes, run # zfs set dnodesize=auto tank/fish The user can also specify literal values for the dnodesize property. These are currently limited to powers of two from 1k to 16k. The power-of-2 limitation is only for simplicity of the user interface. Internally the implementation can handle any multiple of 512 up to 16k, and consumers of the DMU API can specify any legal dnode value. The size of a new dnode is determined at object allocation time and stored as a new field in the znode in-memory structure. New DMU interfaces are added to allow the consumer to specify the dnode size that a newly allocated object should use. Existing interfaces are unchanged to avoid having to update every call site and to preserve compatibility with external consumers such as Lustre. The new interfaces names are given below. The versions of these functions that don't take a dnodesize parameter now just call the _dnsize() versions with a dnodesize of 0, which means use the legacy dnode size. New DMU interfaces: dmu_object_alloc_dnsize() dmu_object_claim_dnsize() dmu_object_reclaim_dnsize() New ZAP interfaces: zap_create_dnsize() zap_create_norm_dnsize() zap_create_flags_dnsize() zap_create_claim_norm_dnsize() zap_create_link_dnsize() The constant DN_MAX_BONUSLEN is renamed to DN_OLD_MAX_BONUSLEN. The spa_maxdnodesize() function should be used to determine the maximum bonus length for a pool. These are a few noteworthy changes to key functions: * The prototype for dnode_hold_impl() now takes a "slots" parameter. When the DNODE_MUST_BE_FREE flag is set, this parameter is used to ensure the hole at the specified object offset is large enough to hold the dnode being created. The slots parameter is also used to ensure a dnode does not span multiple dnode blocks. In both of these cases, if a failure occurs, ENOSPC is returned. Keep in mind, these failure cases are only possible when using DNODE_MUST_BE_FREE. If the DNODE_MUST_BE_ALLOCATED flag is set, "slots" must be 0. dnode_hold_impl() will check if the requested dnode is already consumed as an extra dnode slot by an large dnode, in which case it returns ENOENT. * The function dmu_object_alloc() advances to the next dnode block if dnode_hold_impl() returns an error for a requested object. This is because the beginning of the next dnode block is the only location it can safely assume to either be a hole or a valid starting point for a dnode. * dnode_next_offset_level() and other functions that iterate through dnode blocks may no longer use a simple array indexing scheme. These now use the current dnode's dn_num_slots field to advance to the next dnode in the block. This is to ensure we properly skip the current dnode's bonus area and don't interpret it as a valid dnode. zdb --- The zdb command was updated to display a dnode's size under the "dnsize" column when the object is dumped. For ZIL create log records, zdb will now display the slot count for the object. ztest ----- Ztest chooses a random dnodesize for every newly created object. The random distribution is more heavily weighted toward small dnodes to better simulate real-world datasets. Unused bonus buffer space is filled with non-zero values computed from the object number, dataset id, offset, and generation number. This helps ensure that the dnode traversal code properly skips the interior regions of large dnodes, and that these interior regions are not overwritten by data belonging to other dnodes. A new test visits each object in a dataset. It verifies that the actual dnode size matches what was stored in the ztest block tag when it was created. It also verifies that the unused bonus buffer space is filled with the expected data patterns. ZFS Test Suite -------------- Added six new large dnode-specific tests, and integrated the dnodesize property into existing tests for zfs allow and send/recv. Send/Receive ------------ ZFS send streams for datasets containing large dnodes cannot be received on pools that don't support the large_dnode feature. A send stream with large dnodes sets a DMU_BACKUP_FEATURE_LARGE_DNODE flag which will be unrecognized by an incompatible receiving pool so that the zfs receive will fail gracefully. While not implemented here, it may be possible to generate a backward-compatible send stream from a dataset containing large dnodes. The implementation may be tricky, however, because the send object record for a large dnode would need to be resized to a 512 byte dnode, possibly kicking in a spill block in the process. This means we would need to construct a new SA layout and possibly register it in the SA layout object. The SA layout is normally just sent as an ordinary object record. But if we are constructing new layouts while generating the send stream we'd have to build the SA layout object dynamically and send it at the end of the stream. For sending and receiving between pools that do support large dnodes, the drr_object send record type is extended with a new field to store the dnode slot count. This field was repurposed from unused padding in the structure. ZIL Replay ---------- The dnode slot count is stored in the uppermost 8 bits of the lr_foid field. The bits were unused as the object id is currently capped at 48 bits. Resizing Dnodes --------------- It should be possible to resize a dnode when it is dirtied if the current dnodesize dataset property differs from the dnode's size, but this functionality is not currently implemented. Clearly a dnode can only grow if there are sufficient contiguous unused slots in the dnode block, but it should always be possible to shrink a dnode. Growing dnodes may be useful to reduce fragmentation in a pool with many spill blocks in use. Shrinking dnodes may be useful to allow sending a dataset to a pool that doesn't support the large_dnode feature. Feature Reference Counting -------------------------- The reference count for the large_dnode pool feature tracks the number of datasets that have ever contained a dnode of size larger than 512 bytes. The first time a large dnode is created in a dataset the dataset is converted to an extensible dataset. This is a one-way operation and the only way to decrement the feature count is to destroy the dataset, even if the dataset no longer contains any large dnodes. The complexity of reference counting on a per-dnode basis was too high, so we chose to track it on a per-dataset basis similarly to the large_block feature. Signed-off-by: Ned Bass <bass6@llnl.gov> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes #3542
|
#
7fca1b93 |
|
02-Aug-2018 |
Alexander Motin <mav@FreeBSD.org> |
Do not blindly include illumos kernel headers instead of user-space. It is not needed now, and I doubt it much helped at all, creating more confusions then good.
|
#
acc37ca1 |
|
04-Mar-2017 |
Enji Cooper <ngie@FreeBSD.org> |
cddl: normalize paths using SRCTOP-relative paths or :H when possible This simplifies make logic/output While here, remove bogus CFLAGS which look for headers in cddl/lib/libumem. There aren't any source files there (just Makefiles) MFC after: 1 month Sponsored by: Dell EMC Isilon
|
#
eacae6dc |
|
03-Dec-2015 |
Bryan Drewery <bdrewery@FreeBSD.org> |
Fix LDADD/DPADD that should be LIBADD. Sponsored by: EMC / Isilon Storage Division
|
#
3bdf7758 |
|
12-Apr-2014 |
Warner Losh <imp@FreeBSD.org> |
NO_MAN= has been deprecated in favor of MAN= for some time, go ahead and finish the job. ncurses is now the only Makefile in the tree that uses it since it wasn't a simple mechanical change, and will be addressed in a future commit.
|
#
5a5347c3 |
|
26-Aug-2013 |
Will Andrews <will@FreeBSD.org> |
Build all ZFS testing & debugging tools with -g. These programs and everything using libzpool rely on the embedded asserts to verify the correctness of operations. Given that, the core dumps would be useless without debug symbols.
|
#
2d9cf57e |
|
11-Jun-2012 |
Martin Matuska <mm@FreeBSD.org> |
Introduce "feature flags" for ZFS pools (bump SPA version to 5000). Add first feature "com.delphix:async_destroy" (asynchronous destroy of ZFS datasets). Implement features support in ZFS boot code. Illumos revisions merged: 13700:2889e2596bd6 13701:1949b688d5fb 2619 asynchronous destruction of ZFS file systems 2747 SPA versioning with zfs feature flags References: https://www.illumos.org/issues/2619 https://www.illumos.org/issues/2747 Obtained from: illumos (issue #2619, #2747) MFC after: 1 month
|
#
35362067 |
|
30-Oct-2011 |
Robert Millan <rmh@FreeBSD.org> |
Fix a few gratuitous library dependencies. Some of the ZFS utilities are linked with libraries they don't use: - zinject doesn't use libavl - ztest doesn't use libz - zdb uses neither libavl nor libz - zfs uses neither libbsdxml nor libm, nor libsbuf - zpool uses neither libbsdxml nor libm, nor libsbuf In addition, libzfs needs libm because it uses pow(), however it isn't linked with -lm. This went unnoticed because all its users had -lm before. Reviewed by: pjd, mm Approved by: kib (mentor) MFC after: 1 week
|
#
10b9d77b |
|
27-Feb-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Finally... Import the latest open-source ZFS version - (SPA) 28. Few new things available from now on: - Data deduplication. - Triple parity RAIDZ (RAIDZ3). - zfs diff. - zpool split. - Snapshot holds. - zpool import -F. Allows to rewind corrupted pool to earlier transaction group. - Possibility to import pool in read-only mode. MFC after: 1 month
|
#
a7d5f7eb |
|
19-Oct-2010 |
Jamie Gritton <jamie@FreeBSD.org> |
A new jail(8) with a configuration file, to replace the work currently done by /etc/rc.d/jail.
|
#
fe0506d7 |
|
09-Mar-2010 |
Marcel Moolenaar <marcel@FreeBSD.org> |
Create the altix project branch. The altix project will add support for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting is a two-module system, consisting of a base compute module and a CPU expansion module. SGI's NUMAFlex architecture can be an excellent platform to test CPU affinity and NUMA-aware features in FreeBSD.
|
#
4ef20db2 |
|
02-Mar-2010 |
Ulrich Spörlein <uqs@FreeBSD.org> |
Remove manual .includes in cddl Makefiles - Break the dependency on ../Makefile.inc for .PATH, and include ../Makefile.inc implicitly. This is required to ... - Set WARNS?=6 in top-level Makefile.inc - Remove now redundant WARNS settings, add WARNS?=0 where appropriate - Remove redundant SHLIB_MAJOR overrides - Use NO_MAN, not MK_MAN=no - Remove redundant inclusion of bsd.own.mk - Order Makefiles more according to style.Makefile(9) - Reduce diff of cddl Makefiles against each other No objection: pjd Approved by: ed (co-mentor)
|
#
300d03a8 |
|
14-Mar-2009 |
Roman Divacky <rdivacky@FreeBSD.org> |
Switch over to gnu99 compilation on default for userland. Tested by: make universe Tested by: ports exp build (done by pav) Reviewed by: ru Reviewed by: silence on arch Approved by: ed (mentor)
|
#
1ba4a712 |
|
17-Nov-2008 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Update ZFS from version 6 to 13 and bring some FreeBSD-specific changes. This bring huge amount of changes, I'll enumerate only user-visible changes: - Delegated Administration Allows regular users to perform ZFS operations, like file system creation, snapshot creation, etc. - L2ARC Level 2 cache for ZFS - allows to use additional disks for cache. Huge performance improvements mostly for random read of mostly static content. - slog Allow to use additional disks for ZFS Intent Log to speed up operations like fsync(2). - vfs.zfs.super_owner Allows regular users to perform privileged operations on files stored on ZFS file systems owned by him. Very careful with this one. - chflags(2) Not all the flags are supported. This still needs work. - ZFSBoot Support to boot off of ZFS pool. Not finished, AFAIK. Submitted by: dfr - Snapshot properties - New failure modes Before if write requested failed, system paniced. Now one can select from one of three failure modes: - panic - panic on write error - wait - wait for disk to reappear - continue - serve read requests if possible, block write requests - Refquota, refreservation properties Just quota and reservation properties, but don't count space consumed by children file systems, clones and snapshots. - Sparse volumes ZVOLs that don't reserve space in the pool. - External attributes Compatible with extattr(2). - NFSv4-ACLs Not sure about the status, might not be complete yet. Submitted by: trasz - Creation-time properties - Regression tests for zpool(8) command. Obtained from: OpenSolaris
|
#
d7f03759 |
|
19-Oct-2008 |
Ulf Lilleengen <lulf@FreeBSD.org> |
- Import the HEAD csup code which is the basis for the cvsmode work.
|
#
e327f524 |
|
27-Mar-2008 |
John Birrell <jb@FreeBSD.org> |
The sources covered by Sun's CDDL have been repo copied below the src/cddl and src/sys/cddl directories per the core@ decision following the license review. This change modifies the affected Makefiles to reference the sources in their new location.
|
#
03768697 |
|
01-Oct-2007 |
Ruslan Ermilov <ru@FreeBSD.org> |
Fixed static linkage (build with -DNO_SHARED). Approved by: re (kensmith)
|
#
1c22c47c |
|
17-Apr-2007 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Link libzpool, ztest and zdb against libpthread. Requested by: ru
|
#
5941f035 |
|
16-Apr-2007 |
Ruslan Ermilov <ru@FreeBSD.org> |
Apply a set of style.Makefile(5) changes to src/cddl/ makefiles. There are some insignificant non-style changes as well. Not fixed: makefiles use ${LIBTHR} that doesn't exist, thus breaking "make checkdpadd" and not tracking dependencies properly. Approved by: pjd
|
#
f0a75d27 |
|
05-Apr-2007 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Please welcome ZFS - The last word in file systems. ZFS file system was ported from OpenSolaris operating system. The code in under CDDL license. I'd like to thank all SUN developers that created this great piece of software. Supported by: Wheel LTD (http://www.wheel.pl/) Supported by: The FreeBSD Foundation (http://www.freebsdfoundation.org/) Supported by: Sentex (http://www.sentex.net/)
|