Enable hibernate/resume to nvme(4) disks with 4096 byte sectors.

testing by florian@ mglocker@ mlarkin@

ok deraadt@ mglocker@ mlarkin@

Allow more memory ranges in hibernate

The previous limit of VM_PHYSSEG_MAX ranges (16) was proving too small for
newer machines. This diff reorganizes the hibernate signature block to allow
for 22 ranges by removing the kernel version comparison and replacing it
with a SHA of several unique kernel features (the version string and several
addresses of functions not inside the same .o).

Reported by claudio@, who also helped fix some issues in the diff. Input
from deraadt@ as well.

Tested by myself and claudio on a variety of machines. Only compile tested on
i386 as I have no more S4-capable i386 hardware anymore.

ok claudio@

Remove trailing whitespace in a few places. No code change.

hibernate_clear_signature() is only used by hibernate_resume(), so
pass in the already read hibernate_info instead of reading it again.

ok deraadt@

Save and restore retguard area during hibernate unpack. This copies the
original retguard data to the piglet and bcopys it back in place
immediately before resuming via the ACPI Sx trampoline.

ok deraadt, guenther, tested by many.

calculate a "sum" based upon pointers to functions all over the kernel,
so that an unhibernate kernel can detect if it is running with the
kernel it booted.
ok mlarkin

move from RB macros to RBT functions

New framework that allows hibernate to pass in entropy from it's fresh
boot.
ok mlarkin

remove an unused function and some unused variables in hibernate

Rework piglet and pig allocation. Currently the piglet gets allocated
deep down in the suspend path, where it is really hard to recover from
allocation failure. So allocate the piglet early on in the suspend path.
Also change the piglet and piglet allocation functions to use km_alloc(9)
instead of doing pmemrange magic. This removes a bunch of code which, in the
case of the piglet allocation, is broken since it results in a NULL pointer
dereference. Also switch the piglet allocation to not wait. If we can't
allocate 16MB of phys contig memory on a halfway modern machine we're almost
certainly under a lot of memory pressure and we're better off not trying to
hibernate anyway.

ok mlarkin@

Support hibernating to softraid crypto volumes.

much help and ok from deraadt@

Reenable hibernate RLE support and flush+zero all memory after unpack.

Add a function to drop all clean pages on the page daemon queues and call
it when we hibernate.

ok mlarkin@, miod@, deraadt@

Flush the buffercache to 16MB on hibernate and restore its previous max
size (kern.bufcachepercent) on resume, for better hibernate performance.

ok beck@

Cleanup the chunk placement routine by removing the conflict resolver.
Chunks are now sorted by ascending PA and all chunks are bounced before
unpack. This fixes an issue where the trampoline chunks were being placed
at the end of the unpack ordering, causing overwrite during unpack.

Use suspending kernel's stack smash guard to avoid panicing during unpack.

ok deraadt@

Remove hibernate_get_next_rle function (unused, and we need to redo it
anyway as we move toward a streamed implmentation)

unbias the chunks and chunktable writing and reading. as a result, it
is now possible to move the chunktable right after the chunks, not at
the end of the swap.
ok mlarkin

shorten dev_t in hib_info

use DEV_BSIZE instead of "secsize"
ok mlarkin

teach the side-effect free drivers about the partition they are dealing
by passing a start/length in the HIB_INIT op. Then rebase all
hibernate-time block offsets to be relative to the start of that partition.
This simplifies things a lot.
ok mlarkin

Removed some unused code

Add a magic number to the head of the signature block. Check for magic
number match during signature block read during speculative unhibernate on
boot. If the magic number matches but we have otherwise chosen to not
unhibernate (due to kernel/memory mismatch), clear the signature block
early to avoid accidentally trying to unhibernate on subsequent boots. This
prevents accidental unhibernates and endless unhibernate/reboot cycles.

Add a define for HIBERNATE_DEBUG for various debugging printfs (disabled by
default).

Finally, change some KASSERTs to warning printfs (they probably shouldn't
have been KASSERTs in the first place).

"looks good" deraadt@

Add a hibernate HIB_DONE op. After the memory-side-effect driver finishes
IO to the disk, DVACT_RESUME the controller back to normal operation. That
allows us to do the full DVACT_POWERDOWN sequence afterwards.
ok mlarkin

Reorganize some hibernate functions for easier readability.
Fix some incorrect/old comments.

ok deraadt@

use regular i/o functions in the hibernate resume path. ahci(4) hibernate
i/o will not mix with regular i/o so it can only be used in the hibernate
path.

ok deraadt@ mlarkin@

Fix an integer math error when using the result of uvm_page_rle, and
at the same time increase said function's max RLE page count return value.

Add hooks in the right places to call the hibernate suspend and resume
routines, so that we can enable hibernation with a HIBERNATE option
line in GENERIC and appropriate acpi.c goo.

discussed on and off with deraadt@ over the past few months

ready this for big-endian support later on
ok mlarkin

Reduce use of globals in hibernate code.

discussed with deraadt@

Use a fixed io_page for all hibernate I/O, which is needed for
ahci_hibernate_io, a skeleton of which is also provided in this diff.

This code is from deraadt@. Tested on a few wd machines to ensure it works
there as well.

Add some #defines for the various hibernate I/O modes, some of the
groundwork for *_hibernate_io functions other than wd_hibernate_io

These changes were sent to me by deraadt@

In hibernate resume, free the piglet and other VAs we allocated during
suspend.

ok pirofti@

KNF of mlarkin's code, requested by him. Some improvements to the interface
for talking to the disk driver snuck in.
ok mlarkin

Perform most of the remaining refactoring of hibernate code into
MI/MD parts. This also introduces a chunk placement routine that was
originally developed at c2k11 with help from drahn and ariane.

There are still a few more things to do for hibernate, but those can be
worked on in-tree. This code is disabled by default, and not yet called.

ok deraadt@ (and deraadt@ said kettenis@ also ok'ed it :) )

Add hibernate_read_block and fix a couple of typos in the previous commit.

Call (temporarily, until we have RLE page encoding) uvm_pmr_zero_everything
on suspend to ensure we get good zlib compression.

Add MI signature block (hibernate_info) comparison routine

Extract hibernate_write_signature and hibernate_clear_signature to the MI
hibernate code, and add chunk range overlap checking.

Extract MI pmap function hibernate_enter_resume_mapping, refactor old i386
resume pmap code to match.

Add hibernate deflater and inflater and cache flush routines.

Code is not presently called or automatically built.

Add zlib reset, alloc, and free functions for hibernate image compression

Separate some MD and MI bits and a bit of refactoring to make subsequent
commits easier.

Work in progress, hibernate will still not work for you.

ok deraadt@

Ensure all pages in pmemrange can be marked as dirty.
It'd be a very bad idea to hand out dirty pages as zeroed, just because
we came back from hibernate.

No callers at the moment, will be called on hibernate resume path.

Put in RLE logic for hibernate compressor.

These have the potential to compress 1MB of physmem into 1 byte.
This works by noting the page is not in use and therefor skipping it.

Needed by mlarkin@ for hibernate. No callers yet.

Change pig allocator to a highest-address selection.
Add piglet allocator, which does a lowest-address selection.

No callers yet, needed by mlarkin@ for hibernate voodoo.

Move uvm_pmr_alloc_pig to kern/subr_hibernate.c

No callers, no functional change.

Move uvm_pmr_zero_everything() to subr_hibernate.

This function will probably die before ever being called
from the in-tree code, since hibernate will move to RLE encoding.

No functional change, function had no callers.

Move hiballoc to hibernate.h, subr_hibernate.c
Next few commits will move other hibernate-specific functionality,
like the pig-allocator, to subr_hibernate.

No functional change, no callers either.

Allow more memory ranges in hibernate

The previous limit of VM_PHYSSEG_MAX ranges (16) was proving too small for
newer machines. This diff reorganizes the hibernate signature block to allow
for 22 ranges by removing the kernel version comparison and replacing it
with a SHA of several unique kernel features (the version string and several
addresses of functions not inside the same .o).

Reported by claudio@, who also helped fix some issues in the diff. Input
from deraadt@ as well.

Tested by myself and claudio on a variety of machines. Only compile tested on
i386 as I have no more S4-capable i386 hardware anymore.

ok claudio@

Remove trailing whitespace in a few places. No code change.

hibernate_clear_signature() is only used by hibernate_resume(), so
pass in the already read hibernate_info instead of reading it again.

ok deraadt@

Save and restore retguard area during hibernate unpack. This copies the
original retguard data to the piglet and bcopys it back in place
immediately before resuming via the ACPI Sx trampoline.

ok deraadt, guenther, tested by many.

calculate a "sum" based upon pointers to functions all over the kernel,
so that an unhibernate kernel can detect if it is running with the
kernel it booted.
ok mlarkin

move from RB macros to RBT functions

New framework that allows hibernate to pass in entropy from it's fresh
boot.
ok mlarkin

remove an unused function and some unused variables in hibernate

Rework piglet and pig allocation. Currently the piglet gets allocated
deep down in the suspend path, where it is really hard to recover from
allocation failure. So allocate the piglet early on in the suspend path.
Also change the piglet and piglet allocation functions to use km_alloc(9)
instead of doing pmemrange magic. This removes a bunch of code which, in the
case of the piglet allocation, is broken since it results in a NULL pointer
dereference. Also switch the piglet allocation to not wait. If we can't
allocate 16MB of phys contig memory on a halfway modern machine we're almost
certainly under a lot of memory pressure and we're better off not trying to
hibernate anyway.

ok mlarkin@

Support hibernating to softraid crypto volumes.

much help and ok from deraadt@

Reenable hibernate RLE support and flush+zero all memory after unpack.

Add a function to drop all clean pages on the page daemon queues and call
it when we hibernate.

ok mlarkin@, miod@, deraadt@

Flush the buffercache to 16MB on hibernate and restore its previous max
size (kern.bufcachepercent) on resume, for better hibernate performance.

ok beck@

Cleanup the chunk placement routine by removing the conflict resolver.
Chunks are now sorted by ascending PA and all chunks are bounced before
unpack. This fixes an issue where the trampoline chunks were being placed
at the end of the unpack ordering, causing overwrite during unpack.

Use suspending kernel's stack smash guard to avoid panicing during unpack.

ok deraadt@

Remove hibernate_get_next_rle function (unused, and we need to redo it
anyway as we move toward a streamed implmentation)

unbias the chunks and chunktable writing and reading. as a result, it
is now possible to move the chunktable right after the chunks, not at
the end of the swap.
ok mlarkin

shorten dev_t in hib_info

use DEV_BSIZE instead of "secsize"
ok mlarkin

teach the side-effect free drivers about the partition they are dealing
by passing a start/length in the HIB_INIT op. Then rebase all
hibernate-time block offsets to be relative to the start of that partition.
This simplifies things a lot.
ok mlarkin

Removed some unused code

Add a magic number to the head of the signature block. Check for magic
number match during signature block read during speculative unhibernate on
boot. If the magic number matches but we have otherwise chosen to not
unhibernate (due to kernel/memory mismatch), clear the signature block
early to avoid accidentally trying to unhibernate on subsequent boots. This
prevents accidental unhibernates and endless unhibernate/reboot cycles.

Add a define for HIBERNATE_DEBUG for various debugging printfs (disabled by
default).

Finally, change some KASSERTs to warning printfs (they probably shouldn't
have been KASSERTs in the first place).

"looks good" deraadt@

Add a hibernate HIB_DONE op. After the memory-side-effect driver finishes
IO to the disk, DVACT_RESUME the controller back to normal operation. That
allows us to do the full DVACT_POWERDOWN sequence afterwards.
ok mlarkin

Reorganize some hibernate functions for easier readability.
Fix some incorrect/old comments.

ok deraadt@

use regular i/o functions in the hibernate resume path. ahci(4) hibernate
i/o will not mix with regular i/o so it can only be used in the hibernate
path.

ok deraadt@ mlarkin@

Fix an integer math error when using the result of uvm_page_rle, and
at the same time increase said function's max RLE page count return value.

Add hooks in the right places to call the hibernate suspend and resume
routines, so that we can enable hibernation with a HIBERNATE option
line in GENERIC and appropriate acpi.c goo.

discussed on and off with deraadt@ over the past few months

ready this for big-endian support later on
ok mlarkin

Reduce use of globals in hibernate code.

discussed with deraadt@

Use a fixed io_page for all hibernate I/O, which is needed for
ahci_hibernate_io, a skeleton of which is also provided in this diff.

This code is from deraadt@. Tested on a few wd machines to ensure it works
there as well.

Add some #defines for the various hibernate I/O modes, some of the
groundwork for *_hibernate_io functions other than wd_hibernate_io

These changes were sent to me by deraadt@

In hibernate resume, free the piglet and other VAs we allocated during
suspend.

ok pirofti@

KNF of mlarkin's code, requested by him. Some improvements to the interface
for talking to the disk driver snuck in.
ok mlarkin

Perform most of the remaining refactoring of hibernate code into
MI/MD parts. This also introduces a chunk placement routine that was
originally developed at c2k11 with help from drahn and ariane.

There are still a few more things to do for hibernate, but those can be
worked on in-tree. This code is disabled by default, and not yet called.

ok deraadt@ (and deraadt@ said kettenis@ also ok'ed it :) )

Add hibernate_read_block and fix a couple of typos in the previous commit.

Call (temporarily, until we have RLE page encoding) uvm_pmr_zero_everything
on suspend to ensure we get good zlib compression.

Add MI signature block (hibernate_info) comparison routine

Extract hibernate_write_signature and hibernate_clear_signature to the MI
hibernate code, and add chunk range overlap checking.

Extract MI pmap function hibernate_enter_resume_mapping, refactor old i386
resume pmap code to match.

Add hibernate deflater and inflater and cache flush routines.

Code is not presently called or automatically built.

Add zlib reset, alloc, and free functions for hibernate image compression

Separate some MD and MI bits and a bit of refactoring to make subsequent
commits easier.

Work in progress, hibernate will still not work for you.

ok deraadt@

Ensure all pages in pmemrange can be marked as dirty.
It'd be a very bad idea to hand out dirty pages as zeroed, just because
we came back from hibernate.

No callers at the moment, will be called on hibernate resume path.

Put in RLE logic for hibernate compressor.

These have the potential to compress 1MB of physmem into 1 byte.
This works by noting the page is not in use and therefor skipping it.

Needed by mlarkin@ for hibernate. No callers yet.

Change pig allocator to a highest-address selection.
Add piglet allocator, which does a lowest-address selection.

No callers yet, needed by mlarkin@ for hibernate voodoo.

Move uvm_pmr_alloc_pig to kern/subr_hibernate.c

No callers, no functional change.

Move uvm_pmr_zero_everything() to subr_hibernate.

This function will probably die before ever being called
from the in-tree code, since hibernate will move to RLE encoding.

No functional change, function had no callers.

Move hiballoc to hibernate.h, subr_hibernate.c
Next few commits will move other hibernate-specific functionality,
like the pig-allocator, to subr_hibernate.

No functional change, no callers either.

hibernate_clear_signature() is only used by hibernate_resume(), so
pass in the already read hibernate_info instead of reading it again.

ok deraadt@

Save and restore retguard area during hibernate unpack. This copies the
original retguard data to the piglet and bcopys it back in place
immediately before resuming via the ACPI Sx trampoline.

ok deraadt, guenther, tested by many.

calculate a "sum" based upon pointers to functions all over the kernel,
so that an unhibernate kernel can detect if it is running with the
kernel it booted.
ok mlarkin

move from RB macros to RBT functions

New framework that allows hibernate to pass in entropy from it's fresh
boot.
ok mlarkin

remove an unused function and some unused variables in hibernate

Rework piglet and pig allocation. Currently the piglet gets allocated
deep down in the suspend path, where it is really hard to recover from
allocation failure. So allocate the piglet early on in the suspend path.
Also change the piglet and piglet allocation functions to use km_alloc(9)
instead of doing pmemrange magic. This removes a bunch of code which, in the
case of the piglet allocation, is broken since it results in a NULL pointer
dereference. Also switch the piglet allocation to not wait. If we can't
allocate 16MB of phys contig memory on a halfway modern machine we're almost
certainly under a lot of memory pressure and we're better off not trying to
hibernate anyway.

ok mlarkin@

Support hibernating to softraid crypto volumes.

much help and ok from deraadt@

Reenable hibernate RLE support and flush+zero all memory after unpack.

Add a function to drop all clean pages on the page daemon queues and call
it when we hibernate.

ok mlarkin@, miod@, deraadt@

Flush the buffercache to 16MB on hibernate and restore its previous max
size (kern.bufcachepercent) on resume, for better hibernate performance.

ok beck@

Cleanup the chunk placement routine by removing the conflict resolver.
Chunks are now sorted by ascending PA and all chunks are bounced before
unpack. This fixes an issue where the trampoline chunks were being placed
at the end of the unpack ordering, causing overwrite during unpack.

Use suspending kernel's stack smash guard to avoid panicing during unpack.

ok deraadt@

Remove hibernate_get_next_rle function (unused, and we need to redo it
anyway as we move toward a streamed implmentation)

unbias the chunks and chunktable writing and reading. as a result, it
is now possible to move the chunktable right after the chunks, not at
the end of the swap.
ok mlarkin

shorten dev_t in hib_info

use DEV_BSIZE instead of "secsize"
ok mlarkin

teach the side-effect free drivers about the partition they are dealing
by passing a start/length in the HIB_INIT op. Then rebase all
hibernate-time block offsets to be relative to the start of that partition.
This simplifies things a lot.
ok mlarkin

Removed some unused code

Add a magic number to the head of the signature block. Check for magic
number match during signature block read during speculative unhibernate on
boot. If the magic number matches but we have otherwise chosen to not
unhibernate (due to kernel/memory mismatch), clear the signature block
early to avoid accidentally trying to unhibernate on subsequent boots. This
prevents accidental unhibernates and endless unhibernate/reboot cycles.

Add a define for HIBERNATE_DEBUG for various debugging printfs (disabled by
default).

Finally, change some KASSERTs to warning printfs (they probably shouldn't
have been KASSERTs in the first place).

"looks good" deraadt@

Add a hibernate HIB_DONE op. After the memory-side-effect driver finishes
IO to the disk, DVACT_RESUME the controller back to normal operation. That
allows us to do the full DVACT_POWERDOWN sequence afterwards.
ok mlarkin

Reorganize some hibernate functions for easier readability.
Fix some incorrect/old comments.

ok deraadt@

use regular i/o functions in the hibernate resume path. ahci(4) hibernate
i/o will not mix with regular i/o so it can only be used in the hibernate
path.

ok deraadt@ mlarkin@

Fix an integer math error when using the result of uvm_page_rle, and
at the same time increase said function's max RLE page count return value.

Add hooks in the right places to call the hibernate suspend and resume
routines, so that we can enable hibernation with a HIBERNATE option
line in GENERIC and appropriate acpi.c goo.

discussed on and off with deraadt@ over the past few months

ready this for big-endian support later on
ok mlarkin

Reduce use of globals in hibernate code.

discussed with deraadt@

Use a fixed io_page for all hibernate I/O, which is needed for
ahci_hibernate_io, a skeleton of which is also provided in this diff.

This code is from deraadt@. Tested on a few wd machines to ensure it works
there as well.

Add some #defines for the various hibernate I/O modes, some of the
groundwork for *_hibernate_io functions other than wd_hibernate_io

These changes were sent to me by deraadt@

In hibernate resume, free the piglet and other VAs we allocated during
suspend.

ok pirofti@

KNF of mlarkin's code, requested by him. Some improvements to the interface
for talking to the disk driver snuck in.
ok mlarkin

Perform most of the remaining refactoring of hibernate code into
MI/MD parts. This also introduces a chunk placement routine that was
originally developed at c2k11 with help from drahn and ariane.

There are still a few more things to do for hibernate, but those can be
worked on in-tree. This code is disabled by default, and not yet called.

ok deraadt@ (and deraadt@ said kettenis@ also ok'ed it :) )

Add hibernate_read_block and fix a couple of typos in the previous commit.

Call (temporarily, until we have RLE page encoding) uvm_pmr_zero_everything
on suspend to ensure we get good zlib compression.

Add MI signature block (hibernate_info) comparison routine

Extract hibernate_write_signature and hibernate_clear_signature to the MI
hibernate code, and add chunk range overlap checking.

Extract MI pmap function hibernate_enter_resume_mapping, refactor old i386
resume pmap code to match.

Add hibernate deflater and inflater and cache flush routines.

Code is not presently called or automatically built.

Add zlib reset, alloc, and free functions for hibernate image compression

Separate some MD and MI bits and a bit of refactoring to make subsequent
commits easier.

Work in progress, hibernate will still not work for you.

ok deraadt@

Ensure all pages in pmemrange can be marked as dirty.
It'd be a very bad idea to hand out dirty pages as zeroed, just because
we came back from hibernate.

No callers at the moment, will be called on hibernate resume path.

Put in RLE logic for hibernate compressor.

These have the potential to compress 1MB of physmem into 1 byte.
This works by noting the page is not in use and therefor skipping it.

Needed by mlarkin@ for hibernate. No callers yet.

Change pig allocator to a highest-address selection.
Add piglet allocator, which does a lowest-address selection.

No callers yet, needed by mlarkin@ for hibernate voodoo.

Move uvm_pmr_alloc_pig to kern/subr_hibernate.c

No callers, no functional change.

Move uvm_pmr_zero_everything() to subr_hibernate.

This function will probably die before ever being called
from the in-tree code, since hibernate will move to RLE encoding.

No functional change, function had no callers.

Move hiballoc to hibernate.h, subr_hibernate.c
Next few commits will move other hibernate-specific functionality,
like the pig-allocator, to subr_hibernate.

No functional change, no callers either.

Save and restore retguard area during hibernate unpack. This copies the
original retguard data to the piglet and bcopys it back in place
immediately before resuming via the ACPI Sx trampoline.

ok deraadt, guenther, tested by many.

calculate a "sum" based upon pointers to functions all over the kernel,
so that an unhibernate kernel can detect if it is running with the
kernel it booted.
ok mlarkin

move from RB macros to RBT functions

New framework that allows hibernate to pass in entropy from it's fresh
boot.
ok mlarkin

remove an unused function and some unused variables in hibernate

Rework piglet and pig allocation. Currently the piglet gets allocated
deep down in the suspend path, where it is really hard to recover from
allocation failure. So allocate the piglet early on in the suspend path.
Also change the piglet and piglet allocation functions to use km_alloc(9)
instead of doing pmemrange magic. This removes a bunch of code which, in the
case of the piglet allocation, is broken since it results in a NULL pointer
dereference. Also switch the piglet allocation to not wait. If we can't
allocate 16MB of phys contig memory on a halfway modern machine we're almost
certainly under a lot of memory pressure and we're better off not trying to
hibernate anyway.

ok mlarkin@

Support hibernating to softraid crypto volumes.

much help and ok from deraadt@

Reenable hibernate RLE support and flush+zero all memory after unpack.

Add a function to drop all clean pages on the page daemon queues and call
it when we hibernate.

ok mlarkin@, miod@, deraadt@

Flush the buffercache to 16MB on hibernate and restore its previous max
size (kern.bufcachepercent) on resume, for better hibernate performance.

ok beck@

Cleanup the chunk placement routine by removing the conflict resolver.
Chunks are now sorted by ascending PA and all chunks are bounced before
unpack. This fixes an issue where the trampoline chunks were being placed
at the end of the unpack ordering, causing overwrite during unpack.

Use suspending kernel's stack smash guard to avoid panicing during unpack.

ok deraadt@

Remove hibernate_get_next_rle function (unused, and we need to redo it
anyway as we move toward a streamed implmentation)

unbias the chunks and chunktable writing and reading. as a result, it
is now possible to move the chunktable right after the chunks, not at
the end of the swap.
ok mlarkin

shorten dev_t in hib_info

use DEV_BSIZE instead of "secsize"
ok mlarkin

teach the side-effect free drivers about the partition they are dealing
by passing a start/length in the HIB_INIT op. Then rebase all
hibernate-time block offsets to be relative to the start of that partition.
This simplifies things a lot.
ok mlarkin

Removed some unused code

Add a magic number to the head of the signature block. Check for magic
number match during signature block read during speculative unhibernate on
boot. If the magic number matches but we have otherwise chosen to not
unhibernate (due to kernel/memory mismatch), clear the signature block
early to avoid accidentally trying to unhibernate on subsequent boots. This
prevents accidental unhibernates and endless unhibernate/reboot cycles.

Add a define for HIBERNATE_DEBUG for various debugging printfs (disabled by
default).

Finally, change some KASSERTs to warning printfs (they probably shouldn't
have been KASSERTs in the first place).

"looks good" deraadt@

Add a hibernate HIB_DONE op. After the memory-side-effect driver finishes
IO to the disk, DVACT_RESUME the controller back to normal operation. That
allows us to do the full DVACT_POWERDOWN sequence afterwards.
ok mlarkin

Reorganize some hibernate functions for easier readability.
Fix some incorrect/old comments.

ok deraadt@

use regular i/o functions in the hibernate resume path. ahci(4) hibernate
i/o will not mix with regular i/o so it can only be used in the hibernate
path.

ok deraadt@ mlarkin@

Fix an integer math error when using the result of uvm_page_rle, and
at the same time increase said function's max RLE page count return value.

Add hooks in the right places to call the hibernate suspend and resume
routines, so that we can enable hibernation with a HIBERNATE option
line in GENERIC and appropriate acpi.c goo.

discussed on and off with deraadt@ over the past few months

ready this for big-endian support later on
ok mlarkin

Reduce use of globals in hibernate code.

discussed with deraadt@

Use a fixed io_page for all hibernate I/O, which is needed for
ahci_hibernate_io, a skeleton of which is also provided in this diff.

This code is from deraadt@. Tested on a few wd machines to ensure it works
there as well.

Add some #defines for the various hibernate I/O modes, some of the
groundwork for *_hibernate_io functions other than wd_hibernate_io

These changes were sent to me by deraadt@

In hibernate resume, free the piglet and other VAs we allocated during
suspend.

ok pirofti@

KNF of mlarkin's code, requested by him. Some improvements to the interface
for talking to the disk driver snuck in.
ok mlarkin

Perform most of the remaining refactoring of hibernate code into
MI/MD parts. This also introduces a chunk placement routine that was
originally developed at c2k11 with help from drahn and ariane.

There are still a few more things to do for hibernate, but those can be
worked on in-tree. This code is disabled by default, and not yet called.

ok deraadt@ (and deraadt@ said kettenis@ also ok'ed it :) )

Add hibernate_read_block and fix a couple of typos in the previous commit.

Call (temporarily, until we have RLE page encoding) uvm_pmr_zero_everything
on suspend to ensure we get good zlib compression.

Add MI signature block (hibernate_info) comparison routine

Extract hibernate_write_signature and hibernate_clear_signature to the MI
hibernate code, and add chunk range overlap checking.

Extract MI pmap function hibernate_enter_resume_mapping, refactor old i386
resume pmap code to match.

Add hibernate deflater and inflater and cache flush routines.

Code is not presently called or automatically built.

Add zlib reset, alloc, and free functions for hibernate image compression

Separate some MD and MI bits and a bit of refactoring to make subsequent
commits easier.

Work in progress, hibernate will still not work for you.

ok deraadt@

Ensure all pages in pmemrange can be marked as dirty.
It'd be a very bad idea to hand out dirty pages as zeroed, just because
we came back from hibernate.

No callers at the moment, will be called on hibernate resume path.

Put in RLE logic for hibernate compressor.

These have the potential to compress 1MB of physmem into 1 byte.
This works by noting the page is not in use and therefor skipping it.

Needed by mlarkin@ for hibernate. No callers yet.

Change pig allocator to a highest-address selection.
Add piglet allocator, which does a lowest-address selection.

No callers yet, needed by mlarkin@ for hibernate voodoo.

Move uvm_pmr_alloc_pig to kern/subr_hibernate.c

No callers, no functional change.

Move uvm_pmr_zero_everything() to subr_hibernate.

This function will probably die before ever being called
from the in-tree code, since hibernate will move to RLE encoding.

No functional change, function had no callers.

Move hiballoc to hibernate.h, subr_hibernate.c
Next few commits will move other hibernate-specific functionality,
like the pig-allocator, to subr_hibernate.

No functional change, no callers either.

calculate a "sum" based upon pointers to functions all over the kernel,
so that an unhibernate kernel can detect if it is running with the
kernel it booted.
ok mlarkin

move from RB macros to RBT functions

New framework that allows hibernate to pass in entropy from it's fresh
boot.
ok mlarkin

remove an unused function and some unused variables in hibernate

Rework piglet and pig allocation. Currently the piglet gets allocated
deep down in the suspend path, where it is really hard to recover from
allocation failure. So allocate the piglet early on in the suspend path.
Also change the piglet and piglet allocation functions to use km_alloc(9)
instead of doing pmemrange magic. This removes a bunch of code which, in the
case of the piglet allocation, is broken since it results in a NULL pointer
dereference. Also switch the piglet allocation to not wait. If we can't
allocate 16MB of phys contig memory on a halfway modern machine we're almost
certainly under a lot of memory pressure and we're better off not trying to
hibernate anyway.

ok mlarkin@

Support hibernating to softraid crypto volumes.

much help and ok from deraadt@

Reenable hibernate RLE support and flush+zero all memory after unpack.

Add a function to drop all clean pages on the page daemon queues and call
it when we hibernate.

ok mlarkin@, miod@, deraadt@

Flush the buffercache to 16MB on hibernate and restore its previous max
size (kern.bufcachepercent) on resume, for better hibernate performance.

ok beck@

Cleanup the chunk placement routine by removing the conflict resolver.
Chunks are now sorted by ascending PA and all chunks are bounced before
unpack. This fixes an issue where the trampoline chunks were being placed
at the end of the unpack ordering, causing overwrite during unpack.

Use suspending kernel's stack smash guard to avoid panicing during unpack.

ok deraadt@

Remove hibernate_get_next_rle function (unused, and we need to redo it
anyway as we move toward a streamed implmentation)

unbias the chunks and chunktable writing and reading. as a result, it
is now possible to move the chunktable right after the chunks, not at
the end of the swap.
ok mlarkin

shorten dev_t in hib_info

use DEV_BSIZE instead of "secsize"
ok mlarkin

teach the side-effect free drivers about the partition they are dealing
by passing a start/length in the HIB_INIT op. Then rebase all
hibernate-time block offsets to be relative to the start of that partition.
This simplifies things a lot.
ok mlarkin

Removed some unused code

Add a magic number to the head of the signature block. Check for magic
number match during signature block read during speculative unhibernate on
boot. If the magic number matches but we have otherwise chosen to not
unhibernate (due to kernel/memory mismatch), clear the signature block
early to avoid accidentally trying to unhibernate on subsequent boots. This
prevents accidental unhibernates and endless unhibernate/reboot cycles.

Add a define for HIBERNATE_DEBUG for various debugging printfs (disabled by
default).

Finally, change some KASSERTs to warning printfs (they probably shouldn't
have been KASSERTs in the first place).

"looks good" deraadt@

Add a hibernate HIB_DONE op. After the memory-side-effect driver finishes
IO to the disk, DVACT_RESUME the controller back to normal operation. That
allows us to do the full DVACT_POWERDOWN sequence afterwards.
ok mlarkin

Reorganize some hibernate functions for easier readability.
Fix some incorrect/old comments.

ok deraadt@

use regular i/o functions in the hibernate resume path. ahci(4) hibernate
i/o will not mix with regular i/o so it can only be used in the hibernate
path.

ok deraadt@ mlarkin@

Fix an integer math error when using the result of uvm_page_rle, and
at the same time increase said function's max RLE page count return value.

Add hooks in the right places to call the hibernate suspend and resume
routines, so that we can enable hibernation with a HIBERNATE option
line in GENERIC and appropriate acpi.c goo.

discussed on and off with deraadt@ over the past few months

ready this for big-endian support later on
ok mlarkin

Reduce use of globals in hibernate code.

discussed with deraadt@

Use a fixed io_page for all hibernate I/O, which is needed for
ahci_hibernate_io, a skeleton of which is also provided in this diff.

This code is from deraadt@. Tested on a few wd machines to ensure it works
there as well.

Add some #defines for the various hibernate I/O modes, some of the
groundwork for *_hibernate_io functions other than wd_hibernate_io

These changes were sent to me by deraadt@

In hibernate resume, free the piglet and other VAs we allocated during
suspend.

ok pirofti@

KNF of mlarkin's code, requested by him. Some improvements to the interface
for talking to the disk driver snuck in.
ok mlarkin

Perform most of the remaining refactoring of hibernate code into
MI/MD parts. This also introduces a chunk placement routine that was
originally developed at c2k11 with help from drahn and ariane.

There are still a few more things to do for hibernate, but those can be
worked on in-tree. This code is disabled by default, and not yet called.

ok deraadt@ (and deraadt@ said kettenis@ also ok'ed it :) )

Add hibernate_read_block and fix a couple of typos in the previous commit.

Call (temporarily, until we have RLE page encoding) uvm_pmr_zero_everything
on suspend to ensure we get good zlib compression.

Add MI signature block (hibernate_info) comparison routine

Extract hibernate_write_signature and hibernate_clear_signature to the MI
hibernate code, and add chunk range overlap checking.

Extract MI pmap function hibernate_enter_resume_mapping, refactor old i386
resume pmap code to match.

Add hibernate deflater and inflater and cache flush routines.

Code is not presently called or automatically built.

Add zlib reset, alloc, and free functions for hibernate image compression

Separate some MD and MI bits and a bit of refactoring to make subsequent
commits easier.

Work in progress, hibernate will still not work for you.

ok deraadt@

Ensure all pages in pmemrange can be marked as dirty.
It'd be a very bad idea to hand out dirty pages as zeroed, just because
we came back from hibernate.

No callers at the moment, will be called on hibernate resume path.

Put in RLE logic for hibernate compressor.

These have the potential to compress 1MB of physmem into 1 byte.
This works by noting the page is not in use and therefor skipping it.

Needed by mlarkin@ for hibernate. No callers yet.

Change pig allocator to a highest-address selection.
Add piglet allocator, which does a lowest-address selection.

No callers yet, needed by mlarkin@ for hibernate voodoo.

Move uvm_pmr_alloc_pig to kern/subr_hibernate.c

No callers, no functional change.

Move uvm_pmr_zero_everything() to subr_hibernate.

This function will probably die before ever being called
from the in-tree code, since hibernate will move to RLE encoding.

No functional change, function had no callers.

Move hiballoc to hibernate.h, subr_hibernate.c
Next few commits will move other hibernate-specific functionality,
like the pig-allocator, to subr_hibernate.

No functional change, no callers either.