kernel: Make use of the new ConditionVariable lock-switching APIs in a few places.

kernel/slab: Fix build with SLAB_ALLOCATION_TRACING enabled.

Unfortunately it does not work, I get this when invoking
allocations_per_caller at the kdebug prompt:

92513 9162096 0x0000000000000000 <NULL> + 0x0 (commpage) (nearest)
12602 2108408 0xffffffff80132a50 _Z23add_alloc_tracing_entryP11ObjectCachejPv + 0x70 (kernel_x86_64)

The stack tracing system must be broken, or maybe it has never been
properly adjusted for x86_64.

white space cleanup

Remove useless includes of khash.h

* These files were already converted to BOpenHashTable.
* For #9552.

Fixed formatting of output in some debugger commands.

Currently all debugger commands assume 32-bit pointers when formatting their
output. This means that on x86_64 the output is incorrectly formatted. Fixed
this by adding a B_PRINTF_POINTER_WIDTH definition (16 on 64-bit, 8 on
32-bit), and using this to correctly format the output. Not all commands have
been fixed yet, but all VM, slab, VFS, team, thread and image commands should
be correct.

More 64-bit compilation/safety fixes.

Allow replacing the object cache with the guarded heap.

This allows to use the debug features of the guarded heap also on
allocations made through the object cache API. This is obivously
horrible for performance and uses up huge amounts of memory, so the
initial and grow sizes are adjusted accordingly.

Note that this is a rather simple hack, using the object_cache pointer
to transport the allocation size. The alignment is neglected completely.

Tracking info wasn't always retrieved with the cache locked.

Getting the object slab does a hash lookup which needs to be protected
by the cache lock. Otherwise the hash table may be resized or otherwise
modified while we do the lookup, leading to errors.

Tiny pointer style cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43189 a95241bf-73f2-0310-859d-f6bbb57e9c96

bonefish+mmlr:
* Add "allocation_infos" KDL command. Can be used to print the allocation info,
including the stack trace, given an object cache, slab, allocation within a
slab or allocation address. It can also optionally be filtered by team/thread.
* Fix the AllocationDetailPrinterCallback to not access a possibly invalid
trace entry for printing a stack trace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43118 a95241bf-73f2-0310-859d-f6bbb57e9c96

mmlr + bonefish:
* AllocationTrackingCallback::ProcessTrackingInfo(): Also pass the
allocation pointer.
* "allocations_per_caller" KDL command: Add option "-d". When given,
each allocation for the specified caller is printed, including the
respective stack trace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43087 a95241bf-73f2-0310-859d-f6bbb57e9c96

mmlr + bonefish:
Refactor the "allocations_per_caller" KDL command related functions.
They expect an instance of a class implementing the new
AllocationTrackingCallback interface, now. The only implementation ATM
is AllocationCollectorCallback, which does the work the now removed
slab_debug_add_allocation_for_caller() did before.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43082 a95241bf-73f2-0310-859d-f6bbb57e9c96

mmlr + bonefish:
* Fix build broken in r43078. The slab_debug_add_allocation_for_caller()
wasn't guarded correctly.
* slab_debug_add_allocation_for_caller(): Add bool resetAllocationInfos
parameter, which makes the function clear the allocation tracking
infos after processing the data.
* "allocations_per_caller" KDL command: Add option "-r" to reset the
allocation tracking infos. The next invocation of the command will
only show the allocations made after the reset.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43079 a95241bf-73f2-0310-859d-f6bbb57e9c96

mmlr + bonefish:
* dump_allocations_per_caller(): Compute the total allocation count and
size from the caller infos instead of using return arguments in the
helper functions called.
* Move caller info update code from analyze_allocation_callers() to new
function slab_debug_add_allocation_for_caller(), so it can be reused.
* Add MemoryManager::AnalyzeAllocationCallers() to collect the
allocation information for the memory manager.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43078 a95241bf-73f2-0310-859d-f6bbb57e9c96

mmlr + bonefish:
* Add optional stack trace capturing for slab memory manager tracing.
* Add allocation tracking for the slab allocator (enabled via
SLAB_ALLOCATION_TRACKING). The allocation tracking requires tracing
with stack traces to be enabled for object caches and/or the memory
manager.
- Add class AllocationTrackingInfo that associates an allocation with
its respective tracing entry. The structure is added to the end of
an allocation done by the memory manager. For the object caches
there's a separate array for each slab.
- Add code range markers to the slab code, so that the first caller
into the slab code can be retrieved from the stack traces.
- Add KDL command "allocations_per_caller" that lists all allocations
summarized by caller.
* Move debug definitions from slab_private.h to slab_debug.h.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43072 a95241bf-73f2-0310-859d-f6bbb57e9c96

bonefish+mmlr:
* Introduce "paranoid" malloc/free into the slab allocator (initializing
allocated memory to 0xcc and setting freed memory to 0xdeadbeef).
* Allow for optional stack traces for slab object cache tracing.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43046 a95241bf-73f2-0310-859d-f6bbb57e9c96

Changed wrong help text


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@42172 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added ObjectCache::alignment, the object alignment and used the alignment for
incrementing the cache color cycle. Using the fixed value (8) would
potentially misalign the object again.
* Don't use CACHE_ALIGN_ON_SIZE for object caches any longer -- we have the
alignment parameter anyway (the flag is still used for the MemoryManager,
though).
* ObjectCache::InitSlab(): Slab coloring *was* done when CACHE_ALIGN_ON_SIZE
was given, i.e. exactly the wrong way around. Also the cache_color_cycle
computation was weird -- color 0 was used twice in a row.
* The "slabs" and "slab_cache" KDL commands also print the alignment, now.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37534 a95241bf-73f2-0310-859d-f6bbb57e9c96

object_cache_low_memory(): Reduce the cache pressure indicator even for the
"note" low resource state. Not doing so could result in a very high limit for
the number of empty slabs, which wouldn't change until reaching the "warning"
state.
Fixes #5816.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37171 a95241bf-73f2-0310-859d-f6bbb57e9c96

Replaced no longer necessary send_signal_etc() work-arounds for
resume_thread().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36531 a95241bf-73f2-0310-859d-f6bbb57e9c96

* The slab_cache command now also dumps the actual slab lists.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35737 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Quick&dirty fix of a race condition that caused an endless loop in
object_cache_alloc(): the ObjectCache::total_objects count was increased in
ObjectCache::InitSlab(), but the slab was really only added at a later point
between the cache could be unlocked.
* If a second object_cache_reserve_internal() managed to be called while the
lock was unlocked, it would see that there has to be space available, and
will then return -- however, since the other thread could not yet place the
slab into the cache, object_cache_alloc() cannot find it.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35702 a95241bf-73f2-0310-859d-f6bbb57e9c96

Changed object_depot_store() return value to void. It is now always takes
over ownership of the object. Fixes double free introduced in r35605.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35608 a95241bf-73f2-0310-859d-f6bbb57e9c96

Removed TODO that isn't one.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35606 a95241bf-73f2-0310-859d-f6bbb57e9c96

* The low resource handler now empties the cache depot's magazines; before,
they were never freed unless the cache was destroyed (I just wondered why
my system would bury >1G in the magazines).
* Made the magazine capacity variable per cache, ie. for larger objects, it's
not a good idea to have 64*CPU buffers lying around in the worst case.
* Furthermore, the create_object_cache_etc()/object_depot_init() now have
arguments for the magazine capacity as well as the maximum number of full
unused magazines.
* By default, you might want to initialize both to zero, as then some hopefully
usable defaults are computed. Otherwise (the only current example is the
vm_page_mapping cache) you can just put in the values you'd want there.
The page mapping cache uses larger values, as its objects are usually
allocated and deleted in larger chunks.
* Beware, though, I couldn't test these changes yet as Qemu didn't like to run
today. I'll test these changes on another machine now.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35601 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Fixed maintaining the object depot's free magazine count (it's not really used
though).
* Added/improved some KDL commands to make the slab easier to work with from
KDL.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35466 a95241bf-73f2-0310-859d-f6bbb57e9c96

Moved added the debugger commands from slab_init() to slab_init_post_area().
Those use malloc(), which obviously doesn't work before the heap is
initialized.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35288 a95241bf-73f2-0310-859d-f6bbb57e9c96

MemoryManager:
* Added support to do larger raw allocations (up to one large chunk (128 pages))
in the slab areas. For an even larger allocation an area is created (haven't
seen that happen yet, though).
* Added kernel tracing (SLAB_MEMORY_MANAGER_TRACING).
* _FreeArea(): Copy and paste bug: The meta chunks of the to be freed area
would be added to the free lists instead of being removed from them. This
would corrupt the lists and also lead to all kinds of misuse of meta chunks.

object caches:
* Implemented CACHE_ALIGN_ON_SIZE. It is no longer set for all small object
caches, but the block allocator sets it on all power of two size caches.
* object_cache_reserve_internal(): Detect recursion and don't wait in such a
case. The function could deadlock itself, since
HashedObjectCache::CreateSlab() does allocate memory, thus potentially
reentering.
* object_cache_low_memory():
- I missed some returns when reworking that one in r35254, so the function
might stop early and also leave the cache in maintenance mode, which would
cause it to be ignored by object cache resizer and low memory handler from
that point on.
- Since ReturnSlab() potentially unlocks, the conditions weren't quite correct
and too many slabs could be freed.
- Simplified things a bit.
* object_cache_alloc(): Since object_cache_reserve_internal() does potentially
unlock the cache, the situation might have changed and their might not be an
empty slab available, but a partial one. The function would crash.
* Renamed the object cache tracing variable to SLAB_OBJECT_CACHE_TRACING.
* Renamed debugger command "cache_info" to "slab_cache" to avoid confusion with
the VMCache commands.
* ObjectCache::usage was not maintained anymore since I introduced the
MemoryManager. object_cache_get_usage() would thus always return 0 and the
block cache would not be considered cached memory. This was only of
informational relevance, though.

slab allocator misc.:
* Disable the object depots of block allocator caches for object sizes > 2 KB.
Allocations of those sizes aren't so common that the object depots yield any
benefit.
* The slab allocator is now fully self-sufficient. It allocates its bootstrap
memory from the MemoryManager, and the hash tables for HashedObjectCaches use
the block allocator instead of the heap, now.
* Added option to use the slab allocator for malloc() and friends
(USE_SLAB_ALLOCATOR_FOR_MALLOC). Currently disabled. Works in principle and
has virtually no lock contention. Handling for low memory situations is yet
missing, though.
* Improved the output of some debugger commands.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35283 a95241bf-73f2-0310-859d-f6bbb57e9c96

MemoryManager:
* Does now keep one or two empty areas around, so that even in case of
CACHE_DONT_LOCK_KERNEL_SPACE memory can be provided as long as pages are
available. The object cache maintainer thread is used to asynchronously
allocate/delete the free areas.
* Added new debugger commands "slab_meta_chunk[s]" and improved the existing
ones.
* Moved Area::chunks to MetaChunk.
* Removed unused _AllocationArea() "chunkSize" parameter.
* Fixed serious bug in _FreeChunk(): Empty meta chunks were not removed from
the partial chunk lists and could thus be used twice.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35264 a95241bf-73f2-0310-859d-f6bbb57e9c96

low resource handler: Clear the maintenance_in_progress flag also when
adding the cache to the maintenance queue. Not so important but more correct.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35255 a95241bf-73f2-0310-859d-f6bbb57e9c96

Reworked the object cache resizer (renamed to object cache maintainer) and
low resource handler functions. Particularly fixed the race conditions
between those and delete_object_cache().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35254 a95241bf-73f2-0310-859d-f6bbb57e9c96

Replaced CACHE_DONT_SLEEP by two new flags CACHE_DONT_WAIT_FOR_MEMORY and
CACHE_DONT_LOCK_KERNEL_SPACE. If the former is given, the slab memory manager
does not wait when reserving memory or pages. The latter prevents area
operations. The new flags add a bit of flexibility. E.g. when allocating page
mapping objects for userland areas CACHE_DONT_WAIT_FOR_MEMORY is sufficient,
i.e. the allocation will succeed as long as pages are available.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35246 a95241bf-73f2-0310-859d-f6bbb57e9c96

slab allocator:
* Implemented a more elaborated raw memory allocation backend (MemoryManager).
We allocate 8 MB areas whose pages we allocate and map when needed. An area is
divided into equally-sized chunks which form the basic units of allocation. We
have areas with three possible chunk sizes (small, medium, large), which is
basically what the ObjectCache implementations were using anyway.
* Added "uint32 flags" parameter to several of the slab allocator's object
cache and object depot functions. E.g. object_depot_store() potentially wants
to allocate memory for a magazine. But also in pure freeing functions it
might eventually become useful to have those flags, since they could end up
deleting an area, which might not be allowable in all situations. We should
introduce specific flags to indicate that.
* Reworked the block allocator. Since the MemoryManager allocates block-aligned
areas, maintains a hash table for lookup, and maps chunks to object caches,
we can quickly find out which object cache a to be freed allocation belongs
to and thus don't need the boundary tags anymore.
* Reworked the slab boot strap process. We allocate from the initial area only
when really necessary, i.e. when the object cache for the respective
allocation size has not been created yet. A single page is thus sufficient.

other:
* vm_allocate_early(): Added boolean "blockAlign" parameter. If true, the
semantics is the same as for B_ANY_KERNEL_BLOCK_ADDRESS.
* Use an object cache for page mappings. This significantly reduces the
contention on the heap bin locks.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35232 a95241bf-73f2-0310-859d-f6bbb57e9c96

Always unlock the object cache while allocating memory. This is necessary for
the CACHE_DONT_SLEEP flag to work for real, since otherwise the thread could
block on the mutex held by a thread allocating memory. We use two condition
variables to prevent multiple threads from allocating slabs at the same time.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35206 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Reverted r35203, i.e. ObjectCache::object_per_slab is gone again.
* Changed the semantics of object_cache_reserve_internal(). Now it makes sure
the given number of objects are free. As a side effect this also changes
the semantics of object_cache_reserve() similarly, though I have trouble
seeing the purpose of the function in the first place.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35204 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added ObjectCache::objects_per_slab, which allowed to squash the TODO in
object_cache_reserve_internal().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35203 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Consequently propagate the CACHE_DONT_SLEEP flag.
* block_alloc(): Create B_FULL_LOCK area.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35202 a95241bf-73f2-0310-859d-f6bbb57e9c96

Shuffled functions a bit, so that they are in a reasonable order.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35198 a95241bf-73f2-0310-859d-f6bbb57e9c96

Removed debug left-over.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35197 a95241bf-73f2-0310-859d-f6bbb57e9c96

* object_cache_return_object_wrapper(): Calling object_cache_free() is a bad
idea, since that would potentially add the object back to the object store
or lead to infinite recursion. When the object cache is destroyed it most
likely led to infinite loops, because the object would alternately be
removed from and added back to the object store.
* delete_object_cache(): Lock after destroying the object store, so we don't
deadlock.
* Use the object store on SMP machines. It seems to work, though I only
tested with the network stack and that seems to have problems of its own.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35182 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Split the slab allocator code into separate source files and C++-ified
things a bit.
* Some style cleanup.
* The object depot does now have a cookie that will be passed to the return
hook.
* Fixed object_cache_return_object_wrapper() using the new cookie.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35174 a95241bf-73f2-0310-859d-f6bbb57e9c96

Moved the definition of the CACHE_ALIGN_ON_SIZE flag to the header and fixed
its value.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35163 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Renamed headers/private/kernel/slab/Depot.h to ObjectDepot.h.
* Moved the object depot code to its own source file.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35161 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved the VM headers into subdirectory vm/.
* Renamed vm_cache.h/vm_address_space.h to VMCache.h/VMAddressSpace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34449 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved VMAddressSpace definition to vm_address_space.h.
* "Classified" VMAddressSpace, i.e. turned the vm_address_space_*() functions
into methods, made all attributes (but "areas") private, and added
accessors.
* Also turned the vm.cpp functions vm_area_lookup() and
remove_area_from_address_space() into VMAddressSpace methods. The rest of
the area management functionality will follow soon.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34447 a95241bf-73f2-0310-859d-f6bbb57e9c96

* The slab and the block cache now both also react on address space shortages.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34310 a95241bf-73f2-0310-859d-f6bbb57e9c96

We already panic to indicate that something's wrong, no real need to crash
afterwards instead of returning gracefully.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@32293 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Renamed the ROUNDOWN macro to ROUNDDOWN. Also changed the implementation of
ROUNDUP to use '*' and '/' -- the compiler will optimize that for powers of
two anyway and this implementation works for other numbers as well.
* The thread::fault_handler use in C[++] code was broken with gcc 4. At least
when other functions were invoked. Trying to trick the compiler wasn't a
particularly good idea anyway, since the next compiler version could break
the trick again. So the general policy is to use the fault handlers only in
assembly code where we have full control. Changed that for x86 (save for the
vm86 mode, which has a similar mechanism), but not for the other
architectures.
* Introduced fault_handler, fault_handler_stack_pointer, and fault_jump_buffer
fields in the cpu_ent structure, which must be used instead of
thread::fault_handler in the kernel debugger. Consequently user_memcpy() must
not be used in the kernel debugger either. Introduced a debug_memcpy()
instead.
* Introduced debug_call_with_fault_handler() function which calls a function
in a setjmp() and fault handler context. The architecture specific backend
arch_debug_call_with_fault_handler() has only been implemented for x86 yet.
* Introduced debug_is_kernel_memory_accessible() for use in the kernel
debugger. It determines whether a range of memory can be accessed in the
way specified. The architecture specific back end
arch_vm_translation_map_is_kernel_page_accessible() has only been implemented
for x86 yet.
* Added arch_debug_unset_current_thread() (only implemented for x86) to unset
the current thread pointer in the kernel debugger. When entering the kernel
debugger we do some basic sanity checks of the currently set thread structure
and unset it, if they fail. This allows certain commands (most importantly
the stack trace command) to avoid accessing the thread structure.
* x86: When handling a double fault, we do now install a special handler for
page faults. This allows us to gracefully catch faulting commands, even if
e.g. the thread structure is toast.

We are now in much better shape to deal with double faults. Hopefully avoiding
the triple faults that some people have been experiencing on their hardware
and ideally even allowing to use the kernel debugger normally.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@32073 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added explicit physical address parameter to vm_create_anonymous_area() and
create_area_etc(). 0 for the default behavior.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31941 a95241bf-73f2-0310-859d-f6bbb57e9c96

headers/private/kernel/util/OpenHashTable.h, Hugo's version, is a bit nicer than
Tracker's OpenHashTable.h which it should eventually replace. We've renamed the
class to BOpenHashTable and changed the interface slightly so that HashTableLink
became superfluous.
Adapted all the code that used it. Since the OpenHashTables no longer clash,
this should fix the GCC4 build.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31791 a95241bf-73f2-0310-859d-f6bbb57e9c96

* various fixes identified by gcc4, provided by Joe Prostko


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@30902 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixing warnings under GCC4 in preparation to enable -Werror there as well:
* Replaced the use of offsetof() for structs that aren't PODs. Add a
offset_of_member() macro to util/khash.h because that's what it's used for
in our cases.
* Change the signature of add_debugger_command()/remove_debugger_command() on
GCC > 2 to avoid the depricated conversion from string constants to char *.
* Adding some "suggested" parenthesis. I know that not everyone likes that, but
it pointed out at least one bug that is fixed here as well.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@29113 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Implement CACHE_DONT_SLEEP partially. At least for small object caches
without object depot it should be complete. For the other stuff
internal_alloc() would need to pass the flag on to block_alloc(), but
that isn't possible yet.
* Adjusted the low memory handler to respect the minimum object reserve
of the object caches.

The swap_test_heap test does seem to pass reliably with 128 MB RAM and
128 MB allocation, now. It's quite slow, though. Particularly while the
allocation is filled, the system is completely unusable.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27118 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added object_cache_set_minimum_reserve() which sets the minimal number
of free objects an object cache should try to have ready. If the number
of free objects drops below the threshold, a new urgent priority thread
is asked to asynchronously resize the object cache (pretty similar to
the heap grower thread). Such a mechanism is necessary for code paths
that are supposed to free pages, but may need memory themselves (like
the swap support).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27100 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added object_cache_get_usage() function that returns the memory
allocated by an object cache.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26835 a95241bf-73f2-0310-859d-f6bbb57e9c96

Merged branch haiku/branches/developer/bonefish/vm into trunk. This
introduces the following relevant changes:
* VMCache:
- Renamed vm_cache to VMCache, merged it with vm_store and made it a
C++ class with virtual methods (replacing the store operations).
Turned the different store implementations into subclasses.
- Introduced MergeStore() callback, changed semantics of Commit().
- Changed locking and referencing semantics. A reference can only be
acquired/released with the cache locked. An unreferenced cache is
deleted and a mergeable cache merged when it is unlocked. This
removes the "busy" state of a cache and simplifies the page fault
code.
* Added VMAnonymousCache, which will implement swap support (work by
Zhao Shuai). It is not integrated and used yet, though.
* Enabled the mutex/recursive lock holder asserts.
* Fixed DoublyLinkedList::Swap().
* Generalized the low memory handler to a low resource handler. And made
semaphores and reserved memory handled resources. Made
vm_try_resource_memory() optionally wait (with timeout), and used that
feature to reserve memory for areas.
...


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26572 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Initialize all static mutexes in the kernel through a MUTEX_INITIALIZER()
and remove the then unneeded mutex_init() for them.
* Remove the workaround for allowing uninitialized mutexes on kernel startup.
As they are all initialized statically through the MUTEX_INITIALIZER() now
this is not needed anymore.
* An uninitialized mutex will now cause a panic when used to find possibly
remaining cases.
* Remove now unnecessary driver_settings_init_post_sem() function.

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25812 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Changed recursive_lock to use a mutex instead of a semaphore.
* Adjusted code using recursive locks respectively. The initialization
cannot fail anymore, and it is possible to use recursive locks in the
early boot process (even uninitialized, if in BSS), which simplifies
things a little.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25687 a95241bf-73f2-0310-859d-f6bbb57e9c96

Now uses mutexes instead of benaphores, that also simplified object
instantiation at early boot a bit.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25380 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Paranoia checks were accidentally globally enabled by default. Object
* cache paranoia was always enabled.
* Changed from paranoia on/off to levels. Adjusted the macros to take a
level argument.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25215 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Changed macros that enable tracing for individual components from
defined/undefined to numeric values (0 for undefined). This allows for
trace levels.
* Set SYSCALL_TRACING_IGNORE_KTRACE_OUTPUT default to 1, since this is
what one usually wants.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25213 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added paranoia checks. They reveal that a net buffer data node is freed
twice when running the OpenSSH "forwarding" test, which corrupts the
slab's object list. It's not quite clear to me yet why that happens.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25208 a95241bf-73f2-0310-859d-f6bbb57e9c96

Removed some leftover junk.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25197 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added object cache kernel tracing.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25196 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Also replaced the cpu store's lock with a recursive lock in order
to fix bug #1731.
* However, it turns out that depot destruction obviously doesn't work
correctly, at least we keep partial or full slabs around when we're
using them (which causes the code to panic).
* Therefore, I've now disabled depots completely, until I find the time
to really work on that code.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@23825 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Temporarily switched to a recursive lock for the depot.
* While this is not a really good idea for a lock with supposedly little
contention, but it'll fix bug #1731. I haven't tested it yet, but will
do so in a minute :-)
* I will need to rework the slab anyway so that it's possible to use it
as a replacement for our heap, and then I'll switch back to a benaphore
again.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@23822 a95241bf-73f2-0310-859d-f6bbb57e9c96

We generally handle NULL pointers in free() and friends gracefully.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@23578 a95241bf-73f2-0310-859d-f6bbb57e9c96

Print the missing object.

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@23574 a95241bf-73f2-0310-859d-f6bbb57e9c96

* vm_create_anonymous_area() now accepts B_ANY_KERNEL_BLOCK_ADDRESS.
* As a temporary work-around for the current slab allocator's area usage,
I added the CACHE_LARGE_SLAB flag, which will force the allocator to
use larger areas.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@23564 a95241bf-73f2-0310-859d-f6bbb57e9c96

The slab areas shouldn't be user readable...


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@23231 a95241bf-73f2-0310-859d-f6bbb57e9c96

Slab.h no longer includes Depot.h.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22647 a95241bf-73f2-0310-859d-f6bbb57e9c96

* register_low_memory_handler() was called too early by several kernel
components - now, we divide the initialization of that service into two parts
which allows this (before, most handlers were gone after boot).
* Added debugger command that dumps the low memory handlers.
* The slab allocator now registers its low memory handler with a higher
priority, so that it'll run before the potentially heavier ones.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22374 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved some definitions from vm_types.h that functions in vm.h need as
arguments into vm.h.
* This should fix the broken build from earlier - thanks Stefano for the note!
* That also allowed to clean some other includes a bit.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22331 a95241bf-73f2-0310-859d-f6bbb57e9c96

* vm.h no longer includes vm_types.h - only those that actually need access
to the private VM types are including vm_types.h now.
* Removed vm_page, vm_area, vm_cache, and vm_address_space typedefs; it's
cleaner this way, and the actual types are only used in C++ files now,
anyway.
* And that caused changes in many files...
* Made commpage.h self-containing.
* Minor cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22329 a95241bf-73f2-0310-859d-f6bbb57e9c96

axeld+bonefish: Got rid of the ParentType in the HashTableDefinition; it doesn't really
belong there.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@21766 a95241bf-73f2-0310-859d-f6bbb57e9c96

slab/cache: a couple more fixes.

- on CACHE_DURING_BOOT init benaphore count with 1.
- account for allocated space on early_allocate_pages.
- fixed slab position calculation in small slabs.
- we can now init all allocator sizes early rather than later.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20914 a95241bf-73f2-0310-859d-f6bbb57e9c96

cache/slab: when CACHE_DURING_BOOT is specified, use vm_allocate_early to obtain pages for slabs, and create the associated areas on post bootup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20913 a95241bf-73f2-0310-859d-f6bbb57e9c96

added object_cache_reserve to allow cache users to pre-allocate slabs.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20912 a95241bf-73f2-0310-859d-f6bbb57e9c96

added CACHE_UNLOCKED_PAGES flag to allow caches' pages to be created with no locking instead of full locking.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20911 a95241bf-73f2-0310-859d-f6bbb57e9c96

cache allocator: for allocations > 8k create areas, and delete them on free.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20902 a95241bf-73f2-0310-859d-f6bbb57e9c96

minimal cleanups


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20901 a95241bf-73f2-0310-859d-f6bbb57e9c96

don't need TRACE_SLAB by default now.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20898 a95241bf-73f2-0310-859d-f6bbb57e9c96

pushed the slab init a bit deeper. added a object cache based allocator, including a bootstrap mechanism to have it init during bootup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20896 a95241bf-73f2-0310-859d-f6bbb57e9c96

some object cache / slab improvements.

- call the reclaimer callback when low on memory.
- use the depot when on multi-cpu setups (for scalability).
- fixed the amount of memory spent on slabs for very large objects.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20889 a95241bf-73f2-0310-859d-f6bbb57e9c96

rewrote the object cache (slab) implementation a bit, preparing for further integration.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20887 a95241bf-73f2-0310-859d-f6bbb57e9c96

some slab cleanups, split the C interface to another file.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20875 a95241bf-73f2-0310-859d-f6bbb57e9c96

ups, TRACE_SLAB was on.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20852 a95241bf-73f2-0310-859d-f6bbb57e9c96

removed some of the slab's initial heavy debugging. we now merge the links into the slab itself resulting in zero overhead per buffer with MergedLink strategy.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20841 a95241bf-73f2-0310-859d-f6bbb57e9c96

slab: be smarter when picking the underlying strategy for the slab C API (based on requested object size).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20840 a95241bf-73f2-0310-859d-f6bbb57e9c96

more slab fixes, also introduced a new strategy optimized for medium sized buffers (64 >= x < 512) with lengths other than power of 2 (has an overhead of 2 words per buffer).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20839 a95241bf-73f2-0310-859d-f6bbb57e9c96

added locking to slab's Cache<>. Now we react to system's low memory conditions freeing up empty slabs.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20836 a95241bf-73f2-0310-859d-f6bbb57e9c96

assorted slab fixes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20833 a95241bf-73f2-0310-859d-f6bbb57e9c96

added initial slab code to the kernel. It is still unused, and there is still no VM interaction.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20832 a95241bf-73f2-0310-859d-f6bbb57e9c96

Remove useless includes of khash.h

* These files were already converted to BOpenHashTable.
* For #9552.

Fixed formatting of output in some debugger commands.

Currently all debugger commands assume 32-bit pointers when formatting their
output. This means that on x86_64 the output is incorrectly formatted. Fixed
this by adding a B_PRINTF_POINTER_WIDTH definition (16 on 64-bit, 8 on
32-bit), and using this to correctly format the output. Not all commands have
been fixed yet, but all VM, slab, VFS, team, thread and image commands should
be correct.

More 64-bit compilation/safety fixes.

Allow replacing the object cache with the guarded heap.

This allows to use the debug features of the guarded heap also on
allocations made through the object cache API. This is obivously
horrible for performance and uses up huge amounts of memory, so the
initial and grow sizes are adjusted accordingly.

Note that this is a rather simple hack, using the object_cache pointer
to transport the allocation size. The alignment is neglected completely.

Tracking info wasn't always retrieved with the cache locked.

Getting the object slab does a hash lookup which needs to be protected
by the cache lock. Otherwise the hash table may be resized or otherwise
modified while we do the lookup, leading to errors.

Tiny pointer style cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43189 a95241bf-73f2-0310-859d-f6bbb57e9c96

bonefish+mmlr:
* Add "allocation_infos" KDL command. Can be used to print the allocation info,
including the stack trace, given an object cache, slab, allocation within a
slab or allocation address. It can also optionally be filtered by team/thread.
* Fix the AllocationDetailPrinterCallback to not access a possibly invalid
trace entry for printing a stack trace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43118 a95241bf-73f2-0310-859d-f6bbb57e9c96

mmlr + bonefish:
* AllocationTrackingCallback::ProcessTrackingInfo(): Also pass the
allocation pointer.
* "allocations_per_caller" KDL command: Add option "-d". When given,
each allocation for the specified caller is printed, including the
respective stack trace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43087 a95241bf-73f2-0310-859d-f6bbb57e9c96

mmlr + bonefish:
Refactor the "allocations_per_caller" KDL command related functions.
They expect an instance of a class implementing the new
AllocationTrackingCallback interface, now. The only implementation ATM
is AllocationCollectorCallback, which does the work the now removed
slab_debug_add_allocation_for_caller() did before.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43082 a95241bf-73f2-0310-859d-f6bbb57e9c96

mmlr + bonefish:
* Fix build broken in r43078. The slab_debug_add_allocation_for_caller()
wasn't guarded correctly.
* slab_debug_add_allocation_for_caller(): Add bool resetAllocationInfos
parameter, which makes the function clear the allocation tracking
infos after processing the data.
* "allocations_per_caller" KDL command: Add option "-r" to reset the
allocation tracking infos. The next invocation of the command will
only show the allocations made after the reset.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43079 a95241bf-73f2-0310-859d-f6bbb57e9c96

mmlr + bonefish:
* dump_allocations_per_caller(): Compute the total allocation count and
size from the caller infos instead of using return arguments in the
helper functions called.
* Move caller info update code from analyze_allocation_callers() to new
function slab_debug_add_allocation_for_caller(), so it can be reused.
* Add MemoryManager::AnalyzeAllocationCallers() to collect the
allocation information for the memory manager.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43078 a95241bf-73f2-0310-859d-f6bbb57e9c96

mmlr + bonefish:
* Add optional stack trace capturing for slab memory manager tracing.
* Add allocation tracking for the slab allocator (enabled via
SLAB_ALLOCATION_TRACKING). The allocation tracking requires tracing
with stack traces to be enabled for object caches and/or the memory
manager.
- Add class AllocationTrackingInfo that associates an allocation with
its respective tracing entry. The structure is added to the end of
an allocation done by the memory manager. For the object caches
there's a separate array for each slab.
- Add code range markers to the slab code, so that the first caller
into the slab code can be retrieved from the stack traces.
- Add KDL command "allocations_per_caller" that lists all allocations
summarized by caller.
* Move debug definitions from slab_private.h to slab_debug.h.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43072 a95241bf-73f2-0310-859d-f6bbb57e9c96

bonefish+mmlr:
* Introduce "paranoid" malloc/free into the slab allocator (initializing
allocated memory to 0xcc and setting freed memory to 0xdeadbeef).
* Allow for optional stack traces for slab object cache tracing.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43046 a95241bf-73f2-0310-859d-f6bbb57e9c96

Changed wrong help text


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@42172 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added ObjectCache::alignment, the object alignment and used the alignment for
incrementing the cache color cycle. Using the fixed value (8) would
potentially misalign the object again.
* Don't use CACHE_ALIGN_ON_SIZE for object caches any longer -- we have the
alignment parameter anyway (the flag is still used for the MemoryManager,
though).
* ObjectCache::InitSlab(): Slab coloring *was* done when CACHE_ALIGN_ON_SIZE
was given, i.e. exactly the wrong way around. Also the cache_color_cycle
computation was weird -- color 0 was used twice in a row.
* The "slabs" and "slab_cache" KDL commands also print the alignment, now.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37534 a95241bf-73f2-0310-859d-f6bbb57e9c96

object_cache_low_memory(): Reduce the cache pressure indicator even for the
"note" low resource state. Not doing so could result in a very high limit for
the number of empty slabs, which wouldn't change until reaching the "warning"
state.
Fixes #5816.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37171 a95241bf-73f2-0310-859d-f6bbb57e9c96

Replaced no longer necessary send_signal_etc() work-arounds for
resume_thread().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36531 a95241bf-73f2-0310-859d-f6bbb57e9c96

* The slab_cache command now also dumps the actual slab lists.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35737 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Quick&dirty fix of a race condition that caused an endless loop in
object_cache_alloc(): the ObjectCache::total_objects count was increased in
ObjectCache::InitSlab(), but the slab was really only added at a later point
between the cache could be unlocked.
* If a second object_cache_reserve_internal() managed to be called while the
lock was unlocked, it would see that there has to be space available, and
will then return -- however, since the other thread could not yet place the
slab into the cache, object_cache_alloc() cannot find it.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35702 a95241bf-73f2-0310-859d-f6bbb57e9c96

Changed object_depot_store() return value to void. It is now always takes
over ownership of the object. Fixes double free introduced in r35605.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35608 a95241bf-73f2-0310-859d-f6bbb57e9c96

Removed TODO that isn't one.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35606 a95241bf-73f2-0310-859d-f6bbb57e9c96

* The low resource handler now empties the cache depot's magazines; before,
they were never freed unless the cache was destroyed (I just wondered why
my system would bury >1G in the magazines).
* Made the magazine capacity variable per cache, ie. for larger objects, it's
not a good idea to have 64*CPU buffers lying around in the worst case.
* Furthermore, the create_object_cache_etc()/object_depot_init() now have
arguments for the magazine capacity as well as the maximum number of full
unused magazines.
* By default, you might want to initialize both to zero, as then some hopefully
usable defaults are computed. Otherwise (the only current example is the
vm_page_mapping cache) you can just put in the values you'd want there.
The page mapping cache uses larger values, as its objects are usually
allocated and deleted in larger chunks.
* Beware, though, I couldn't test these changes yet as Qemu didn't like to run
today. I'll test these changes on another machine now.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35601 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Fixed maintaining the object depot's free magazine count (it's not really used
though).
* Added/improved some KDL commands to make the slab easier to work with from
KDL.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35466 a95241bf-73f2-0310-859d-f6bbb57e9c96

Moved added the debugger commands from slab_init() to slab_init_post_area().
Those use malloc(), which obviously doesn't work before the heap is
initialized.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35288 a95241bf-73f2-0310-859d-f6bbb57e9c96

MemoryManager:
* Added support to do larger raw allocations (up to one large chunk (128 pages))
in the slab areas. For an even larger allocation an area is created (haven't
seen that happen yet, though).
* Added kernel tracing (SLAB_MEMORY_MANAGER_TRACING).
* _FreeArea(): Copy and paste bug: The meta chunks of the to be freed area
would be added to the free lists instead of being removed from them. This
would corrupt the lists and also lead to all kinds of misuse of meta chunks.

object caches:
* Implemented CACHE_ALIGN_ON_SIZE. It is no longer set for all small object
caches, but the block allocator sets it on all power of two size caches.
* object_cache_reserve_internal(): Detect recursion and don't wait in such a
case. The function could deadlock itself, since
HashedObjectCache::CreateSlab() does allocate memory, thus potentially
reentering.
* object_cache_low_memory():
- I missed some returns when reworking that one in r35254, so the function
might stop early and also leave the cache in maintenance mode, which would
cause it to be ignored by object cache resizer and low memory handler from
that point on.
- Since ReturnSlab() potentially unlocks, the conditions weren't quite correct
and too many slabs could be freed.
- Simplified things a bit.
* object_cache_alloc(): Since object_cache_reserve_internal() does potentially
unlock the cache, the situation might have changed and their might not be an
empty slab available, but a partial one. The function would crash.
* Renamed the object cache tracing variable to SLAB_OBJECT_CACHE_TRACING.
* Renamed debugger command "cache_info" to "slab_cache" to avoid confusion with
the VMCache commands.
* ObjectCache::usage was not maintained anymore since I introduced the
MemoryManager. object_cache_get_usage() would thus always return 0 and the
block cache would not be considered cached memory. This was only of
informational relevance, though.

slab allocator misc.:
* Disable the object depots of block allocator caches for object sizes > 2 KB.
Allocations of those sizes aren't so common that the object depots yield any
benefit.
* The slab allocator is now fully self-sufficient. It allocates its bootstrap
memory from the MemoryManager, and the hash tables for HashedObjectCaches use
the block allocator instead of the heap, now.
* Added option to use the slab allocator for malloc() and friends
(USE_SLAB_ALLOCATOR_FOR_MALLOC). Currently disabled. Works in principle and
has virtually no lock contention. Handling for low memory situations is yet
missing, though.
* Improved the output of some debugger commands.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35283 a95241bf-73f2-0310-859d-f6bbb57e9c96

MemoryManager:
* Does now keep one or two empty areas around, so that even in case of
CACHE_DONT_LOCK_KERNEL_SPACE memory can be provided as long as pages are
available. The object cache maintainer thread is used to asynchronously
allocate/delete the free areas.
* Added new debugger commands "slab_meta_chunk[s]" and improved the existing
ones.
* Moved Area::chunks to MetaChunk.
* Removed unused _AllocationArea() "chunkSize" parameter.
* Fixed serious bug in _FreeChunk(): Empty meta chunks were not removed from
the partial chunk lists and could thus be used twice.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35264 a95241bf-73f2-0310-859d-f6bbb57e9c96

low resource handler: Clear the maintenance_in_progress flag also when
adding the cache to the maintenance queue. Not so important but more correct.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35255 a95241bf-73f2-0310-859d-f6bbb57e9c96

Reworked the object cache resizer (renamed to object cache maintainer) and
low resource handler functions. Particularly fixed the race conditions
between those and delete_object_cache().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35254 a95241bf-73f2-0310-859d-f6bbb57e9c96

Replaced CACHE_DONT_SLEEP by two new flags CACHE_DONT_WAIT_FOR_MEMORY and
CACHE_DONT_LOCK_KERNEL_SPACE. If the former is given, the slab memory manager
does not wait when reserving memory or pages. The latter prevents area
operations. The new flags add a bit of flexibility. E.g. when allocating page
mapping objects for userland areas CACHE_DONT_WAIT_FOR_MEMORY is sufficient,
i.e. the allocation will succeed as long as pages are available.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35246 a95241bf-73f2-0310-859d-f6bbb57e9c96

slab allocator:
* Implemented a more elaborated raw memory allocation backend (MemoryManager).
We allocate 8 MB areas whose pages we allocate and map when needed. An area is
divided into equally-sized chunks which form the basic units of allocation. We
have areas with three possible chunk sizes (small, medium, large), which is
basically what the ObjectCache implementations were using anyway.
* Added "uint32 flags" parameter to several of the slab allocator's object
cache and object depot functions. E.g. object_depot_store() potentially wants
to allocate memory for a magazine. But also in pure freeing functions it
might eventually become useful to have those flags, since they could end up
deleting an area, which might not be allowable in all situations. We should
introduce specific flags to indicate that.
* Reworked the block allocator. Since the MemoryManager allocates block-aligned
areas, maintains a hash table for lookup, and maps chunks to object caches,
we can quickly find out which object cache a to be freed allocation belongs
to and thus don't need the boundary tags anymore.
* Reworked the slab boot strap process. We allocate from the initial area only
when really necessary, i.e. when the object cache for the respective
allocation size has not been created yet. A single page is thus sufficient.

other:
* vm_allocate_early(): Added boolean "blockAlign" parameter. If true, the
semantics is the same as for B_ANY_KERNEL_BLOCK_ADDRESS.
* Use an object cache for page mappings. This significantly reduces the
contention on the heap bin locks.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35232 a95241bf-73f2-0310-859d-f6bbb57e9c96

Always unlock the object cache while allocating memory. This is necessary for
the CACHE_DONT_SLEEP flag to work for real, since otherwise the thread could
block on the mutex held by a thread allocating memory. We use two condition
variables to prevent multiple threads from allocating slabs at the same time.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35206 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Reverted r35203, i.e. ObjectCache::object_per_slab is gone again.
* Changed the semantics of object_cache_reserve_internal(). Now it makes sure
the given number of objects are free. As a side effect this also changes
the semantics of object_cache_reserve() similarly, though I have trouble
seeing the purpose of the function in the first place.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35204 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added ObjectCache::objects_per_slab, which allowed to squash the TODO in
object_cache_reserve_internal().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35203 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Consequently propagate the CACHE_DONT_SLEEP flag.
* block_alloc(): Create B_FULL_LOCK area.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35202 a95241bf-73f2-0310-859d-f6bbb57e9c96

Shuffled functions a bit, so that they are in a reasonable order.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35198 a95241bf-73f2-0310-859d-f6bbb57e9c96

Removed debug left-over.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35197 a95241bf-73f2-0310-859d-f6bbb57e9c96

* object_cache_return_object_wrapper(): Calling object_cache_free() is a bad
idea, since that would potentially add the object back to the object store
or lead to infinite recursion. When the object cache is destroyed it most
likely led to infinite loops, because the object would alternately be
removed from and added back to the object store.
* delete_object_cache(): Lock after destroying the object store, so we don't
deadlock.
* Use the object store on SMP machines. It seems to work, though I only
tested with the network stack and that seems to have problems of its own.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35182 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Split the slab allocator code into separate source files and C++-ified
things a bit.
* Some style cleanup.
* The object depot does now have a cookie that will be passed to the return
hook.
* Fixed object_cache_return_object_wrapper() using the new cookie.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35174 a95241bf-73f2-0310-859d-f6bbb57e9c96

Moved the definition of the CACHE_ALIGN_ON_SIZE flag to the header and fixed
its value.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35163 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Renamed headers/private/kernel/slab/Depot.h to ObjectDepot.h.
* Moved the object depot code to its own source file.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35161 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved the VM headers into subdirectory vm/.
* Renamed vm_cache.h/vm_address_space.h to VMCache.h/VMAddressSpace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34449 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved VMAddressSpace definition to vm_address_space.h.
* "Classified" VMAddressSpace, i.e. turned the vm_address_space_*() functions
into methods, made all attributes (but "areas") private, and added
accessors.
* Also turned the vm.cpp functions vm_area_lookup() and
remove_area_from_address_space() into VMAddressSpace methods. The rest of
the area management functionality will follow soon.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34447 a95241bf-73f2-0310-859d-f6bbb57e9c96

* The slab and the block cache now both also react on address space shortages.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34310 a95241bf-73f2-0310-859d-f6bbb57e9c96

We already panic to indicate that something's wrong, no real need to crash
afterwards instead of returning gracefully.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@32293 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Renamed the ROUNDOWN macro to ROUNDDOWN. Also changed the implementation of
ROUNDUP to use '*' and '/' -- the compiler will optimize that for powers of
two anyway and this implementation works for other numbers as well.
* The thread::fault_handler use in C[++] code was broken with gcc 4. At least
when other functions were invoked. Trying to trick the compiler wasn't a
particularly good idea anyway, since the next compiler version could break
the trick again. So the general policy is to use the fault handlers only in
assembly code where we have full control. Changed that for x86 (save for the
vm86 mode, which has a similar mechanism), but not for the other
architectures.
* Introduced fault_handler, fault_handler_stack_pointer, and fault_jump_buffer
fields in the cpu_ent structure, which must be used instead of
thread::fault_handler in the kernel debugger. Consequently user_memcpy() must
not be used in the kernel debugger either. Introduced a debug_memcpy()
instead.
* Introduced debug_call_with_fault_handler() function which calls a function
in a setjmp() and fault handler context. The architecture specific backend
arch_debug_call_with_fault_handler() has only been implemented for x86 yet.
* Introduced debug_is_kernel_memory_accessible() for use in the kernel
debugger. It determines whether a range of memory can be accessed in the
way specified. The architecture specific back end
arch_vm_translation_map_is_kernel_page_accessible() has only been implemented
for x86 yet.
* Added arch_debug_unset_current_thread() (only implemented for x86) to unset
the current thread pointer in the kernel debugger. When entering the kernel
debugger we do some basic sanity checks of the currently set thread structure
and unset it, if they fail. This allows certain commands (most importantly
the stack trace command) to avoid accessing the thread structure.
* x86: When handling a double fault, we do now install a special handler for
page faults. This allows us to gracefully catch faulting commands, even if
e.g. the thread structure is toast.

We are now in much better shape to deal with double faults. Hopefully avoiding
the triple faults that some people have been experiencing on their hardware
and ideally even allowing to use the kernel debugger normally.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@32073 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added explicit physical address parameter to vm_create_anonymous_area() and
create_area_etc(). 0 for the default behavior.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31941 a95241bf-73f2-0310-859d-f6bbb57e9c96

headers/private/kernel/util/OpenHashTable.h, Hugo's version, is a bit nicer than
Tracker's OpenHashTable.h which it should eventually replace. We've renamed the
class to BOpenHashTable and changed the interface slightly so that HashTableLink
became superfluous.
Adapted all the code that used it. Since the OpenHashTables no longer clash,
this should fix the GCC4 build.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31791 a95241bf-73f2-0310-859d-f6bbb57e9c96

* various fixes identified by gcc4, provided by Joe Prostko


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@30902 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixing warnings under GCC4 in preparation to enable -Werror there as well:
* Replaced the use of offsetof() for structs that aren't PODs. Add a
offset_of_member() macro to util/khash.h because that's what it's used for
in our cases.
* Change the signature of add_debugger_command()/remove_debugger_command() on
GCC > 2 to avoid the depricated conversion from string constants to char *.
* Adding some "suggested" parenthesis. I know that not everyone likes that, but
it pointed out at least one bug that is fixed here as well.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@29113 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Implement CACHE_DONT_SLEEP partially. At least for small object caches
without object depot it should be complete. For the other stuff
internal_alloc() would need to pass the flag on to block_alloc(), but
that isn't possible yet.
* Adjusted the low memory handler to respect the minimum object reserve
of the object caches.

The swap_test_heap test does seem to pass reliably with 128 MB RAM and
128 MB allocation, now. It's quite slow, though. Particularly while the
allocation is filled, the system is completely unusable.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27118 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added object_cache_set_minimum_reserve() which sets the minimal number
of free objects an object cache should try to have ready. If the number
of free objects drops below the threshold, a new urgent priority thread
is asked to asynchronously resize the object cache (pretty similar to
the heap grower thread). Such a mechanism is necessary for code paths
that are supposed to free pages, but may need memory themselves (like
the swap support).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27100 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added object_cache_get_usage() function that returns the memory
allocated by an object cache.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26835 a95241bf-73f2-0310-859d-f6bbb57e9c96

Merged branch haiku/branches/developer/bonefish/vm into trunk. This
introduces the following relevant changes:
* VMCache:
- Renamed vm_cache to VMCache, merged it with vm_store and made it a
C++ class with virtual methods (replacing the store operations).
Turned the different store implementations into subclasses.
- Introduced MergeStore() callback, changed semantics of Commit().
- Changed locking and referencing semantics. A reference can only be
acquired/released with the cache locked. An unreferenced cache is
deleted and a mergeable cache merged when it is unlocked. This
removes the "busy" state of a cache and simplifies the page fault
code.
* Added VMAnonymousCache, which will implement swap support (work by
Zhao Shuai). It is not integrated and used yet, though.
* Enabled the mutex/recursive lock holder asserts.
* Fixed DoublyLinkedList::Swap().
* Generalized the low memory handler to a low resource handler. And made
semaphores and reserved memory handled resources. Made
vm_try_resource_memory() optionally wait (with timeout), and used that
feature to reserve memory for areas.
...


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26572 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Initialize all static mutexes in the kernel through a MUTEX_INITIALIZER()
and remove the then unneeded mutex_init() for them.
* Remove the workaround for allowing uninitialized mutexes on kernel startup.
As they are all initialized statically through the MUTEX_INITIALIZER() now
this is not needed anymore.
* An uninitialized mutex will now cause a panic when used to find possibly
remaining cases.
* Remove now unnecessary driver_settings_init_post_sem() function.

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25812 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Changed recursive_lock to use a mutex instead of a semaphore.
* Adjusted code using recursive locks respectively. The initialization
cannot fail anymore, and it is possible to use recursive locks in the
early boot process (even uninitialized, if in BSS), which simplifies
things a little.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25687 a95241bf-73f2-0310-859d-f6bbb57e9c96

Now uses mutexes instead of benaphores, that also simplified object
instantiation at early boot a bit.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25380 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Paranoia checks were accidentally globally enabled by default. Object
* cache paranoia was always enabled.
* Changed from paranoia on/off to levels. Adjusted the macros to take a
level argument.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25215 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Changed macros that enable tracing for individual components from
defined/undefined to numeric values (0 for undefined). This allows for
trace levels.
* Set SYSCALL_TRACING_IGNORE_KTRACE_OUTPUT default to 1, since this is
what one usually wants.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25213 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added paranoia checks. They reveal that a net buffer data node is freed
twice when running the OpenSSH "forwarding" test, which corrupts the
slab's object list. It's not quite clear to me yet why that happens.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25208 a95241bf-73f2-0310-859d-f6bbb57e9c96

Removed some leftover junk.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25197 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added object cache kernel tracing.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@25196 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Also replaced the cpu store's lock with a recursive lock in order
to fix bug #1731.
* However, it turns out that depot destruction obviously doesn't work
correctly, at least we keep partial or full slabs around when we're
using them (which causes the code to panic).
* Therefore, I've now disabled depots completely, until I find the time
to really work on that code.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@23825 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Temporarily switched to a recursive lock for the depot.
* While this is not a really good idea for a lock with supposedly little
contention, but it'll fix bug #1731. I haven't tested it yet, but will
do so in a minute :-)
* I will need to rework the slab anyway so that it's possible to use it
as a replacement for our heap, and then I'll switch back to a benaphore
again.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@23822 a95241bf-73f2-0310-859d-f6bbb57e9c96

We generally handle NULL pointers in free() and friends gracefully.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@23578 a95241bf-73f2-0310-859d-f6bbb57e9c96

Print the missing object.

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@23574 a95241bf-73f2-0310-859d-f6bbb57e9c96

* vm_create_anonymous_area() now accepts B_ANY_KERNEL_BLOCK_ADDRESS.
* As a temporary work-around for the current slab allocator's area usage,
I added the CACHE_LARGE_SLAB flag, which will force the allocator to
use larger areas.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@23564 a95241bf-73f2-0310-859d-f6bbb57e9c96

The slab areas shouldn't be user readable...


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@23231 a95241bf-73f2-0310-859d-f6bbb57e9c96

Slab.h no longer includes Depot.h.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22647 a95241bf-73f2-0310-859d-f6bbb57e9c96

* register_low_memory_handler() was called too early by several kernel
components - now, we divide the initialization of that service into two parts
which allows this (before, most handlers were gone after boot).
* Added debugger command that dumps the low memory handlers.
* The slab allocator now registers its low memory handler with a higher
priority, so that it'll run before the potentially heavier ones.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22374 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved some definitions from vm_types.h that functions in vm.h need as
arguments into vm.h.
* This should fix the broken build from earlier - thanks Stefano for the note!
* That also allowed to clean some other includes a bit.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22331 a95241bf-73f2-0310-859d-f6bbb57e9c96

* vm.h no longer includes vm_types.h - only those that actually need access
to the private VM types are including vm_types.h now.
* Removed vm_page, vm_area, vm_cache, and vm_address_space typedefs; it's
cleaner this way, and the actual types are only used in C++ files now,
anyway.
* And that caused changes in many files...
* Made commpage.h self-containing.
* Minor cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22329 a95241bf-73f2-0310-859d-f6bbb57e9c96

axeld+bonefish: Got rid of the ParentType in the HashTableDefinition; it doesn't really
belong there.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@21766 a95241bf-73f2-0310-859d-f6bbb57e9c96

slab/cache: a couple more fixes.

- on CACHE_DURING_BOOT init benaphore count with 1.
- account for allocated space on early_allocate_pages.
- fixed slab position calculation in small slabs.
- we can now init all allocator sizes early rather than later.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20914 a95241bf-73f2-0310-859d-f6bbb57e9c96

cache/slab: when CACHE_DURING_BOOT is specified, use vm_allocate_early to obtain pages for slabs, and create the associated areas on post bootup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20913 a95241bf-73f2-0310-859d-f6bbb57e9c96

added object_cache_reserve to allow cache users to pre-allocate slabs.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20912 a95241bf-73f2-0310-859d-f6bbb57e9c96

added CACHE_UNLOCKED_PAGES flag to allow caches' pages to be created with no locking instead of full locking.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20911 a95241bf-73f2-0310-859d-f6bbb57e9c96

cache allocator: for allocations > 8k create areas, and delete them on free.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20902 a95241bf-73f2-0310-859d-f6bbb57e9c96

minimal cleanups


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20901 a95241bf-73f2-0310-859d-f6bbb57e9c96

don't need TRACE_SLAB by default now.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20898 a95241bf-73f2-0310-859d-f6bbb57e9c96

pushed the slab init a bit deeper. added a object cache based allocator, including a bootstrap mechanism to have it init during bootup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20896 a95241bf-73f2-0310-859d-f6bbb57e9c96

some object cache / slab improvements.

- call the reclaimer callback when low on memory.
- use the depot when on multi-cpu setups (for scalability).
- fixed the amount of memory spent on slabs for very large objects.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20889 a95241bf-73f2-0310-859d-f6bbb57e9c96

rewrote the object cache (slab) implementation a bit, preparing for further integration.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20887 a95241bf-73f2-0310-859d-f6bbb57e9c96

some slab cleanups, split the C interface to another file.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20875 a95241bf-73f2-0310-859d-f6bbb57e9c96

ups, TRACE_SLAB was on.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20852 a95241bf-73f2-0310-859d-f6bbb57e9c96

removed some of the slab's initial heavy debugging. we now merge the links into the slab itself resulting in zero overhead per buffer with MergedLink strategy.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20841 a95241bf-73f2-0310-859d-f6bbb57e9c96

slab: be smarter when picking the underlying strategy for the slab C API (based on requested object size).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20840 a95241bf-73f2-0310-859d-f6bbb57e9c96

more slab fixes, also introduced a new strategy optimized for medium sized buffers (64 >= x < 512) with lengths other than power of 2 (has an overhead of 2 words per buffer).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20839 a95241bf-73f2-0310-859d-f6bbb57e9c96

added locking to slab's Cache<>. Now we react to system's low memory conditions freeing up empty slabs.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20836 a95241bf-73f2-0310-859d-f6bbb57e9c96

assorted slab fixes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20833 a95241bf-73f2-0310-859d-f6bbb57e9c96

added initial slab code to the kernel. It is still unused, and there is still no VM interaction.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@20832 a95241bf-73f2-0310-859d-f6bbb57e9c96