drm/ttm: remove unused paramter

remove the unsed the paramter in the function
ttm_bo_bounce_temp_buffer and ttm_bo_add_move_fence.
V2:rebase the patch on top of drm-misc-next (Christian)

Signed-off-by: Jesse Zhang <Jesse.Zhang@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20240401030443.3384494-1-jesse.zhang@amd.com
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: improve idle/busy handling v5

Previously we would never try to move a BO into the preferred placements
when it ever landed in a busy placement since those were considered
compatible.

Rework the whole handling and finally unify the idle and busy handling.
ttm_bo_validate() is now responsible to try idle placement first and then
use the busy placement if that didn't worked.

Drawback is that we now always try the idle placement first for each
validation which might cause some additional CPU overhead on overcommit.

v2: fix kerneldoc warning and coding style
v3: take care of XE as well
v4: keep the ttm_bo_mem_space functionality as it is for now, only add
new handling for ttm_bo_validate as suggested by Thomas
v5: fix bug pointed out by Matthew

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Zack Rusin <zack.rusin@broadcom.com> v3
Link: https://patchwork.freedesktop.org/patch/msgid/20240229134003.3688-1-christian.koenig@amd.com
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>

drm: remove drm_debug_printer in favor of drm_dbg_printer

Convert the remaining drm_debug_printer users over to drm_dbg_printer,
as it can handle the cases without struct drm_device pointer, and also
provides drm debug category and prefix support. Remove drm_debug_printer
altogether.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
Reviewed-by: Luca Coelho <luciano.coelho@intel.com>
Acked-by: Maxime Ripard <mripard@kernel.org>
Link: https://patchwork.freedesktop.org/patch/msgid/18b5b91e62d071675a651f6f91c58f05ad74134a.1705410327.git.jani.nikula@intel.com

drm/ttm: replace busy placement with flags v6

Instead of a list of separate busy placement add flags which indicate
that a placement should only be used when there is room or if we need to
evict.

v2: add missing TTM_PL_FLAG_IDLE for i915
v3: fix auto build test ERROR on drm-tip/drm-tip
v4: fix some typos pointed out by checkpatch
v5: cleanup some rebase problems with VMWGFX
v6: implement some missing VMWGFX functionality pointed out by Zack,
rename the flags as suggested by Michel, rebase on drm-tip and
adjust XE as well

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
Reviewed-by: Zack Rusin <zack.rusin@broadcom.com>
Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20240112125158.2748-4-christian.koenig@amd.com

drm/ttm: return ENOSPC from ttm_bo_mem_space v3

Only convert it to ENOMEM in ttm_bo_validate.

This allows ttm_bo_validate to distinguish between an out of memory
situation and just out of space in a placement domain.

v2: improve commit message
v3: fix kerneldoc typos

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Zack Rusin <zack.rusin@broadcom.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20240112125158.2748-3-christian.koenig@amd.com

drm/ttm: Schedule delayed_delete worker closer

Try to allocate system memory on the NUMA node the device is closest to
and try to run delayed_delete workers on a CPU of this node as well.

To optimize the memory clearing operation when a TTM BO gets freed by
the delayed_delete worker, scheduling it closer to a NUMA node where the
memory was initially allocated helps avoid the cases where the worker
gets randomly scheduled on the CPU cores that are across interconnect
boundaries such as xGMI, PCIe etc.

This change helps USWC GTT allocations on NUMA systems (dGPU) and AMD
APU platforms such as GFXIP9.4.3.

Signed-off-by: Rajneesh Bhardwaj <rajneesh.bhardwaj@amd.com>
Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20231111130856.1168304-1-rajneesh.bhardwaj@amd.com
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: Use init_on_free to delay release TTM BOs

Delay release TTM BOs when the kernel default setting is init_on_free.
This offloads the overhead of clearing the system memory to the work
item and potentially a different CPU. This could be very beneficial when
the application does a lot of malloc/free style allocations of system
memory.

Signed-off-by: Rajneesh Bhardwaj <rajneesh.bhardwaj@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>.
Link: https://patchwork.freedesktop.org/patch/msgid/20230708011355.853-1-rajneesh.bhardwaj@amd.com
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: Don't shadow the operation context

ttm_bo_swapout() shadows the ttm operation context which may cause
major confusion in driver callbacks when swapping out !TTM_PL_SYSTEM
memory. Fix this by reusing the operation context argument to
ttm_bo_swapout().

Cc: "Christian König" <christian.koenig@amd.com>
Cc: Roger He <Hongbo.He@amd.com>
Cc: <dri-devel@lists.freedesktop.org>
Cc: <intel-gfx@lists.freedesktop.org>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Acked-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Christian König <ckoenig.leichtzumerken@gmail.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230626091450.14757-3-thomas.hellstrom@linux.intel.com

drm/ttm: check null pointer before accessing when swapping

Add a check to avoid null pointer dereference as below:

[ 90.002283] general protection fault, probably for non-canonical
address 0xdffffc0000000000: 0000 [#1] PREEMPT SMP KASAN NOPTI
[ 90.002292] KASAN: null-ptr-deref in range
[0x0000000000000000-0x0000000000000007]
[ 90.002346] ? exc_general_protection+0x159/0x240
[ 90.002352] ? asm_exc_general_protection+0x26/0x30
[ 90.002357] ? ttm_bo_evict_swapout_allowable+0x322/0x5e0 [ttm]
[ 90.002365] ? ttm_bo_evict_swapout_allowable+0x42e/0x5e0 [ttm]
[ 90.002373] ttm_bo_swapout+0x134/0x7f0 [ttm]
[ 90.002383] ? __pfx_ttm_bo_swapout+0x10/0x10 [ttm]
[ 90.002391] ? lock_acquire+0x44d/0x4f0
[ 90.002398] ? ttm_device_swapout+0xa5/0x260 [ttm]
[ 90.002412] ? lock_acquired+0x355/0xa00
[ 90.002416] ? do_raw_spin_trylock+0xb6/0x190
[ 90.002421] ? __pfx_lock_acquired+0x10/0x10
[ 90.002426] ? ttm_global_swapout+0x25/0x210 [ttm]
[ 90.002442] ttm_device_swapout+0x198/0x260 [ttm]
[ 90.002456] ? __pfx_ttm_device_swapout+0x10/0x10 [ttm]
[ 90.002472] ttm_global_swapout+0x75/0x210 [ttm]
[ 90.002486] ttm_tt_populate+0x187/0x3f0 [ttm]
[ 90.002501] ttm_bo_handle_move_mem+0x437/0x590 [ttm]
[ 90.002517] ttm_bo_validate+0x275/0x430 [ttm]
[ 90.002530] ? __pfx_ttm_bo_validate+0x10/0x10 [ttm]
[ 90.002544] ? kasan_save_stack+0x33/0x60
[ 90.002550] ? kasan_set_track+0x25/0x30
[ 90.002554] ? __kasan_kmalloc+0x8f/0xa0
[ 90.002558] ? amdgpu_gtt_mgr_new+0x81/0x420 [amdgpu]
[ 90.003023] ? ttm_resource_alloc+0xf6/0x220 [ttm]
[ 90.003038] amdgpu_bo_pin_restricted+0x2dd/0x8b0 [amdgpu]
[ 90.003210] ? __x64_sys_ioctl+0x131/0x1a0
[ 90.003210] ? do_syscall_64+0x60/0x90

Fixes: a2848d08742c ("drm/ttm: never consider pinned BOs for eviction&swap")
Tested-by: Mikhail Gavrilov <mikhail.v.gavrilov@gmail.com>
Signed-off-by: Guchun Chen <guchun.chen@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Link: https://patchwork.freedesktop.org/patch/msgid/20230724024229.1118444-1-guchun.chen@amd.com
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: never consider pinned BOs for eviction&swap

There is a small window where we have already incremented the pin count
but not yet moved the bo from the lru to the pinned list.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reported-by: Pelloux-Prayer, Pierre-Eric <Pierre-eric.Pelloux-prayer@amd.com>
Tested-by: Pelloux-Prayer, Pierre-Eric <Pierre-eric.Pelloux-prayer@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org
Link: https://patchwork.freedesktop.org/patch/msgid/20230707120826.3701-1-christian.koenig@amd.com

drm/ttm: Don't leak a resource on swapout move error

If moving the bo to system for swapout failed, we were leaking
a resource. Fix.

Fixes: bfa3357ef9ab ("drm/ttm: allocate resource object instead of embedding it v2")
Cc: Christian König <christian.koenig@amd.com>
Cc: "Christian König" <ckoenig.leichtzumerken@gmail.com>
Cc: dri-devel@lists.freedesktop.org
Cc: <stable@vger.kernel.org> # v5.14+
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Nirmoy Das <nirmoy.das@intel.com>
Reviewed-by: Andi Shyti <andi.shyti@linux.intel.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230626091450.14757-5-thomas.hellstrom@linux.intel.com

drm/ttm: Don't leak a resource on eviction error

On eviction errors other than -EMULTIHOP we were leaking a resource.
Fix.

v2:
- Avoid yet another goto (Andi Shyti)

Fixes: 403797925768 ("drm/ttm: Fix multihop assert on eviction.")
Cc: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: dri-devel@lists.freedesktop.org
Cc: <stable@vger.kernel.org> # v5.15+
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Nirmoy Das <nirmoy.das@intel.com> #v1
Reviewed-by: Andi Shyti <andi.shyti@linux.intel.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230626091450.14757-4-thomas.hellstrom@linux.intel.com

drm/ttm/ttm_bo: Provide a missing 'bulk' description and correct misnaming of 'placement'

'bulk' description taken from another in the same file.

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/ttm/ttm_bo.c:98: warning: Function parameter or member 'bulk' not described in 'ttm_bo_set_bulk_move'
drivers/gpu/drm/ttm/ttm_bo.c:768: warning: Function parameter or member 'placement' not described in 'ttm_bo_mem_space'
drivers/gpu/drm/ttm/ttm_bo.c:768: warning: Excess function parameter 'proposed_placement' description in 'ttm_bo_mem_space'

Signed-off-by: Lee Jones <lee@kernel.org>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230317081718.2650744-6-lee@kernel.org

drm/ttm: Don't print error message if eviction was interrupted

Avoid printing an error message if eviction was interrupted by,
for example, the user pressing CTRL-C. That may happen if eviction
is waiting for something, like for example a free batch-buffer.

Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230307144621.10748-6-thomas.hellstrom@linux.intel.com

drm/ttm: prevent moving of pinned BOs

We have checks for this in the individual drivers move callback, but
it's probably better to generally forbid that on a higher level.

Also stops exporting ttm_resource_compat() since that's not necessary
any more after removing the extra checks in vmwgfx.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Signed-off-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230130120636.63765-6-matthew.auld@intel.com

drm/ttm: stop allocating dummy resources during BO creation

That should not be necessary any more when drivers should at least be
able to handle the move without a resource.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Signed-off-by: Matthew Auld <matthew.auld@intel.com>
Acked-by: Nirmoy Das <nirmoy.das@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230130120636.63765-4-matthew.auld@intel.com

drm/ttm: clear the ttm_tt when bo->resource is NULL

In the next few patches, when initially creating a ttm BO, the
bo->resource is NULL, and the driver is then expected to handle the
initial dummy move. However, if this is created as a system resource
the first ttm_tt we create will always have the clear value set to
false. Previously the initial ttm_tt would be created in
ttm_bo_validate() with the clear parameter always set to true.

Signed-off-by: Matthew Auld <matthew.auld@intel.com>
Reviewed-by: Christian König <ckoenig.leichtzumerken@gmail.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230130101230.25347-3-matthew.auld@intel.com
Acked-by: Nirmoy Das <nirmoy.das@intel.com>
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: drop extra ttm_bo_put in ttm_bo_cleanup_refs

That was accidentially left over when we switched to the delayed delete
worker.

Suggested-by: Matthew Auld <matthew.william.auld@gmail.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Fixes: 9bff18d13473 ("drm/ttm: use per BO cleanup workers")
Reported-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Tested-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230316072647.406707-1-christian.koenig@amd.com

drm/ttm: revert "stop allocating dummy resources during BO creation"

This reverts commit 00984ad39599bb2a1e6ec5d4e9c75a749f7f45c9.

It seems to still breka i915.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Matthew Auld <matthew.william.auld@gmail.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230125155023.105584-1-christian.koenig@amd.com

drm/ttm: revert "prevent moving of pinned BOs"

This reverts commit b49323aa35d502b0d9a7950327f30a1a52eae534.

This still seems to break i915.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Matthew Auld <matthew.william.auld@gmail.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230125155023.105584-1-christian.koenig@amd.com

drm/ttm: prevent moving of pinned BOs

We have checks for this in the individual drivers move callback, but
it's probably better to generally forbid that on a higher level.

Also stops exporting ttm_resource_compat() since that's not necessary
any more after removing the extra checks in vmwgfx.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230124125726.13323-4-christian.koenig@amd.com

drm/ttm: stop allocating dummy resources during BO creation

That should not be necessary any more when drivers should at least be
able to handle the move without a resource.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230124125726.13323-2-christian.koenig@amd.com

drm/ttm/vmwgfx: move ttm_bo_wait into VMWGFX

Not used anymore by other drivers or TTM itself.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Zack Rusin <zackr@vmware.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20221125102137.1801-9-christian.koenig@amd.com

drm/ttm: use ttm_bo_wait_ctx instead of ttm_bo_wait

Make sure that we use the correct settings from the context.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20221125102137.1801-8-christian.koenig@amd.com

drm/ttm: merge ttm_bo_api.h and ttm_bo_driver.h v2

Merge and cleanup the two headers into a single description of the
object API. Also move all the documentation to the implementation and
drop unnecessary includes from the header.

No functional change.

v2: minimal checkpatch.pl cleanup

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20221125102137.1801-4-christian.koenig@amd.com

drm/ttm: use per BO cleanup workers

Instead of a single worker going over the list of delete BOs in regular
intervals use a per BO worker which blocks for the resv object and
locking of the BO.

This not only simplifies the handling massively, but also results in
much better response time when cleaning up buffers.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20221125102137.1801-3-christian.koenig@amd.com

drm/ttm: remove ttm_bo_(un)lock_delayed_workqueue

Those functions never worked correctly since it is still perfectly
possible that a buffer object is released and the background worker
restarted even after calling them.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20221125102137.1801-2-christian.koenig@amd.com

drm/ttm: rework on ttm_resource to use size_t type

Change ttm_resource structure from num_pages to size_t size in bytes.
v1 -> v2: change PFN_UP(dst_mem->size) to ttm->num_pages
v1 -> v2: change bo->resource->size to bo->base.size at some places
v1 -> v2: remove the local variable
v1 -> v2: cleanup cmp_size_smaller_first()
v2 -> v3: adding missing PFN_UP in ttm_bo_vm_fault_reserved

Signed-off-by: Somalapuram Amaranath <Amaranath.Somalapuram@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20221027091237.983582-1-Amaranath.Somalapuram@amd.com
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: Add new callbacks to ttm res mgr

We are adding two new callbacks to ttm resource manager
function to handle intersection and compatibility of
placement and resources.

v2: move the amdgpu and ttm_range_manager changes to
separate patches (Christian)
v3: rename "intersect" to "intersects" (Matthew)
v4: move !place check to the !res if and return false
in ttm_resource_compatible() function (Christian)
v5: move bits of code from patch number 6 to avoid
temporary driver breakup (Christian)

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20220820073304.178444-1-Arunpravin.PaneerSelvam@amd.com

drm/ttm: audit bo->resource usage v2

Allow BOs to exist without backing store.

v2: handle ttm_bo_move_memcpy as well.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michael J. Ruhl <michael.j.ruhl@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20220707102453.3633-5-christian.koenig@amd.com

drm/ttm: rename and cleanup ttm_bo_init

Rename ttm_bo_init to ttm_bo_init_validate since that better matches
what the function is actually doing.

Remove the unused size parameter, move the function's kerneldoc to the
implementation and cleanup the whole error handling.

Signed-off-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20220707102453.3633-2-christian.koenig@amd.com
Reviewed-by: Michael J. Ruhl <michael.j.ruhl@intel.com>

drm/ttm: Fix dummy res NULL ptr deref bug

Check the bo->resource value before accessing the resource
mem_type.

v2: Fix commit description unwrapped warning

<log snip>
[ 40.191227][ T184] general protection fault, probably for non-canonical address 0xdffffc0000000002: 0000 [#1] SMP KASAN PTI
[ 40.192995][ T184] KASAN: null-ptr-deref in range [0x0000000000000010-0x0000000000000017]
[ 40.194411][ T184] CPU: 1 PID: 184 Comm: systemd-udevd Not tainted 5.19.0-rc4-00721-gb297c22b7070 #1
[ 40.196063][ T184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.0-debian-1.16.0-4 04/01/2014
[ 40.199605][ T184] RIP: 0010:ttm_bo_validate+0x1b3/0x240 [ttm]
[ 40.200754][ T184] Code: e8 72 c5 ff ff 83 f8 b8 74 d4 85 c0 75 54 49 8b 9e 58 01 00 00 48 b8 00 00 00 00 00 fc ff df 48 8d 7b 10 48 89 fa 48 c1 ea 03 <0f> b6 04 02 84 c0 74 04 3c 03 7e 44 8b 53 10 31 c0 85 d2 0f 85 58
[ 40.203685][ T184] RSP: 0018:ffffc900006df0c8 EFLAGS: 00010202
[ 40.204630][ T184] RAX: dffffc0000000000 RBX: 0000000000000000 RCX: 1ffff1102f4bb71b
[ 40.205864][ T184] RDX: 0000000000000002 RSI: ffffc900006df208 RDI: 0000000000000010
[ 40.207102][ T184] RBP: 1ffff920000dbe1a R08: ffffc900006df208 R09: 0000000000000000
[ 40.208394][ T184] R10: ffff88817a5f0000 R11: 0000000000000001 R12: ffffc900006df110
[ 40.209692][ T184] R13: ffffc900006df0f0 R14: ffff88817a5db800 R15: ffffc900006df208
[ 40.210862][ T184] FS: 00007f6b1d16e8c0(0000) GS:ffff88839d700000(0000) knlGS:0000000000000000
[ 40.212250][ T184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 40.213275][ T184] CR2: 000055a1001d4ff0 CR3: 00000001700f4000 CR4: 00000000000006e0
[ 40.214469][ T184] Call Trace:
[ 40.214974][ T184] <TASK>
[ 40.215438][ T184] ? ttm_bo_bounce_temp_buffer+0x140/0x140 [ttm]
[ 40.216572][ T184] ? mutex_spin_on_owner+0x240/0x240
[ 40.217456][ T184] ? drm_vma_offset_add+0xaa/0x100 [drm]
[ 40.218457][ T184] ttm_bo_init_reserved+0x3d6/0x540 [ttm]
[ 40.219410][ T184] ? shmem_get_inode+0x744/0x980
[ 40.220231][ T184] ttm_bo_init_validate+0xb1/0x200 [ttm]
[ 40.221172][ T184] ? bo_driver_evict_flags+0x340/0x340 [drm_vram_helper]
[ 40.222530][ T184] ? ttm_bo_init_reserved+0x540/0x540 [ttm]
[ 40.223643][ T184] ? __do_sys_finit_module+0x11a/0x1c0
[ 40.224654][ T184] ? __shmem_file_setup+0x102/0x280
[ 40.234764][ T184] drm_gem_vram_create+0x305/0x480 [drm_vram_helper]
[ 40.235766][ T184] ? bo_driver_evict_flags+0x340/0x340 [drm_vram_helper]
[ 40.236846][ T184] ? __kasan_slab_free+0x108/0x180
[ 40.237650][ T184] drm_gem_vram_fill_create_dumb+0x134/0x340 [drm_vram_helper]
[ 40.238864][ T184] ? local_pci_probe+0xdf/0x180
[ 40.239674][ T184] ? drmm_vram_helper_init+0x400/0x400 [drm_vram_helper]
[ 40.240826][ T184] drm_client_framebuffer_create+0x19c/0x400 [drm]
[ 40.241955][ T184] ? drm_client_buffer_delete+0x200/0x200 [drm]
[ 40.243001][ T184] ? drm_client_pick_crtcs+0x554/0xb80 [drm]
[ 40.244030][ T184] drm_fb_helper_generic_probe+0x23f/0x940 [drm_kms_helper]
[ 40.245226][ T184] ? __cond_resched+0x1c/0xc0
[ 40.245987][ T184] ? drm_fb_helper_memory_range_to_clip+0x180/0x180 [drm_kms_helper]
[ 40.247316][ T184] ? mutex_unlock+0x80/0x100
[ 40.248005][ T184] ? __mutex_unlock_slowpath+0x2c0/0x2c0
[ 40.249083][ T184] drm_fb_helper_single_fb_probe+0x907/0xf00 [drm_kms_helper]
[ 40.250314][ T184] ? drm_fb_helper_check_var+0x1180/0x1180 [drm_kms_helper]
[ 40.251540][ T184] ? __cond_resched+0x1c/0xc0
[ 40.252321][ T184] ? mutex_lock+0x9f/0x100
[ 40.253062][ T184] __drm_fb_helper_initial_config_and_unlock+0xb9/0x2c0 [drm_kms_helper]
[ 40.254394][ T184] drm_fbdev_client_hotplug+0x56f/0x840 [drm_kms_helper]
[ 40.255477][ T184] drm_fbdev_generic_setup+0x165/0x3c0 [drm_kms_helper]
[ 40.256607][ T184] bochs_pci_probe+0x6b7/0x900 [bochs]
[ 40.257515][ T184] ? _raw_spin_lock_irqsave+0x87/0x100
[ 40.258312][ T184] ? bochs_hw_init+0x480/0x480 [bochs]
[ 40.259244][ T184] ? bochs_hw_init+0x480/0x480 [bochs]
[ 40.260186][ T184] local_pci_probe+0xdf/0x180
[ 40.260928][ T184] pci_call_probe+0x15f/0x500
[ 40.265798][ T184] ? _raw_spin_lock+0x81/0x100
[ 40.266508][ T184] ? pci_pm_suspend_noirq+0x980/0x980
[ 40.267322][ T184] ? pci_assign_irq+0x81/0x280
[ 40.268096][ T184] ? pci_match_device+0x351/0x6c0
[ 40.268883][ T184] ? kernfs_put+0x18/0x40
[ 40.269611][ T184] pci_device_probe+0xee/0x240
[ 40.270352][ T184] really_probe+0x435/0xa80
[ 40.271021][ T184] __driver_probe_device+0x2ab/0x480
[ 40.271828][ T184] driver_probe_device+0x49/0x140
[ 40.272627][ T184] __driver_attach+0x1bd/0x4c0
[ 40.273372][ T184] ? __device_attach_driver+0x240/0x240
[ 40.274273][ T184] bus_for_each_dev+0x11e/0x1c0
[ 40.275080][ T184] ? subsys_dev_iter_exit+0x40/0x40
[ 40.275951][ T184] ? klist_add_tail+0x132/0x280
[ 40.276767][ T184] bus_add_driver+0x39b/0x580
[ 40.277574][ T184] driver_register+0x20f/0x3c0
[ 40.278281][ T184] ? 0xffffffffc04a2000
[ 40.278894][ T184] do_one_initcall+0x8a/0x300
[ 40.279642][ T184] ? trace_event_raw_event_initcall_level+0x1c0/0x1c0
[ 40.280707][ T184] ? kasan_unpoison+0x23/0x80
[ 40.281479][ T184] ? kasan_unpoison+0x23/0x80
[ 40.282197][ T184] do_init_module+0x190/0x640
[ 40.282926][ T184] load_module+0x221b/0x2780
[ 40.283611][ T184] ? layout_and_allocate+0x5c0/0x5c0
[ 40.284401][ T184] ? kernel_read_file+0x286/0x6c0
[ 40.285216][ T184] ? __x64_sys_fspick+0x2c0/0x2c0
[ 40.286043][ T184] ? mmap_region+0x4e7/0x1300
[ 40.286832][ T184] ? __do_sys_finit_module+0x11a/0x1c0
[ 40.287743][ T184] __do_sys_finit_module+0x11a/0x1c0
[ 40.288636][ T184] ? __ia32_sys_init_module+0xc0/0xc0
[ 40.289557][ T184] ? __seccomp_filter+0x15e/0xc80
[ 40.290341][ T184] ? vm_mmap_pgoff+0x185/0x240
[ 40.291060][ T184] do_syscall_64+0x3b/0xc0
[ 40.291763][ T184] entry_SYSCALL_64_after_hwframe+0x46/0xb0
[ 40.292678][ T184] RIP: 0033:0x7f6b1d6279b9
[ 40.293438][ T184] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a7 54 0c 00 f7 d8 64 89 01 48
[ 40.296302][ T184] RSP: 002b:00007ffe7f51b798 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 40.297633][ T184] RAX: ffffffffffffffda RBX: 00005642dcca2880 RCX: 00007f6b1d6279b9
[ 40.298890][ T184] RDX: 0000000000000000 RSI: 00007f6b1d7b2e2d RDI: 0000000000000016
[ 40.300199][ T184] RBP: 0000000000020000 R08: 0000000000000000 R09: 00005642dccd5530
[ 40.301547][ T184] R10: 0000000000000016 R11: 0000000000000246 R12: 00007f6b1d7b2e2d
[ 40.302698][ T184] R13: 0000000000000000 R14: 00005642dcca4230 R15: 00005642dcca2880

Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Reported-by: kernel test robot <oliver.sang@intel.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20220726162205.2778-1-Arunpravin.PaneerSelvam@amd.com
Link: https://patchwork.freedesktop.org/patch/msgid/20220809095623.3569-1-Arunpravin.PaneerSelvam@amd.com
Signed-off-by: Christian König <christian.koenig@amd.com>
CC: stable@vger.kernel.org

drm/ttm: move default BO destructor into VMWGFX v2

It's the only driver using this.

v2: use BUG_ON() in vmw_bo_create() as suggested by Zack

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Zack Rusin <zackr@vmware.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20220329110243.6335-5-christian.koenig@amd.com

drm/ttm: fix bulk move handling v2

The resource must be on the LRU before ttm_lru_bulk_move_add() is called
and we need to check if the BO is pinned or not before adding it.

Additional to that we missed taking the LRU spinlock in ttm_bo_unpin().

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Acked-by: Luben Tuikov <luben.tuikov@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20220613080816.4965-1-christian.koenig@amd.com
Fixes: fee2ede15542 ("drm/ttm: rework bulk move handling v5")

drm/ttm: fixup ttm_bo_add_move_fence v2

It looks like we still need to call dma_fence_put() on the man->move,
otherwise we just end up leaking it, leading to fireworks later.

v2(Daniel):
- Simplify the function tail

Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/5689
Fixes: 8bb31587820a ("drm/ttm: remove bo->moving")
Cc: Christian König <christian.koenig@amd.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Matthew Auld <matthew.auld@intel.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20220413082133.272445-1-matthew.auld@intel.com

drm/ttm: remove bo->moving

This is now handled by the DMA-buf framework in the dma_resv obj.

Also remove the workaround inside VMWGFX to update the moving fence.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20220407085946.744568-14-christian.koenig@amd.com

dma-buf: add DMA_RESV_USAGE_BOOKKEEP v3

Add an usage for submissions independent of implicit sync but still
interesting for memory management.

v2: cleanup the kerneldoc a bit
v3: separate amdgpu changes from this

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20220407085946.744568-10-christian.koenig@amd.com

dma-buf: add DMA_RESV_USAGE_KERNEL v3

Add an usage for kernel submissions. Waiting for those are mandatory for
dynamic DMA-bufs.

As a precaution this patch also changes all occurrences where fences are
added as part of memory management in TTM, VMWGFX and i915 to use the
new value because it now becomes possible for drivers to ignore fences
with the WRITE usage.

v2: use "must" in documentation, fix whitespaces
v3: separate out some driver changes and better document why some
changes should still be part of this patch.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20220407085946.744568-5-christian.koenig@amd.com

dma-buf: specify usage while adding fences to dma_resv obj v7

Instead of distingting between shared and exclusive fences specify
the fence usage while adding fences.

Rework all drivers to use this interface instead and deprecate the old one.

v2: some kerneldoc comments suggested by Daniel
v3: fix a missing case in radeon
v4: rebase on nouveau changes, fix lockdep and temporary disable warning
v5: more documentation updates
v6: separate internal dma_resv changes from this patch, avoids to
disable warning temporary, rebase on upstream changes
v7: fix missed case in lima driver, minimize changes to i915_gem_busy_ioctl

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20220407085946.744568-3-christian.koenig@amd.com

dma-buf: add enum dma_resv_usage v4

This change adds the dma_resv_usage enum and allows us to specify why a
dma_resv object is queried for its containing fences.

Additional to that a dma_resv_usage_rw() helper function is added to aid
retrieving the fences for a read or write userspace submission.

This is then deployed to the different query functions of the dma_resv
object and all of their users. When the write paratermer was previously
true we now use DMA_RESV_USAGE_WRITE and DMA_RESV_USAGE_READ otherwise.

v2: add KERNEL/OTHER in separate patch
v3: some kerneldoc suggestions by Daniel
v4: some more kerneldoc suggestions by Daniel, fix missing cases lost in
the rebase pointed out by Bas.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20220407085946.744568-2-christian.koenig@amd.com

dma-buf/drivers: make reserving a shared slot mandatory v4

Audit all the users of dma_resv_add_excl_fence() and make sure they
reserve a shared slot also when only trying to add an exclusive fence.

This is the next step towards handling the exclusive fence like a
shared one.

v2: fix missed case in amdgpu
v3: and two more radeon, rename function
v4: add one more case to TTM, fix i915 after rebase

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20220406075132.3263-2-christian.koenig@amd.com

drm/ttm: rework bulk move handling v5

Instead of providing the bulk move structure for each LRU update set
this as property of the BO. This should avoid costly bulk move rebuilds
with some games under RADV.

v2: some name polishing, add a few more kerneldoc words.
v3: add some lockdep
v4: fix bugs, handle pin/unpin as well
v5: improve kerneldoc

Signed-off-by: Christian König <christian.koenig@amd.com>
Tested-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20220321132601.2161-5-christian.koenig@amd.com

drm/ttm: de-inline ttm_bo_pin/unpin

Those functions are going to become more complex, don't inline them any
more.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20220321132601.2161-4-christian.koenig@amd.com

drm/ttm: add resource iterator v4

Instead of duplicating that at different places add an iterator over all
the resources in a resource manager.

v2: add lockdep annotation and kerneldoc
v3: fix various bugs pointed out by Felix
v4: simplify the code a bit more

Signed-off-by: Christian König <christian.koenig@amd.com>
Tested-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch> (v2)
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20220321132601.2161-2-christian.koenig@amd.com

drm/ttm: move the LRU into resource handling v4

This way we finally fix the problem that new resource are
not immediately evict-able after allocation.

That has caused numerous problems including OOM on GDS handling
and not being able to use TTM as general resource manager.

v2: stop assuming in ttm_resource_fini that res->bo is still valid.
v3: cleanup kerneldoc, add more lockdep annotation
v4: consistently use res->num_pages

Signed-off-by: Christian König <christian.koenig@amd.com>
Tested-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20220321132601.2161-1-christian.koenig@amd.com

drm/ttm: stop pruning fences after wait

This is just abusing internals of the dma_resv object.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20211129120659.1815-3-christian.koenig@amd.com

drm/ttm: Put BO in its memory manager's lru list

After we move BO to a new memory region, we should put it to
the new memory manager's lru list regardless we unlock the resv or not.

Cc: stable@vger.kernel.org
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: xinhui pan <xinhui.pan@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20211110043149.57554-1-xinhui.pan@amd.com
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: fix ttm_bo_swapout

Commit 7120a447c7fe ("drm/ttm: Double check mem_type of BO while eviction")
made ttm_bo_evict_swapout_allowable() function actually check the
placement, but we always used a dummy placement in ttm_bo_swapout.

Fix this by using the real placement instead.

Signed-off-by: Christian König <christian.koenig@amd.com>
Fixes: 7120a447c7fe ("drm/ttm: Double check mem_type of BO while eviction")
Reviewed-by: Pan, Xinhui <Xinhui.Pan@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20211202103828.44573-1-christian.koenig@amd.com

drm/ttm: Double check mem_type of BO while eviction

BO might sit in a wrong lru list as there is a small period of memory
moving and lru list updating.

Lets skip eviction if we hit such mismatch.

Suggested-by: Christian König <christian.koenig@amd.com>
Signed-off-by: xinhui pan <xinhui.pan@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20211110043149.57554-2-xinhui.pan@amd.com
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: use the new iterator in ttm_bo_flush_all_fences

This is probably a fix since we didn't even grabed a reference to the
fences.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20211005113742.1101-11-christian.koenig@amd.com

drm/ttm: s/FLAG_SG/FLAG_EXTERNAL/

It covers more than just ttm_bo_type_sg usage, like with say dma-buf,
since one other user is userptr in amdgpu, and in the future we might
have some more. Hence EXTERNAL is likely a more suitable name.

v2(Christian):
- Rename these to TTM_TT_FLAGS_*
- Fix up all the holes in the flag values

Suggested-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Matthew Auld <matthew.auld@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Christian König <christian.koenig@amd.com>
Acked-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210929132629.353541-1-matthew.auld@intel.com
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: cleanup ttm_resource_compat

Move that function into the resource handling and remove an unused parameter.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210831112110.113196-1-christian.koenig@amd.com

drm/ttm: Create pinned list

This list will be used to capture all non VRAM BOs not
on LRU so when device is hot unplugged we can iterate
the list and unmap DMA mappings before device is removed.

v2: Reanme function to ttm_bo_move_to_pinned
v3: Move the pinned list to ttm device

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
Suggested-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/451614/?series=93971

drm/ttm: remove ttm_tt_destroy_common v2

Move the functionality into ttm_tt_fini and ttm_bo_tt_destroy instead.

We don't need this any more since we removed the unbind from the destroy
code paths in the drivers.

Also add a warning to ttm_tt_fini() if we try to fini a still populated TT
object.

v2: instead of reverting the patch move the functionality to different
places.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210728130552.2074-5-christian.koenig@amd.com

drm/ttm: Fix a deadlock if the target BO is not idle during swap

The ret value might be -EBUSY, caller will think lru lock is still
locked but actually NOT. So return -ENOSPC instead. Otherwise we hit
list corruption.

ttm_bo_cleanup_refs might fail too if BO is not idle. If we return 0,
caller(ttm_tt_populate -> ttm_global_swapout ->ttm_device_swapout) will
be stuck as we actually did not free any BO memory. This usually happens
when the fence is not signaled for a long time.

Signed-off-by: xinhui pan <xinhui.pan@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Fixes: ebd59851c796 ("drm/ttm: move swapout logic around v3")
Link: https://patchwork.freedesktop.org/patch/msgid/20210907040832.1107747-1-xinhui.pan@amd.com
Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/ttm: add TTM_PL_FLAG_TEMPORARY flag v3

Sometimes drivers need to use bounce buffers to evict BOs. While those reside
in some domain they are not necessarily suitable for CS.

Add a flag so that drivers can note that a bounce buffers needs to be
reallocated during validation.

v2: add detailed comments
v3 (chk): merge commits and rework commit message

Suggested-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Lang Yu <Lang.Yu@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
Acked-by: Nirmoy Das <nirmoy.das@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210622162339.761651-1-andrey.grodzovsky@amd.com

drm/ttm: Fix multihop assert on eviction.

Problem:
Under memory pressure when GTT domain is almost full multihop assert
will come up when trying to evict LRU BO from VRAM to SYSTEM.

Fix:
Don't assert on multihop error in evict code but rather do a retry
as we do in ttm_bo_move_buffer

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210622162339.761651-6-andrey.grodzovsky@amd.com

drm/ttm: add missing NULL checks

My local syzbot instance hit GPF in ttm_bo_release().
Unfortunately, syzbot didn't produce a reproducer for this, but I
found out possible scenario:

drm_gem_vram_create() <-- drm_gem_vram_object kzalloced
(bo embedded in this object)
ttm_bo_init()
ttm_bo_init_reserved()
ttm_resource_alloc()
man->func->alloc() <-- allocation failure
ttm_bo_put()
ttm_bo_release()
ttm_mem_io_free() <-- bo->resource == NULL passed
as second argument
*GPF*

Added NULL check inside ttm_mem_io_free() to prevent reported GPF and
make this function NULL save in future.

Same problem was in ttm_bo_move_to_lru_tail() as Christian reported.
ttm_bo_move_to_lru_tail() is called in ttm_bo_release() and mem pointer
can be NULL as well as in ttm_mem_io_free().

Fail log:

KASAN: null-ptr-deref in range [0x0000000000000020-0x0000000000000027]
...
RIP: 0010:ttm_mem_io_free+0x28/0x170 drivers/gpu/drm/ttm/ttm_bo_util.c:66
..
Call Trace:
ttm_bo_release+0xd94/0x10a0 drivers/gpu/drm/ttm/ttm_bo.c:422
kref_put include/linux/kref.h:65 [inline]
ttm_bo_put drivers/gpu/drm/ttm/ttm_bo.c:470 [inline]
ttm_bo_init_reserved+0x7cb/0x960 drivers/gpu/drm/ttm/ttm_bo.c:1050
ttm_bo_init+0x105/0x270 drivers/gpu/drm/ttm/ttm_bo.c:1074
drm_gem_vram_create+0x332/0x4c0 drivers/gpu/drm/drm_gem_vram_helper.c:228

Fixes: d3116756a710 ("drm/ttm: rename bo->mem and make it a pointer")
Signed-off-by: Pavel Skripkin <paskripkin@gmail.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210708112518.17271-1-paskripkin@gmail.com

drm/ttm: fix access to uninitialized variable.

The resource is not allocated yet, so no chance that this will work.

Use the placement instead.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210607171152.15914-1-christian.koenig@amd.com

drm/ttm, drm/amdgpu: Allow the driver some control over swapping

We are calling the eviction_valuable driver callback at eviction time to
determine whether we actually can evict a buffer object.
The upcoming i915 TTM backend needs the same functionality for swapout,
and that might actually be beneficial to other drivers as well.

Add an eviction_valuable call also in the swapout path. Try to keep the
current behaviour for all drivers by returning true if the buffer object
is already in the TTM_PL_SYSTEM placement. We change behaviour for the
case where a buffer object is in a TT backed placement when swapped out,
in which case the drivers normal eviction_valuable path is run.

Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Acked-by: Christian König <christian.koenig@amd.com>
Link: https://lore.kernel.org/r/20210602083818.241793-8-thomas.hellstrom@linux.intel.com
Link: https://patchwork.freedesktop.org/patch/msgid/20210602083818.241793-8-thomas.hellstrom@linux.intel.com

drm/ttm: Document and optimize ttm_bo_pipeline_gutting()

If the bo is idle when calling ttm_bo_pipeline_gutting(), we unnecessarily
create a ghost object and push it out to delayed destroy.
Fix this by adding a path for idle, and document the function.

Also avoid having the bo end up in a bad state vulnerable to user-space
triggered kernel BUGs if the call to ttm_tt_create() fails.

Finally reuse ttm_bo_pipeline_gutting() in ttm_bo_evict().

Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://lore.kernel.org/r/20210602083818.241793-7-thomas.hellstrom@linux.intel.com

dma-buf: drop the _rcu postfix on function names v3

The functions can be called both in _rcu context as well
as while holding the lock.

v2: add some kerneldoc as suggested by Daniel
v3: fix indentation

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20210602111714.212426-7-christian.koenig@amd.com

dma-buf: rename and cleanup dma_resv_get_list v2

When the comment needs to state explicitly that this is doesn't get a reference
to the object then the function is named rather badly.

Rename the function and use it in even more places.

v2: use dma_resv_shared_list as new name

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20210602111714.212426-5-christian.koenig@amd.com

dma-buf: rename and cleanup dma_resv_get_excl v3

When the comment needs to state explicitly that this
doesn't get a reference to the object then the function
is named rather badly.

Rename the function and use rcu_dereference_check(), this
way it can be used from both rcu as well as lock protected
critical sections.

v2: improve kerneldoc as suggested by Daniel
v3: use dma_resv_excl_fence as function name

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Link: https://patchwork.freedesktop.org/patch/msgid/20210602111714.212426-4-christian.koenig@amd.com

drm/ttm: allocate resource object instead of embedding it v2

To improve the handling we want the establish the resource object as base
class for the backend allocations.

v2: add missing error handling

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210602100914.46246-1-christian.koenig@amd.com

drm/ttm: rename bo->mem and make it a pointer

When we want to decouble resource management from buffer management we need to
be able to handle resources separately.

Add a resource pointer and rename bo->mem so that all code needs to
change to access the pointer instead.

No functional change.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210430092508.60710-4-christian.koenig@amd.com

drm/ttm: Explain why ttm_bo_add_move_fence uses a shared slot

Motivated because I got confused and Christian confirmed why this
works. I think this is non-obvious enough that it merits a slightly
longer comment.

Acked-by: Huang Rui <ray.huang@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Cc: Christian König <ckoenig.leichtzumerken@gmail.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210519082409.672016-1-daniel.vetter@ffwll.ch

drm/ttm: properly allocate sys resource during swapout

Drop the special handling here.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210430092508.60710-3-christian.koenig@amd.com

drm/ttm: always initialize the full ttm_resource v2

Init all fields in ttm_resource_alloc() when we create a new resource.

v2: use place->mem_type instead of res->mem_type

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210430092508.60710-2-christian.koenig@amd.com

drm/ttm: move the page_alignment into the BO v2

The alignment is a constant property and shouldn't change.

v2: move documentation as well as suggested by Matthew.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210413135248.1266-4-christian.koenig@amd.com

drm/ttm: remove special handling for non GEM drivers

vmwgfx is the only driver actually using this. Move the handling into
the driver instead.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Huang Rui <ray.huang@amd.com>
Reviewed-by: Zack Rusin <zackr@vmware.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210419092853.1605-1-christian.koenig@amd.com

drm/ttm/ttm_bo: Fix incorrectly documented function 'ttm_bo_cleanup_refs'

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/ttm/ttm_bo.c:293: warning: expecting prototype for function ttm_bo_cleanup_refs(). Prototype was for ttm_bo_cleanup_refs() instead

Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Sumit Semwal <sumit.semwal@linaro.org>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-media@vger.kernel.org
Cc: linaro-mm-sig@lists.linaro.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Link: https://patchwork.freedesktop.org/patch/msgid/20210416143725.2769053-24-lee.jones@linaro.org
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: warn stricter about freeing pinned BOs

So far we only warned when the BOs where pinned and not idle.

Also warn if we see a pinned BO in general.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Leo Liu <leo.liu@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210415084730.2057-3-christian.koenig@amd.com

drm/ttm: fix deref of bo->ttm without holding the lock v2

We need to grab the resv lock first before doing that check.

v2 (chk): simplify the change for -fixes

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210528130041.1683-1-christian.koenig@amd.com

drm/ttm: Do not add non-system domain BO into swap list

BO would be added into swap list if it is validated into system domain.
If BO is validated again into non-system domain, say, VRAM domain. It
actually should not be in the swap list.

Acked-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: xinhui pan <xinhui.pan@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210224032808.150465-1-xinhui.pan@amd.com
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: Ignore signaled move fences

Move fences that have already signaled should not prevent memory
allocations with no_wait_gpu.

Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210227034524.21763-1-Felix.Kuehling@amd.com
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: fix invalid NULL deref

The BO might be NULL in this function, use the bdev directly.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reported-by: Colin Ian King <colin.king@canonical.com>
Reviewed-by: Colin Ian King <colin.king@canonical.com>
Fixes: a1f091f8ef2b ("drm/ttm: switch to per device LRU lock")
Link: https://patchwork.freedesktop.org/patch/msgid/20210325152740.82633-1-christian.koenig@amd.com

drm/ttm: switch to per device LRU lock

Instead of having a global lock for potentially less contention.

Signed-off-by: Christian König <christian.koenig@amd.com>
Tested-by: Nirmoy Das <nirmoy.das@amd.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patchwork.freedesktop.org/patch/424010/

drm/ttm: remove swap LRU v3

Instead evict round robin from each devices SYSTEM and TT domain.

v2: reorder num_pages access reported by Dan's script
v3: fix rebase fallout, num_pages should be 32bit

Signed-off-by: Christian König <christian.koenig@amd.com>
Tested-by: Nirmoy Das <nirmoy.das@amd.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patchwork.freedesktop.org/patch/424009/

drm/ttm: move swapout logic around v3

Move the iteration of the global lru into the new function
ttm_global_swapout() and use that instead in drivers.

v2: consistently return int
v3: fix build fail

Signed-off-by: Christian König <christian.koenig@amd.com>
Tested-by: Nirmoy Das <nirmoy.das@amd.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patchwork.freedesktop.org/patch/424008/

drm/ttm: Do not add non-system domain BO into swap list

BO would be added into swap list if it is validated into system domain.
If BO is validated again into non-system domain, say, VRAM domain. It
actually should not be in the swap list.

Signed-off-by: xinhui pan <xinhui.pan@amd.com>
Acked-by: Guchun Chen <guchun.chen@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210224032808.150465-1-xinhui.pan@amd.com
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: move memory accounting into vmwgfx v4

This is just another feature which is only used by VMWGFX, so move
it into the driver instead.

I've tried to add the accounting sysfs file to the kobject of the drm
minor, but I'm not 100% sure if this works as expected.

v2: fix typo in KFD and avoid 64bit divide
v3: fix init order in VMWGFX
v4: use pdev sysfs reference instead of drm

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Zack Rusin <zackr@vmware.com> (v3)
Tested-by: Nirmoy Das <nirmoy.das@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210208133226.36955-2-christian.koenig@amd.com

drm/ttm: rework ttm_tt page limit v4

TTM implements a rather extensive accounting of allocated memory.

There are two reasons for this:
1. It tries to block userspace allocating a huge number of very small
BOs without accounting for the kmalloced memory.

2. Make sure we don't over allocate and run into an OOM situation
during swapout while trying to handle the memory shortage.

This is only partially a good idea. First of all it is perfectly
valid for an application to use all of system memory, limiting it to
50% is not really acceptable.

What we need to take care of is that the application is held
accountable for the memory it allocated. This is what control
mechanisms like memcg and the normal Linux page accounting already do.

Making sure that we don't run into an OOM situation while trying to
cope with a memory shortage is still a good idea, but this is also
not very well implemented since it means another opportunity of
recursion from the driver back into TTM.

So start to rework all of this by implementing a shrinker callback which
allows for TT object to be swapped out if necessary.

v2: Switch from limit to shrinker callback.
v3: fix gfp mask handling, use atomic for swapable_pages, add debugfs
v4: drop the extra gfp_mask checks

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20210208133226.36955-1-christian.koenig@amd.com

drm/ttm: device naming cleanup

Rename ttm_bo_device to ttm_device.
Rename ttm_bo_driver to ttm_device_funcs.
Rename ttm_bo_global to ttm_global.

Move global and device related functions to ttm_device.[ch].

No functional change.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/415222/

drm/ttm: add debugfs directory v2

As far as I can tell the buffer_count was never used by an
userspace application.

The number of BOs in the system is far better suited in
debugfs than sysfs and we now should be able to add other
information here as well.

v2: add that additionally to sysfs

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/414954/

drm/ttm: soften TTM warnings

QXL indeed unrefs pinned BOs and the warnings are spamming peoples log files.

Make sure we warn only once until the QXL driver is fixed.

Signed-off-by: Christian König <christian.koenig@amd.com>
References: https://lore.kernel.org/lkml/YD+eYcMMcdlXB8PY@alley/
Link: https://patchwork.freedesktop.org/patch/422834/
Reviewed-by: Daniel Vetter <daniel@ffwll.ch>
Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>

drm/ttm: Fix a memory leak

Free the memory on failure.
Also no need to re-alloc memory on retry.

Signed-off-by: xinhui pan <xinhui.pan@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210219042547.44855-1-xinhui.pan@amd.com
Reviewed-by: Christian König <christian.koenig@amd.com>
CC: stable@vger.kernel.org # 5.11
Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>

drm/ttm: Remove pinned bos from LRU in ttm_bo_move_to_lru_tail() v2

Recently a regression was introduced which caused TTM's buffer eviction to
attempt to evict already-pinned BOs, causing issues with buffer eviction
under memory pressure along with suspend/resume:

nouveau 0000:1f:00.0: DRM: evicting buffers...
nouveau 0000:1f:00.0: DRM: Moving pinned object 00000000c428c3ff!
nouveau 0000:1f:00.0: fifo: fault 00 [READ] at 0000000000200000 engine 04
[BAR1] client 07 [HUB/HOST_CPU] reason 02 [PTE] on channel -1 [00ffeaa000
unknown]
nouveau 0000:1f:00.0: fifo: DROPPED_MMU_FAULT 00001000
nouveau 0000:1f:00.0: fifo: fault 01 [WRITE] at 0000000000020000 engine
0c [HOST6] client 07 [HUB/HOST_CPU] reason 02 [PTE] on channel 1
[00ffb28000 DRM]
nouveau 0000:1f:00.0: fifo: channel 1: killed
nouveau 0000:1f:00.0: fifo: runlist 0: scheduled for recovery
[TTM] Buffer eviction failed
nouveau 0000:1f:00.0: DRM: waiting for kernel channels to go idle...
nouveau 0000:1f:00.0: DRM: failed to idle channel 1 [DRM]
nouveau 0000:1f:00.0: DRM: resuming display...

After some bisection and investigation, it appears this resulted from the
recent changes to ttm_bo_move_to_lru_tail(). Previously when a buffer was
pinned, the buffer would be removed from the LRU once ttm_bo_unreserve
to maintain the LRU list when pinning or unpinning BOs. However, since:

commit 3d1a88e1051f ("drm/ttm: cleanup LRU handling further")

We've been exiting from ttm_bo_move_to_lru_tail() at the very beginning of
the function if the bo we're looking at is pinned, resulting in the pinned
BO never getting removed from the lru and as a result - causing issues when
it eventually becomes time for eviction.

So, let's fix this by calling ttm_bo_del_from_lru() from
ttm_bo_move_to_lru_tail() in the event that we're dealing with a pinned
buffer.

v2 (chk): reduce to only the fixing one liner since we always want to
call the callback whenever we would move on the LRU.

Fixes: 3d1a88e1051f ("drm/ttm: cleanup LRU handling further")
Cc: Dave Airlie <airlied@redhat.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20210105114505.38210-1-christian.koenig@amd.com

drm/ttm: cleanup LRU handling further

We only completely delete the BO from the LRU on destruction.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
Link: https://patchwork.freedesktop.org/patch/404618/

drm/ttm: use pin_count more extensively

Check the pin_count instead of the lru list is empty here.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
Link: https://patchwork.freedesktop.org/patch/404617/

drm/ttm: cleanup BO size handling v3

Based on an idea from Dave, but cleaned up a bit.

We had multiple fields for essentially the same thing.

Now bo->base.size is the original size of the BO in
arbitrary units, usually bytes.

bo->mem.num_pages is the size in number of pages in the
resource domain of bo->mem.mem_type.

v2: use the GEM object size instead of the BO size
v3: fix printks in some places

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Huang Rui <ray.huang@amd.com> (v1)
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/406831/

drm/ttm/drivers: remove unecessary ttm_module.h include v2

ttm_module.h deals with internals of TTM and should never
be include outside of it.

v2: also move the file around

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/404885/

drm/ttm: Warn on pinning without holding a reference

Not technically a problem for ttm, but very likely a driver bug and
pretty big time confusing for reviewing code.

So warn about it, both at cleanup time (so we catch these for sure)
and at pin/unpin time (so we know who's the culprit).

Reviewed-by: Huang Rui <ray.huang@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201028113120.3641237-1-daniel.vetter@ffwll.ch

drm/ttm/ttm_bo: Fix one function header - demote lots of kernel-doc abuses

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/ttm/ttm_bo.c:51: warning: Function parameter or member 'ttm_global_mutex' not described in 'DEFINE_MUTEX'
drivers/gpu/drm/ttm/ttm_bo.c:286: warning: Function parameter or member 'bo' not described in 'ttm_bo_cleanup_memtype_use'
drivers/gpu/drm/ttm/ttm_bo.c:359: warning: Function parameter or member 'bo' not described in 'ttm_bo_cleanup_refs'
drivers/gpu/drm/ttm/ttm_bo.c:359: warning: Function parameter or member 'interruptible' not described in 'ttm_bo_cleanup_refs'
drivers/gpu/drm/ttm/ttm_bo.c:359: warning: Function parameter or member 'no_wait_gpu' not described in 'ttm_bo_cleanup_refs'
drivers/gpu/drm/ttm/ttm_bo.c:359: warning: Function parameter or member 'unlock_resv' not described in 'ttm_bo_cleanup_refs'
drivers/gpu/drm/ttm/ttm_bo.c:424: warning: Function parameter or member 'bdev' not described in 'ttm_bo_delayed_delete'
drivers/gpu/drm/ttm/ttm_bo.c:424: warning: Function parameter or member 'remove_all' not described in 'ttm_bo_delayed_delete'
drivers/gpu/drm/ttm/ttm_bo.c:635: warning: Function parameter or member 'bo' not described in 'ttm_bo_evict_swapout_allowable'
drivers/gpu/drm/ttm/ttm_bo.c:635: warning: Function parameter or member 'ctx' not described in 'ttm_bo_evict_swapout_allowable'
drivers/gpu/drm/ttm/ttm_bo.c:635: warning: Function parameter or member 'locked' not described in 'ttm_bo_evict_swapout_allowable'
drivers/gpu/drm/ttm/ttm_bo.c:635: warning: Function parameter or member 'busy' not described in 'ttm_bo_evict_swapout_allowable'
drivers/gpu/drm/ttm/ttm_bo.c:769: warning: Function parameter or member 'bo' not described in 'ttm_bo_add_move_fence'
drivers/gpu/drm/ttm/ttm_bo.c:769: warning: Function parameter or member 'man' not described in 'ttm_bo_add_move_fence'
drivers/gpu/drm/ttm/ttm_bo.c:769: warning: Function parameter or member 'mem' not described in 'ttm_bo_add_move_fence'
drivers/gpu/drm/ttm/ttm_bo.c:769: warning: Function parameter or member 'no_wait_gpu' not described in 'ttm_bo_add_move_fence'
drivers/gpu/drm/ttm/ttm_bo.c:806: warning: Function parameter or member 'bo' not described in 'ttm_bo_mem_force_space'
drivers/gpu/drm/ttm/ttm_bo.c:806: warning: Function parameter or member 'place' not described in 'ttm_bo_mem_force_space'
drivers/gpu/drm/ttm/ttm_bo.c:806: warning: Function parameter or member 'mem' not described in 'ttm_bo_mem_force_space'
drivers/gpu/drm/ttm/ttm_bo.c:806: warning: Function parameter or member 'ctx' not described in 'ttm_bo_mem_force_space'
drivers/gpu/drm/ttm/ttm_bo.c:872: warning: Function parameter or member 'bo' not described in 'ttm_bo_mem_space'
drivers/gpu/drm/ttm/ttm_bo.c:872: warning: Function parameter or member 'placement' not described in 'ttm_bo_mem_space'
drivers/gpu/drm/ttm/ttm_bo.c:872: warning: Function parameter or member 'mem' not described in 'ttm_bo_mem_space'
drivers/gpu/drm/ttm/ttm_bo.c:872: warning: Function parameter or member 'ctx' not described in 'ttm_bo_mem_space'
drivers/gpu/drm/ttm/ttm_bo.c:1387: warning: Function parameter or member 'ctx' not described in 'ttm_bo_swapout'

Reviewed-by: Christian König <christian.koenig@amd.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Sumit Semwal <sumit.semwal@linaro.org>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-media@vger.kernel.org
Cc: linaro-mm-sig@lists.linaro.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20201116174112.1833368-32-lee.jones@linaro.org

drm/ttm: add multihop infrastrucutre (v3)

Currently drivers get called to move a buffer, but if they have to
move it temporarily through another space (SYSTEM->VRAM via TT)
then they can end up with a lot of ttm->driver->ttm call stacks,
if the temprorary space moves requires eviction.

Instead of letting the driver do all the placement/space for the
temporary, allow it to report back (-EMULTIHOP) and a placement (hop)
to the move code, which will then do the temporary move, and the
correct placement move afterwards.

This removes a lot of code from drivers, at the expense of
adding some midlayering. I've some further ideas on how to turn
it inside out, but I think this is a good solution to the call
stack problems.

v2: separate out the driver patches, add WARN for getting
MULTHOP in paths we shouldn't (Daniel)
v3: use memset (Christian)

Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: hristian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201109005432.861936-2-airlied@gmail.com

drm/ttm: replace context flags with bools v2

The ttm_operation_ctx structure has a mixture of flags and bools. Drop the
flags and replace them with bools as well.

v2: fix typos, improve comments

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/398686/

drm/ttm: nuke old page allocator

Not used any more.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Reviewed-by: Madhav Chauhan <madhav.chauhan@amd.com>
Tested-by: Huang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/397087/?series=83051&rev=1

drm/ttm: wire up the new pool as default one v2

Provide the necessary parameters by all drivers and use the new pool alloc
when no driver specific function is provided.

v2: fix the GEM VRAM helpers

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Reviewed-by: Madhav Chauhan <madhav.chauhan@amd.com>
Tested-by: Huang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/397081/?series=83051&rev=1

drm/ttm: merge ttm_dma_tt back into ttm_tt

It makes no difference to kmalloc if the structure
is 48 or 64 bytes in size.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/396950/

drm/ttm: replace last move_notify with delete_mem_notify

The move notify callback is only used in one place, this should
be removed in the future, but for now just rename it to the use
case which is to notify the driver that the GPU memory is to be
deleted.

Drivers can be cleaned up after this separately.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201021044031.1752624-2-airlied@gmail.com

drm/ttm: ttm_bo_mem_placement doesn't need ctx parameter.

Removed unused parameter.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201021044031.1752624-3-airlied@gmail.com

drm/ttm: move last binding into the drivers.

This moves the call to tt binding into the driver move,
and drops the driver callback.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201020010319.1692445-8-airlied@gmail.com

drm/ttm: drop move notify around move.

The drivers now do this in the move callback.

move_notify is still needed in the destroy path.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201020010319.1692445-7-airlied@gmail.com

drm/ttm: remove move to new and inline into remainging place.

This show the remaining bind callback, which my next series of
patches will aim to remove.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201020010319.1692445-6-airlied@gmail.com

drm/ttm: drop unbind callback.

The drivers now control this, so drop unbinding.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201020010319.1692445-5-airlied@gmail.com

drm/ttm: fix eviction valuable range check.

This was adding size to start, but pfn and start are in pages,
so it should be using num_pages.

Not sure this fixes anything in the real world, just noticed it
during refactoring.

Signed-off-by: Dave Airlie <airlied@redhat.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Link: https://patchwork.freedesktop.org/patch/msgid/20201019222257.1684769-2-airlied@gmail.com

drm/ttm: refactor out common code to setup a new tt backed resource

This factors out the code to setup non-system tt.

The same code was used twice in the move paths.

Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201019071314.1671485-2-airlied@gmail.com

drm/ttm: nuke caching placement flags

Changing the caching on the fly never really worked
flawlessly.

So stop this completely and just let drivers specific the
desired caching in the tt or bus object.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michael J. Ruhl <michael.j.ruhl@intel.com>
Link: https://patchwork.freedesktop.org/patch/394256/

drm/vmwgfx: move ttm_bo_swapout_all into vmwgfx

It is the sole user of this.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/393498/

drm/ttm: drop glob parameter from ttm_bo_swapout

We can always access the global state.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/393499/

drm/ttm: nuke ttm_bo_evict_mm and rename mgr function v3

Make it more clear what the resource manager function
does and nuke the wrapper function.

v2: nuke the wrapper
v3: fix typo in radeon, rebased

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch> (v2)
Link: https://patchwork.freedesktop.org/patch/393914/

drm/ttm: make move callback compulstory

All drivers should have a move callback now so make it compulsory.

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20201006000644.1005758-6-airlied@gmail.com

drm/ttm: handle the SYSTEM->TT path in same place as others.

This just consolidates the code making the flow easier to understand
and also helps when moving move to the driver side.

Reviewed-by: Christian König <christian.koenig@amd.com>.
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200924051845.397177-3-airlied@gmail.com

drm/ttm: remove TTM_PL_FLAG_NO_EVICT

Not used any more.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/391604/?series=81973&rev=1

drm/ttm: remove ttm_bo_create

Not used any more.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/391616/?series=81973&rev=1

drm/ttm: add ttm_bo_pin()/ttm_bo_unpin() v2

As an alternative to the placement flag add a
pin count to the ttm buffer object.

v2: add dma_resv_assert_help() calls

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/391596/?series=81973&rev=1

drm/ttm: remove persistent_swap_storage

Not used any more. Cleanup the code as well while at it.

Signed-off-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/391079/?series=81804&rev=1
Reviewed-by: Dave Airlie <airlied@redhat.com>

drm/ttm: stop dangerous caching attribute change

When we swapout/in a BO we try to change the caching
attributes of the pages before/after doing the copy.

On x86 this is done by calling set_pages_uc(),
set_memory_wc() or set_pages_wb() for not highmem pages
to update the linear mapping of the page.

On all other platforms we do exactly nothing.

Now on x86 this is unnecessary because copy_highpage() will
either create a temporary mapping of the page which is wb
anyway and destroyed immediately again or use the linear
mapping with the correct caching attributes.

So stop this nonsense and just keep the caching as it is and
return an error when a driver tries to change the caching of
an already populated TT object.

This is much more defensive since changing caching
attributes is platform and driver specific and usually
doesn't work after the page was initially allocated.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Link: https://patchwork.freedesktop.org/patch/391293/

drm/ttm: drop evicted from ttm_bo.

This was unused.

Signed-off-by: Dave Airlie <airlied@redhat.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200917064132.148521-3-airlied@gmail.com

drm/ttm/drivers: call the bind function directly.

Now the bind functions have all the protection explicitly the
drivers can just call them directly, and the api can be unexported

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200917043040.146575-5-airlied@gmail.com

drm/ttm: move unbind into the tt destroy.

This moves unbind into the driver side on destroy paths.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200917043040.146575-4-airlied@gmail.com

drm/ttm: protect against reentrant bind in the drivers

This moves the generic tracking into the drivers and protects
against reentrancy in the drivers. It fixes up radeon and agp
to be able to query the bound status as that is required.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200917043040.146575-2-airlied@gmail.com

drm/ttm: some cleanups

Unexport ttm_check_under_lowerlimit.
Make ttm_bo_acc_size static and unexport it.
Remove ttm_get_kernel_zone_memory_size.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/390515/

drm/ttm: move ttm binding/unbinding out of ttm_tt paths.

Move these up to the bo level, moving ttm_tt to just being
backing store. Next step is to move the bound flag out.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200915024007.67163-6-airlied@gmail.com

drm/ttm: split populate out from binding.

Drivers have to call populate themselves now before binding.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200915024007.67163-5-airlied@gmail.com

drm/ttm: tt destroy move null check to outer function.

This just makes things easier later.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200915024007.67163-4-airlied@gmail.com

drm/ttm: wrap tt destroy. (v2)

All places this was called was using bo->ttm either direct
or indirectly.

v2: move to ttm_bo

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200915024007.67163-3-airlied@gmail.com

drm/ttm: remove available_caching

Instead of letting TTM make an educated guess based on
some mask all drivers should just specify what caching
they want for their CPU mappings.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/390207/

drm/ttm: remove default caching

As far as I can tell this was never used either and we just
always fallback to the order cached > wc > uncached anyway.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/390142/

drm/ttm: nuke memory type flags

It's not supported to specify more than one of those flags.
So it never made sense to make this a flag in the first place.

Nuke the flags and specify directly which memory type to use.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/389826/?series=81551&rev=1

drm/ttm: merge offset and base in ttm_bus_placement

This is used by TTM to communicate the physical address
which should be used with ioremap(), ioremap_wc(). We don't
need to separate the base and offset in any way here.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/389457/

drm/ttm: remove bdev from ttm_tt

I want to split this structure up and use it differently,
step one remove bdev pointer from it and pass it explicitly.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200826014428.828392-4-airlied@gmail.com

drm/ttm: remove io_reserve_lru handling v3

That is not used any more.

v2: keep the NULL checks in TTM.
v3: remove unused variable

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Link: https://patchwork.freedesktop.org/patch/388646/

drm/ttm: make sure that we always zero init mem.bus v2

We are trying to remove the io_lru handling and depend
on zero init base, offset and addr here.

v2: init addr as well

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Link: https://patchwork.freedesktop.org/patch/388642/

drm/ttm: fix broken merge between drm-next and drm-misc-next

drm-next reverted the changes to ttm_tt_create() to do the
NULL check inside the function, but drm-misc-next adds new
users of this approach.

Re-apply the NULL check change inside the function to fix this.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/386628/

drm/ttm: give resource functions their own [ch] files

This is a separate object we work within TTM.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/384338/?series=80346&rev=1

drm/ttm: rename ttm_resource_manager_func callbacks

The names get/put are associated with reference counting
in the Linux kernel, use alloc/free instead.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/384340/?series=80346&rev=1

drm/ttm: revert "drm/ttm: make TT creation purely optional v3"

This reverts commit 2ddef17678bc2ea1d20517dd2b4ed4aa967ffa8b.

As it turned out VMWGFX needs a much wider audit to fix this.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200811092400.188124-1-christian.koenig@amd.com

drm/ttm: rename ttm_mem_reg to ttm_resource.

This name better reflects what the object does. I didn't rename
all the pointers it seemed too messy.

Signed-off-by: Dave Airlie <airlied@redhat.com>
Acked-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-60-airlied@gmail.com

drm/ttm: rename ttm_mem_type_manager -> ttm_resource_manager.

This name makes a lot more sense, since these are about managing
driver resources rather than just memory ranges.

Acked-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-59-airlied@gmail.com

drm/ttm: add a wrapper for checking if manager is in use

This converts vmwgfx over to using an interface to set the
in use and check the in use flag.

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-57-airlied@gmail.com

drm/ttm: drop type manager has_type

under driver control, this flag isn't needed anymore,
remove the API that used to access it, and consoldiate
with the used api.

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-56-airlied@gmail.com

drm/ttm: drop list of memory managers from device. (v2)

The driver now controls these, the core just controls the system
memory one.

v2: init sysman explicitly and assign it as a driver manager
to simplify the lookup sequence.

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-55-airlied@gmail.com

drm/ttm: drop man->bdev link.

This link isn't needed anymore, drop it from the init interface.

Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-54-airlied@gmail.com

drm/ttm: rename manager variable to make sure wrapper is used.

Other users of this should notice this change and switch to wrapper.

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-44-airlied@gmail.com

drm/ttm: add wrapper to get manager from bdev.

This will allow different abstractions later.

Acked-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-37-airlied@gmail.com

drm/ttm: make TTM responsible for cleaning system only.

Drivers should all be cleaning up their memory managers
themselves now, so let the core just clean the system one up.

Remove the legacy cleaning interface.

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-36-airlied@gmail.com

drm/ttm: start allowing drivers to use new takedown path (v2)

Allow the takedown path callback to be optional as well.

v2: use fini for range manager

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-27-airlied@gmail.com

drm/ttm: make some inline helper functions for cleanup paths. (v2)

The disable path is just temporary for now, it will be dropped once has_type
is gone in a later patch.

v2: add docs.
rename to ttm_mem_type_manager namespace

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-26-airlied@gmail.com

drm/ttm: pass man around instead of mem_type in some places

This makes it easier to cleanup things

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-25-airlied@gmail.com

drm/ttm: purge old manager init path.

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-24-airlied@gmail.com

drm/ttm: convert system manager init to new code.

Remove the exit path, since this can't fail now.

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-23-airlied@gmail.com

drm/ttm: split the mm manager init code (v2)

This will allow the driver to control the ordering here better.

Eventually the old path will be removed.

v2: add docs for new APIs.
rename new path to ttm_mem_type_manager_init/set_used(for now)

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-14-airlied@gmail.com

drm/ttm: export memory type debug entrypoint.

As suggested on review, just export the memory type debug for
drivers to use, while also making the debug callback optional
(don't need to test for system as it won't init it).

rename it to be more consistent with object name for now.
(we may rename all the objects later.)

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-9-airlied@gmail.com

drm/ttm: use a helper for unlocked moves to the lru tail

The pattern was repeated a few times, just make an inline for it.

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-7-airlied@gmail.com

drm/ttm: remove the init_mem_type callback

It is a very strange concept to call a function which just
calls back the caller for the functions parameters.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Link: https://patchwork.freedesktop.org/patch/382085/

drm/ttm: remove TTM_MEMTYPE_FLAG_FIXED v2

Instead use a boolean field in the memory manager structure.

Also invert the meaning of the field since the use of a TT
structure is the special case here.

v2: cleanup zero init.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Link: https://patchwork.freedesktop.org/patch/382079/

drm/ttm: initialize the system domain with defaults v2

Instead of repeating that in each driver.

v2: keep the caching limitation for VMWGFX for now.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de>
Link: https://patchwork.freedesktop.org/patch/382078/

drm/ttm: ttm_bo_swapout_all doesn't use it's argument.

Just drop the argument from this.

This does ask the question if this is the function vmwgfx
should be using or should it be doing an evict all like
the other drivers.

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Zack Rusin <zackr@vmware.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200728034254.20114-1-airlied@gmail.com

drm/ttm: fix pipelined gutting for evictions v2

We can't pipeline that during eviction because the memory needs
to be available immediately.

v2: fix how we cleanup the BOs resources

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
Tested-by: Alex Sierra <alex.sierra@amd.com>
Link: https://patchwork.freedesktop.org/patch/379039/

drm/ttm: remove TTM_MEMTYPE_FLAG_CMA

The original intention was to avoid CPU page table unmaps
when BOs move between the GTT and SYSTEM domain.

The problem is that this never correctly handled changes
in the caching attributes or backing pages.

Just drop this for now and simply unmap the CPU page
tables in all cases.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/378240/

drm/ttm: remove io_reserve_fastpath flag

Just use the use_io_reserve_lru flag. It doesn't make much
sense to have two flags.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/378238/

drm/ttm: further cleanup ttm_mem_reg handling

Stop touching the backend private pointer alltogether and
make sure we never put the same mem twice by.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Madhav Chauhan <madhav.chauhan@amd.com>
Link: https://patchwork.freedesktop.org/patch/375613/

drm/ttm: make TT creation purely optional v3

We only need the page array when the BO is about to be accessed.

So not only populate, but also create it on demand.

v2: move NULL check into ttm_tt_create()
v3: fix the occurrence in ttm_bo_kmap_ttm as well

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michael J. Ruhl <michael.j.ruhl@intel.com>
Link: https://patchwork.freedesktop.org/patch/373182/

drm/ttm: cleanup ttm_mem_type_manager_func.get_node interface v3

Instead of signaling failure by setting the node pointer to
NULL do so by returning -ENOSPC.

v2: add memset() to make sure that mem is always initialized.
v3: drop memset() only set mm_node = NULL, move mm_node init in amdgpu

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michael J. Ruhl <michael.j.ruhl@intel.com>
Link: https://patchwork.freedesktop.org/patch/373181/

drm/ttm: do not keep GPU dependent addresses

GPU address handling is device specific and should be handle by its device
driver.

Signed-off-by: Nirmoy Das <nirmoy.das@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/372937/
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: Fix dma_fence refcnt leak when adding move fence

ttm_bo_add_move_fence() invokes dma_fence_get(), which returns a
reference of the specified dma_fence object to "fence" with increased
refcnt.

When ttm_bo_add_move_fence() returns, local variable "fence" becomes
invalid, so the refcount should be decreased to keep refcount balanced.

The reference counting issue happens in one exception handling path of
ttm_bo_add_move_fence(). When no_wait_gpu flag is equals to true, the
function forgets to decrease the refcnt increased by dma_fence_get(),
causing a refcnt leak.

Fix this issue by calling dma_fence_put() when no_wait_gpu flag is
equals to true.

Signed-off-by: Xiyu Yang <xiyuyang19@fudan.edu.cn>
Signed-off-by: Xin Tan <tanxin.ctf@gmail.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/370221/
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: lock resv object during destruction

Calling ttm_bo_cleanup_memtype_use() destroys the TT object
which in turn could result in warnings without this.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Acked-by: Sumit Semwal <sumit.semwal@linaro.org>
Link: https://patchwork.freedesktop.org/patch/359288

drm/ttm: fix false positive assert

The assert sometimes incorrectly triggers when pinned BOs are destroyed.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Tested-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Link: https://patchwork.freedesktop.org/patch/356737/

drm/ttm: remove the backing store if no placement is given

Pipeline removal of the BOs backing store when no placement is given
during validation.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/353994/?series=73646&rev=1

drm/ttm: individualize resv objects before calling release_notify

This allows release_notify to add and remove fences from deleted objects.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: xinhui pan <xinhui.pan@amd.com>
Link: https://patchwork.freedesktop.org/patch/352750/

drm/ttm: replace dma_resv object on deleted BOs v3

When non-imported BOs are resurrected for delayed delete we replace
the dma_resv object to allow for easy reclaiming of the resources.

v2: move that to ttm_bo_individualize_resv
v3: add a comment to explain what's going on

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: xinhui pan <xinhui.pan@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/352927/

drm/ttm: rework BO delayed delete. v2

This patch reworks the whole delayed deletion of BOs which aren't idle.

Instead of having two counters for the BO structure we resurrect the BO
when we find that a deleted BO is not idle yet.

This has many advantages, especially that we don't need to
increment/decrement the BOs reference counter any more when it
moves on the LRUs.

v2: remove duplicate ttm_tt_destroy, fix holde lock for LRU move

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: xinhui pan <xinhui.pan@amd.com>
Link: https://patchwork.freedesktop.org/patch/352912/

drm/ttm: use RCU in ttm_bo_flush_all_fences

This allows it to call the function without the lock held.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: xinhui pan <xinhui.pan@amd.com>
Link: https://patchwork.freedesktop.org/patch/352742/

drm/ttm: cleanup ttm_buffer_object_transfer

The function is always called with deleted BOs.

While at it cleanup the indentation as well.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: xinhui pan <xinhui.pan@amd.com>
Link: https://patchwork.freedesktop.org/patch/352743/

drm/ttm: flush the fence on the bo after we individualize the reservation object

As we move the ttm_bo_individualize_resv() upwards, we need flush the
copied fence too. Otherwise the driver keeps waiting for fence.

run&Kill kfdtest, then perf top.

25.53% [ttm] [k] ttm_bo_delayed_delete
24.29% [kernel] [k] dma_resv_test_signaled_rcu
19.72% [kernel] [k] ww_mutex_lock

Fix: 378e2d5b("drm/ttm: fix ttm_bo_cleanup_refs_or_queue once more")
Signed-off-by: xinhui pan <xinhui.pan@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/series/72339/
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/ttm: nuke invalidate_caches callback

Another completely unused feature.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Link: https://patchwork.freedesktop.org/patch/348265/

drm/ttm: remove ttm_bo_wait_unreserved

With nouveau fixed all ttm-using drives have the correct nesting of
mmap_sem vs dma_resv, and we can just lock the buffer.

Assuming I didn't screw up anything with my audit of course.

v2:
- Dont forget wu_mutex (Christian König)
- Keep the mmap_sem-less wait optimization (Thomas)
- Use _lock_interruptible to be good citizens (Thomas)

v3: Rebase over fault handler helperification.

Reviewed-by: Christian König <christian.koenig@amd.com> (v2)
Reviewed-by: Thomas Hellström <thellstrom@vmware.com> (v2)
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: "VMware Graphics" <linux-graphics-maintainer@vmware.com>
Cc: Thomas Hellstrom <thellstrom@vmware.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191104173801.2972-3-daniel.vetter@ffwll.ch

drm/ttm: remove pointers to globals

As the name says global memory and bo accounting is global. So it doesn't
make to much sense having pointers to global structures all around the code.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Thomas Hellström <thellstrom@vmware.com>
Link: https://patchwork.freedesktop.org/patch/332879/

drm/ttm: always keep BOs on the LRU

This allows blocking for BOs to become available
in the memory management.

Amdgpu is doing this for quite a while now during CS. Now
apply the new behavior to all drivers using TTM.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Thomas Hellstrom <thellstrom@vmware.com>
Link: https://patchwork.freedesktop.org/patch/332878/

drm/ttm, drm/vmwgfx: move cpu_writers handling into vmwgfx

This feature is only used by vmwgfx and superfluous for everybody else.

Signed-off-by: Christian König <christian.koenig@amd.com>
Co-developed-by: Thomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: Thomas Hellstrom <thellstrom@vmware.com>
Tested-by: Thomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: https://patchwork.freedesktop.org/patch/333650/

drm/ttm: fix handling in ttm_bo_add_mem_to_lru

We should not add the BO to the swap LRU when the new mem is fixed and
the TTM object about to be destroyed.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Kevin Wang <kevin1.wang@amd.com>
Link: https://patchwork.freedesktop.org/patch/335246/

drm/ttm: fix busy reference in ttm_mem_evict_first

The busy BO might actually be already deleted,
so grab only a list reference.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Thomas Hellström <thellstrom@vmware.com>
Link: https://patchwork.freedesktop.org/patch/332877/

drm/ttm: return -EBUSY on pipelining with no_gpu_wait (v2)

Setting the no_gpu_wait flag means that the allocate BO must be available
immediately and we can't wait for any GPU operation to finish.

v2: squash in mem leak fix, rebase

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: remove embedded vma_offset_manager

No users left. Drivers either setup vma_offset_manager themself
(vmwgfx) or pass the gem vma_offset_manager to ttm_bo_device_init
(all other drivers).

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20190905070509.22407-9-kraxel@redhat.com

drm/ttm: turn ttm_bo_device.vma_manager into a pointer

Rename the embedded struct vma_offset_manager, new name is _vma_manager.
ttm_bo_device.vma_manager changed to a pointer.

The ttm_bo_device_init() function gets an additional vma_manager
argument which allows to initialize ttm with a different vma manager.
When passing NULL the embedded _vma_manager is used.

All callers are updated to pass NULL, so the behavior doesn't change.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20190905070509.22407-2-kraxel@redhat.com

dma-buf: rename reservation_object to dma_resv

Be more consistent with the naming of the other DMA-buf objects.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/323401/

drm/ttm: drop ttm_buffer_object->resv

All users moved to ttm_buffer_object->base.resv

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20190805140119.7337-18-kraxel@redhat.com

drm/ttm: switch ttm core from bo->resv to bo->base.resv

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20190805140119.7337-11-kraxel@redhat.com

drm/ttm: set both resv and base.resv pointers

Initialize both ttm_buffer_object->resv and ttm_buffer_object->base.resv
pointers. This allows to move users from the former to the latter. When
all users are moved we can drop ttm_buffer_object->resv.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20190805140119.7337-10-kraxel@redhat.com

drm/ttm: use gem vma_node

Drop vma_node from ttm_buffer_object, use the gem struct
(base.vma_node) instead.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20190805140119.7337-9-kraxel@redhat.com

drm/ttm: use gem reservation object

Drop ttm_resv from ttm_buffer_object, use the gem reservation object
(base._resv) instead.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20190805140119.7337-8-kraxel@redhat.com

dma-buf: add more reservation object locking wrappers

Complete the abstraction of the ww_mutex inside the reservation object.

This allows us to add more handling and debugging to the reservation
object in the future.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/320761/

drm/ttm: Add release_notify callback to ttm_bo_driver

This notifies the driver that a BO is about to be released.

Releasing a BO also invokes the move_notify callback from
ttm_bo_cleanup_memtype_use, but that happens too late for anything
that would add fences to the BO and require a delayed delete.

Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

Revert "Merge branch 'vmwgfx-next' of git://people.freedesktop.org/~thomash/linux into drm-next"

This reverts commit 031e610a6a21448a63dff7a0416e5e206724caac, reversing
changes made to 52d2d44eee8091e740d0d275df1311fb8373c9a9.

The mm changes in there we premature and not fully ack or reviewed by core mm folks,
I dropped the ball by merging them via this tree, so lets take em all back out.

Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/ttm: return -EBUSY if waiting for busy BO fails

Returning -EAGAIN prevents ttm_bo_mem_space from trying alternate
placements and can lead to live-locks in amdgpu_cs, retrying
indefinitely and never succeeding.

Fixes: d367bd2a5e2b12 ("drm/ttm: fix busy memory to fail other user v10")
Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Allow the driver to provide the ttm struct vm_operations_struct

Add a pointer to the struct vm_operations_struct in the bo_device, and
assign that pointer to the default value currently used.

The driver can then optionally modify that pointer and the new value
can be used for each new vma created.

Cc: "Christian König" <christian.koenig@amd.com>

Signed-off-by: Thomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: Christian König <christian.koenig@amd.com>

drm/ttm: fix busy memory to fail other user v10

BOs on the LRU might be blocked during command submission
and cause OOM situations.

Avoid this by blocking for the first busy BO not locked by
the same ticket as the BO we are searching space for.

v10: completely start over with the patch since we didn't
handled a whole bunch of corner cases.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Tested-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: immediately move BOs to the new LRU v3

Move BOs which are currently in a lower domain to the new
LRU before allocating backing space while validating.

This makes sure that we always have enough entries on the
LRU to allow for other processes to wait for an operation
to complete.

v2: generalize the test
v3: fix rebase error

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Tested-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: cleanup ttm_bo_mem_space

We tried this once before, but that turned out to be more
complicated than thought. With all the right prerequisites
it looks like we can do this now.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Chunming Zhou <david1.zhou@amd.com>
Tested-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: remove manual placement preference

If drivers don't prefer a system memory placement
they should not but it into the placement list first.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Chunming Zhou <david1.zhou@amd.com>
Tested-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: return immediately in case of a signal

When a signal arrives we should return immediately for
handling it and not try other placements first.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Chunming Zhou <david1.zhou@amd.com>
Tested-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Make LRU removal optional v2

We are already doing this for DMA-buf imports and also for
amdgpu VM BOs for quite a while now.

If this doesn't run into any problems we are probably going
to stop removing BOs from the LRU altogether.

v2: drop BUG_ON from ttm_bo_add_to_lru

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Tested-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: fix re-init of global structures

When a driver unloads without unloading TTM we don't correctly
clear the global structures leading to errors on re-init.

Next step should probably be to remove the global structures and
kobjs all together, but this is tricky since we need to maintain
backward compatibility.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Tested-by: Karol Herbst <kherbst@redhat.com>
CC: stable@vger.kernel.org # 5.0.x
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: fix dma_fence refcount imbalance on error path

the ttm_bo_add_move_fence takes a reference to the struct dma_fence, but
failed to release it on the error path, leading to a memory leak.
add dma_fence_put before return when error occur.

Signed-off-by: Lin Yi <teroincn@163.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm: Use the same mmap-range offset and size for GEM and TTM

GEM defines DRM_FILE_PAGE_OFFSET_{START,SIZE} constants for the
mmap-able range of addresses. TTM can use them as well.

Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Christian König <christian.koenig@amd.com>
Acked-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Remove file_page_offset parameter from ttm_bo_device_init()

The parameter file_page_offset is a constant shared by all drivers. Just
replace it with the constant itself.

Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Christian König <christian.koenig@amd.com>
Acked-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Remove ttm_bo_reference and ttm_bo_unref

Both functions are obsolete and all calls have been replaced by
ttm_bo_get and ttm_bo_put.

Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: add lru notify to bo driver v2

allow driver do somethings when lru changed.
v2:
address Michel's comments.

Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Use drm_debug_printer for all ttm_bo_mem_space_debug output

No need for pr_err here, the pr_err message in ttm_bo_evict is enough
to draw attention to something not going as planned.

Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Junwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Fix bo_global and mem_global kfree error

ttm_bo_glob and ttm_mem_glob are defined as structure instance, while
not allocated by kzalloc, so kfree should not be invoked to release
them anymore. Otherwise, it will cause the following kernel BUG when
unloading amdgpu module

[ 48.419294] kernel BUG at /build/linux-5s7Xkn/linux-4.15.0/mm/slub.c:3894!
[ 48.419352] invalid opcode: 0000 [#1] SMP PTI
[ 48.419387] Modules linked in: amdgpu(OE-) amdchash(OE) amdttm(OE) amd_sched(OE) amdkcl(OE) amd_iommu_v2 drm_kms_helper drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi pcbc snd_seq snd_seq_device snd_timer aesni_intel snd soundcore joydev aes_x86_64 crypto_simd glue_helper cryptd input_leds mac_hid serio_raw binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables autofs4 8139too psmouse i2c_piix4 8139cp mii floppy pata_acpi
[ 48.419782] CPU: 1 PID: 1281 Comm: modprobe Tainted: G OE 4.15.0-20-generic #21-Ubuntu
[ 48.419838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[ 48.419901] RIP: 0010:kfree+0x137/0x180
[ 48.419934] RSP: 0018:ffffb02101273bf8 EFLAGS: 00010246
[ 48.419974] RAX: ffffeee1418ad7e0 RBX: ffffffffc075f100 RCX: ffff8fed7fca7ed0
[ 48.420025] RDX: 0000000000000000 RSI: 000000000003440e RDI: 0000000022400000
[ 48.420073] RBP: ffffb02101273c10 R08: 0000000000000010 R09: ffff8fed7ffd3680
[ 48.420121] R10: ffffeee1418ad7c0 R11: ffff8fed7ffd3000 R12: ffffffffc075e2c0
[ 48.420169] R13: ffffffffc074ec10 R14: ffff8fed73063900 R15: ffff8fed737428e8
[ 48.420216] FS: 00007fdc912ec540(0000) GS:ffff8fed7fc80000(0000) knlGS:0000000000000000
[ 48.420267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 48.420308] CR2: 000055fa40c30060 CR3: 000000023470a006 CR4: 00000000003606e0
[ 48.420358] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 48.420405] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 48.420452] Call Trace:
[ 48.420485] ttm_bo_global_kobj_release+0x20/0x30 [amdttm]
[ 48.420528] kobject_release+0x6a/0x180
[ 48.420562] kobject_put+0x28/0x50
[ 48.420595] ttm_bo_global_release+0x36/0x50 [amdttm]
[ 48.420636] amdttm_bo_device_release+0x119/0x180 [amdttm]
[ 48.420678] ? amdttm_bo_clean_mm+0xa6/0xf0 [amdttm]
[ 48.420760] amdgpu_ttm_fini+0xc9/0x180 [amdgpu]
[ 48.420821] amdgpu_bo_fini+0x12/0x40 [amdgpu]
[ 48.420889] gmc_v9_0_sw_fini+0x40/0x50 [amdgpu]
[ 48.420947] amdgpu_device_fini+0x36f/0x4c0 [amdgpu]
[ 48.421007] amdgpu_driver_unload_kms+0xb4/0x150 [amdgpu]
[ 48.421058] drm_dev_unregister+0x46/0xf0 [drm]
[ 48.421102] drm_dev_unplug+0x12/0x70 [drm]

Signed-off-by: Trigger Huang <Trigger.Huang@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: initialize globals during device init (v2)

Make sure that the global BO state is always correctly initialized.

This allows removing all the device code to initialize it.

v2: fix up vbox (Alex)

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Junwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: use a static ttm_bo_global instance

As the name says we only need one global instance of ttm_bo_global.

Just use a single exported instance which is save to initialize multiple times.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Junwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: make the device list mutex static

This way it can protect the whole BO global state.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Junwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: use a static ttm_mem_global instance

As the name says we only need one global instance of ttm_mem_global.

Drop all the driver initialization and just use a single exported
instance which is initialized during BO global initialization.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Junwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Provide ttm_bo_global_{init/release}() for struct ttm_bo_global

So far, struct ttm_bo_global_ref was the only way of initializing a struct
ttm_bo_global. Providing separate initializer and release functions for
struct ttm_bo_global gives drivers the option of implementing their own
init and release callbacks for drm_global_references of type
DRM_GLOBAL_TTM_BO.

The original functions for initializing and releasing via struct
ttm_bo_global_ref are wrappers around the new interfaces.

Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Rename ttm_bo_global_{init,release}() to ttm_bo_global_ref_{,}()

The functions ttm_bo_global_init() and ttm_bo_global_release() do not
receive an argument of type struct ttm_bo_global. Both take a struct
drm_global_reference that contains points to a struct ttm_bo_global_ref.
Renaming them reflects this.

Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

dma-buf: allow reserving more than one shared fence slot

Let's support simultaneous submissions to multiple engines.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Junwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Link: https://patchwork.kernel.org/patch/10626149/

list: introduce list_bulk_move_tail helper

Move all entries between @first and including @last before @head.

This is useful for LRU lists where a whole block of entries should be
moved to the end of the list.

Used as a band aid in TTM, but better placed in the common list headers.

Acked-by: Dave Airlie <airlied@redhat.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Reviewed-by: Junwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: once more fix ttm_bo_bulk_move_lru_tail

While cutting the lists we sometimes accidentally added a list_head from
the stack to the LRUs, effectively corrupting the list.

Remove the list cutting and use explicit list manipulation instead.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-and-Tested: Huang Rui <ray.huang@amd.com>
Tested-by: Mike Lothian <mike@fireburn.co.uk>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: fix ttm_bo_bulk_move_helper

Staring at the function for six hours, just to essentially move one line
of code. The problem was that the first list_cut_position call could result
in list2 pointing to la-la-land.

Signed-off-by: Christian König <christian.koenig@amd.com>
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Initialize local lists in ttm_bo_bulk_move_helper

The first parameter of list_cut_position() must point to an initialized
list.

Noticed thanks to KASAN pointing out something's fishy here.

Fixes: "drm/ttm: add bulk move function on LRU"
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: add bulk move function on LRU

This function allow us to bulk move a group of BOs to the tail of their LRU.
The positions of group of BOs are stored on the (first, last) bulk_move_pos
structure.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Huang Rui <ray.huang@amd.com>
Tested-by: Mike Lothian <mike@fireburn.co.uk>
Tested-by: Dieter Nützel <Dieter@nuetzel-hh.de>
Acked-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Junwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: revise ttm_bo_move_to_lru_tail to support bulk moves

When move a BO to the end of LRU, it need remember the BO positions.
Make sure all moved bo in between "first" and "last". And they will be bulk
moving together.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Huang Rui <ray.huang@amd.com>
Tested-by: Mike Lothian <mike@fireburn.co.uk>
Tested-by: Dieter Nützel <Dieter@nuetzel-hh.de>
Acked-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Junwei Zhang <Jerry.Zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Replace ttm_bo_unref() with ttm_bo_put()

A call to ttm_bo_unref() clears the supplied pointer to NULL, while
ttm_bo_put() does not. None of the converted call sites requires the
pointer to become NULL, so the respective assign operations has been
left out from the patch.

Signed-off-by: Thomas Zimmermann <contact@tzimmermann.org>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Introduce ttm_bo_get() and ttm_bo_put() for ref counting

The TTM buffer-object interface provides ttm_bo_reference() and
ttm_bo_unref() for managing reference counts. Replacing them with
ttm_bo_get() and ttm_bo_put() aligns the API with conventions used
throughout the Linux kernel.

The implementation of ttm_bo_unref() clears the supplied pointer
to NULL. This leads to workarounds where the caller saves the
pointer's value before de-referencing the BO. ttm_bo_put() does
not clear the supplied pointer.

Signed-off-by: Thomas Zimmermann <contact@tzimmermann.org>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: use swap macro in ttm_bo_handle_move_mem

Make use of the swap macro and remove unnecessary variable *tmp_mem*.
This makes the code easier to read and maintain. Also, reduces the
stack usage.

This code was detected with the help of Coccinelle.

Signed-off-by: Gustavo A. R. Silva <gustavo@embeddedor.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

Add SPDX idenitifier and clarify license

This is dual licensed under GPL-2.0 or MIT.

Signed-off-by: Dirk Hohndel (VMware) <dirk@hohndel.org>
Signed-off-by: Thomas Hellstrom <thellstrom@vmware.com>
Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: remove priority hard code when initializing ttm bo

Then priority could be set before initialization.
By default, it requires to kzalloc ttm bo. In fact, we always do so.

Signed-off-by: Junwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: David Zhou <david1.zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: add ttm_bo_pipeline_gutting

Allows us to gut a BO of it's backing store when the driver says that it
isn't needed any more.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Roger He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: move ttm_tt_create into ttm_tt.c v2

Rename ttm_bo_add_ttm to ttm_tt_create and move it into ttm_tt.c.

v2: separate the cleanup.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Roger He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: drop persistent_swap_storage from ttm_bo_init and co

Never used as parameter, the only driver actually using this is nouveau
and there it is initialized after the BO is initialized.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: drop ttm->dummy_read_page

Only used by the AGP backend and there it can be easily accessed using
ttm->bdev->glob.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: drop bo->glob

The pointer is available as bo->bdev->glob as well.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: set TTM_OPT_FLAG_FORCE_ALLOC in ttm_bo_force_list_clean

Because ttm_bo_force_list_clean() is only called on two occasions:
1. By ttm_bo_evict_mm() during suspend.
2. By ttm_bo_clean_mm() when the driver unloads.
On both cases we absolutely don't want any memory allocation failure.

Signed-off-by: Roger He <Hongbo.He@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: use bit flag to replace allow_reserved_eviction in ttm_operation_ctx

for saving memory and more bit flag can be used in future

Signed-off-by: Roger He <Hongbo.He@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Add a default BO destructor to simplify code (v2)

(v2): Remove stray ; noticed by Felix

Signed-off-by: Tom St Denis <tom.stdenis@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Fix coding style in ttm_bo.c

Correct indentation and {} brace style.

Signed-off-by: Tom St Denis <tom.stdenis@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Allow page allocations w/o triggering OOM..

This to allow drivers to choose to avoid OOM invocation and handle
page allocation failures instead.

v2:
Remove extra new lines.

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Roger He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: fix missing parameter change for ttm_bo_cleanup_refs

Missed in the patche:
dc94777 drm/ttm: enable swapout for reserved BOs during allocation.
don't unreserve the BO if it is not reserved by itself.

Signed-off-by: Roger He <Hongbo.He@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Don't unreserve swapped BOs that were previously reserved

If ttm_bo_swapout doesn't own the lock, don't release it. Someone
else probably depends on it still being locked.

Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Don't add swapped BOs to swap-LRU list

A BO that's already swapped would be added back to the swap-LRU list
for example if its validation failed under high memory pressure. This
could later lead to swapping it out again and leaking previous swap
storage.

This commit adds a condition to prevent that from happening.

v2: Check page_flags instead of swap_storage

Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/ttm: enable swapout for reserved BOs during allocation

if the bo shares same reservation object then not lock it again
at swapout time to make it possible to swap out.

v2: refine the commmit message

Reviewed-by: Thomas Hellström <thellstrom@vmware.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chuming Zhou <david1.zhou@amd.com>
Signed-off-by: Roger He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: add new function to check if bo is allowable to evict or swapout

extract a function as ttm_bo_evict_swapout_allowable since eviction and
swapout can share same logic.

v2: modify commit message and add description in the code

Reviewed-by: Thomas Hellström <thellstrom@vmware.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chuming Zhou <david1.zhou@amd.com>
Signed-off-by: Roger He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: use an operation ctx for ttm_tt_bind

forward the operation context to ttm_tt_bind as well,
and the ultimate goal is swapout enablement for reserved BOs.

v2: use common term rather than amd specific

Reviewed-by: Thomas Hellström <thellstrom@vmware.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chuming Zhou <david1.zhou@amd.com>
Signed-off-by: Roger He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: use an operation ctx for ttm_mem_global_alloc

forward the operation context to ttm_mem_global_alloc as well, and the
ultimate goal is swapout enablement for reserved BOs

Reviewed-by: Thomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Roger He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: call ttm_bo_swapout directly when ttm shrink

remove the extra indirection because we have only one implementation anyway

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Roger He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: drop the spin in delayed delete if the trylock doesn't work

Thomas actually noticed that, but I didn't realized what he meant until
now.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Roger He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: cleanup some old defines

Use pr_debug instead of TTM_DEBUG, fix the lockdep assert and remove the
unused constant.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Roger He <Hongbo.He@amd.com>
Reviewed-by: Thomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: cleanup some more resv->lock uses

Use the reservation wrapper for this.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Roger He <Hongbo.He@amd.com>
Reviewed-by: Thomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: use try_lock in ttm_bo_delayed_delete again

We only need to wait for the contended lock when the reservation object is
shared or when we want to remove everything. A trylock should be sufficient
in all other cases.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Roger He <Hongbo.He@amd.com>
Reviewed-by: Thomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: use an ttm operation ctx for ttm_bo_move_xxx

include ttm_bo_move_memcpy and ttm_bo_move_ttm

Signed-off-by: Roger He <Hongbo.He@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: enable eviction for Per-VM-BO

allow eviction of BOs reserved by the caller when they are
not part of the current working set.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Roger He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: init locked again to prevent incorrect unlock

Signed-off-by: Roger He <Hongbo.He@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: add number of bytes moved to the operation context

Add some statistics how many bytes we have moved.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Tested-by: Dieter Nützel <Dieter@nuetzel-hh.de>
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: add context to driver move callback as well

Instead of passing the parameters manually.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Tested-by: Dieter Nützel <Dieter@nuetzel-hh.de>
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: use the operation context inside TTM

Instead of passing down the parameters manually to every function.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Tested-by: Dieter Nützel <Dieter@nuetzel-hh.de>
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: use an operation context for ttm_bo_mem_space v2

Instead of specifying interruptible and no_wait_gpu manually.

v2: rebase

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Tested-by: Dieter Nützel <Dieter@nuetzel-hh.de>
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: use an operation ctx for ttm_bo_init_reserved

Instead of specifying if sleeping should be interruptible.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Tested-by: Dieter Nützel <Dieter@nuetzel-hh.de>
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: add operation ctx to ttm_bo_validate v2

Give moving a BO into place an operation context to work with.

v2: rebased

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Tested-by: Dieter Nützel <Dieter@nuetzel-hh.de>
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: remove cur_placement

Not used any more.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: completely rework ttm_bo_delayed_delete

There is no guarantee that the next entry on the ddelete list stays on
the list when we drop the locks.

Completely rework this mess by moving processed entries on a temporary
list.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-and-Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: fix ttm_mem_evict_first once more

The code path isn't hit at the moment, but we need to take the lock to
add the BO back to the LRU.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-and-Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Roger He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: optimize ttm_mem_evict_first v5

Deleted BOs with the same reservation object can be reaped even if they
can't be reserved.

v2: rebase and we still need to remove/add the BO from/to the LRU.
v3: fix remove/add one more time, cleanup the logic a bit
v4: we should still check if the eviction is valuable
v5: add comment suggested by Michel

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-and-Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: make unlocking in ttm_bo_cleanup_refs optional v3

Needed for the next patch.

v2: actually predicate all unlocks
v3: add some cleanups suggested by Michel.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-and-Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: user reservation object wrappers v2

Consistently use the reservation object wrappers instead of accessing
the ww_mutex directly.

Additional to that use the reservation object wrappers directly instead of
calling __ttm_bo_reserve with fixed parameters.

v2: fix typo

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-and-Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: consistently use reservation_object_unlock

Instead of having a confusing wrapper or call the underlying ww_mutex
function directly.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-and-Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: move unlocking out of ttm_bo_cleanup_memtype_use

Needed for the next patch and makes the code quite a bit easier to
understand.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-and-Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Always and only destroy bo->ttm_resv in ttm_bo_release_list

Fixes a use-after-free due to a race condition in
ttm_bo_cleanup_refs_and_unlock, which allows one task to reserve a BO
and destroy its ttm_resv while another task is waiting for it to signal
in reservation_object_wait_timeout_rcu.

v2:
* Always initialize bo->ttm_resv in ttm_bo_init_reserved
(Christian König)

Fixes: 0d2bd2ae045d "drm/ttm: fix memory leak while individualizing BOs"
Reviewed-by: Chunming Zhou <david1.zhou@amd.com> # v1
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm:fix memory leak due to individualize

after individualize we need manually call reservation_object_fini()
if all fences on resv signaled during test, otherwise kmemory leak

Signed-off-by: Monk Liu <Monk.Liu@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: fix memory leak while individualizing BOs

We need to free the reservation object before we take the BO
from the delayed delete list.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Monk Liu <monk.liu@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: fix ttm_bo_cleanup_refs_or_queue once more

With shared reservation objects __ttm_bo_reserve() can easily fail even on
destroyed BOs. This prevents correct handling when we need to individualize
the reservation object.

Fix this by individualizing the object before even trying to reserve it.

Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Chunming Zhou <david1.zhou@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm:fix wrong decoding of bo_count

we observe abnormal number from:
/sys/devices/virtual/drm/amdttm/buffer_objects/bo_count

bo_count is atomic_inc which is "int" type,
shouldn't explicitly turn it to unsigned long.

Signed-off-by: Monk Liu <monk.liu@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm:fix wrong decoding of bo_count

we observe abnormal number from:
/sys/devices/virtual/drm/amdttm/buffer_objects/bo_count

bo_count is atomic_inc which is "int" type,
shouldn't explicitly turn it to unsigned long.

Signed-off-by: Monk Liu <monk.liu@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: use reservation_object_trylock in ttm_bo_individualize_resv v2

Fixes a false positive from might_sleep(). The reservation object is freshly
initialized, so nobody else can hold the mutex but the function is
called from atomic context.

v2: Correctly invert the check as well.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: make ttm_mem_type_manager_func debug more useful

Provide the drm printer directly instead of just the callback.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: individualize BO reservation obj when they are freed

Use the BOs reservation object when it is put on the ddelete list. This way we
avoid delaying freeing up the BO because of new fences on the reservation object.

This is used by dma-buf and amdgpu's VM page tables.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: remove nonsense wait in ttm_bo_cleanup_refs_and_unlock

With shared reservation objects the assumption that no fence
could have been added isn't true any more.

Additional to that the BO is about to be destroyed, so removing the
fences now has no advantage whatsoever.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Fix use-after-free in ttm_bo_clean_mm

We unref the man->move fence in ttm_bo_clean_mm() and then call
ttm_bo_force_list_clean() which waits on it, except the refcount is now
zero so a warning is generated (or worse):

[149492.279301] refcount_t: increment on 0; use-after-free.
[149492.279309] ------------[ cut here ]------------
[149492.279315] WARNING: CPU: 3 PID: 18726 at lib/refcount.c:150 refcount_inc+0x2b/0x30
[149492.279315] Modules linked in: vhost_net vhost tun x86_pkg_temp_thermal crc32_pclmul ghash_clmulni_intel efivarfs amdgpu(
-) i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm
[149492.279326] CPU: 3 PID: 18726 Comm: rmmod Not tainted 4.12.0-rc5-drm-next-4.13-ttmpatch+ #1
[149492.279326] Hardware name: Gigabyte Technology Co., Ltd. Z97X-UD3H-BK/Z97X-UD3H-BK-CF, BIOS F6 06/17/2014
[149492.279327] task: ffff8804ddfedcc0 task.stack: ffffc90008d20000
[149492.279329] RIP: 0010:refcount_inc+0x2b/0x30
[149492.279330] RSP: 0018:ffffc90008d23c30 EFLAGS: 00010286
[149492.279331] RAX: 000000000000002b RBX: 0000000000000170 RCX: 0000000000000000
[149492.279331] RDX: 0000000000000000 RSI: ffff88051ecccbe8 RDI: ffff88051ecccbe8
[149492.279332] RBP: ffffc90008d23c30 R08: 0000000000000001 R09: 00000000000003ee
[149492.279333] R10: ffffc90008d23bb0 R11: 00000000000003ee R12: ffff88043aaac960
[149492.279333] R13: ffff8805005e28a8 R14: 0000000000000002 R15: ffff88050115e178
[149492.279334] FS: 00007fc540168700(0000) GS:ffff88051ecc0000(0000) knlGS:0000000000000000
[149492.279335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[149492.279336] CR2: 00007fc3e8654140 CR3: 000000027ba77000 CR4: 00000000001426e0
[149492.279337] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[149492.279337] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[149492.279338] Call Trace:
[149492.279345] ttm_bo_force_list_clean+0xb9/0x110 [ttm]
[149492.279348] ttm_bo_clean_mm+0x7a/0xe0 [ttm]
[149492.279375] amdgpu_ttm_fini+0xc9/0x1f0 [amdgpu]
[149492.279392] amdgpu_bo_fini+0x12/0x40 [amdgpu]
[149492.279415] gmc_v7_0_sw_fini+0x32/0x40 [amdgpu]
[149492.279430] amdgpu_fini+0x2c9/0x490 [amdgpu]
[149492.279445] amdgpu_device_fini+0x58/0x1b0 [amdgpu]
[149492.279461] amdgpu_driver_unload_kms+0x4f/0xa0 [amdgpu]
[149492.279470] drm_dev_unregister+0x3c/0xe0 [drm]
[149492.279485] amdgpu_pci_remove+0x19/0x30 [amdgpu]
[149492.279487] pci_device_remove+0x39/0xc0
[149492.279490] device_release_driver_internal+0x155/0x210
[149492.279491] driver_detach+0x38/0x70
[149492.279493] bus_remove_driver+0x4c/0xa0
[149492.279494] driver_unregister+0x2c/0x40
[149492.279496] pci_unregister_driver+0x21/0x90
[149492.279520] amdgpu_exit+0x15/0x406 [amdgpu]
[149492.279523] SyS_delete_module+0x1a8/0x270
[149492.279525] ? exit_to_usermode_loop+0x92/0xa0
[149492.279528] entry_SYSCALL_64_fastpath+0x13/0x94
[149492.279529] RIP: 0033:0x7fc53fcb68e7
[149492.279529] RSP: 002b:00007ffcfbfaabb8 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
[149492.279531] RAX: ffffffffffffffda RBX: 0000563117adb200 RCX: 00007fc53fcb68e7
[149492.279531] RDX: 000000000000000a RSI: 0000000000000800 RDI: 0000563117adb268
[149492.279532] RBP: 0000000000000003 R08: 0000000000000000 R09: 1999999999999999
[149492.279533] R10: 0000000000000883 R11: 0000000000000206 R12: 00007ffcfbfa9ba0
[149492.279533] R13: 0000000000000000 R14: 0000000000000000 R15: 0000563117adb200
[149492.279534] Code: 55 48 89 e5 e8 77 fe ff ff 84 c0 74 02 5d c3 80 3d 40 f2 a4 00 00 75 f5 48 c7 c7 20 3c ca 81 c6 05 30 f2 a4 00 01 e8 91 f0 d7 ff <0f> ff 5d c3 90 55 48 89 fe bf 01 00 00 00 48 89 e5 e8 9f fe ff
[149492.279557] ---[ end trace 2d4e0ffcb66a1016 ]---

Unref the fence *after* waiting for it.

v2: Set man->move to NULL after dropping the last ref (Christian König)

Fixes: aff98ba1fdb8 (drm/ttm: wait for eviction in ttm_bo_force_list_clean)
Signed-off-by: John Brooks <john@fastquake.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/ttm: cleanup unuse ret value

The ret must be 0 here, otherwise, the function will return after init_mem_type.

Signed-off-by: Huang Rui <ray.huang@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: add TTM_PL_FLAG_CONTIGUOUS v2

This allows drivers to specify if they need a contiguous allocation or not.

v2: use space instead of tab

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: cleanup and optimize ttm_bo_mem_compat v2

No need to implement the same logic twice. Also check if the busy placements
are identical to the already scanned placements before checking them.

v2: improve check even more as suggested by Michel.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: add ttm_bo_init_reserved

This variant of ttm_bo_init returns the validated buffer object with
the reservation lock held when resv == NULL. This is convenient for
callers that want to use the BO immediately, e.g. for initializing its
contents.

Signed-off-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: never add BO that failed to validate to the LRU list

Fixes a potential race condition in amdgpu that looks as follows:

Task 1: attempt ttm_bo_init, but ttm_bo_validate fails
Task 1: add BO to global list anyway
Task 2: grabs hold of the BO, waits on its reservation lock
Task 1: releases its reference of the BO; never gives up the
reservation lock

The patch "drm/amdgpu: fix a potential deadlock in
amdgpu_bo_create_restricted()" attempts to fix that by releasing
the reservation lock in amdgpu code; unfortunately, it introduces
a use-after-free when this race _doesn't_ happen.

This patch should fix the race properly by never adding the BO
to the global list in the first place.

Cc: zhoucm1 <david1.zhou@amd.com>
Signed-off-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
Tested-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

gpu: drm: drivers: Convert printk(KERN_<LEVEL> to pr_<level>

Use a more common logging style.

Miscellanea:

o Coalesce formats and realign arguments
o Neaten a few macros now using pr_<level>

Signed-off-by: Joe Perches <joe@perches.com>
Acked-by: Rob Clark <robdclark@gmail.com>
Acked-by: Sinclair Yeh <syeh@vmware.com>
Acked-by: Patrik Jakobsson <patrik.r.jakobsson@gmail.com>
Acked-by: Tomi Valkeinen <tomi.valkeinen@ti.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: http://patchwork.freedesktop.org/patch/msgid/76355db47b31668bb64d996865ceee53bd66b11f.1488285953.git.joe@perches.com

drm/ttm: revert "add optional LRU removal callback v2"

Without the custom LRU management the callback is not used any more.

agd: fix trivial warning

Reviewed-by: Sinclair Yeh <syeh@vmware.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-and-Tested-by: Roger.He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: revert "implement LRU add callbacks v2"

The additional housekeeping had too much CPU overhead,
let's use the BO priorities instead.

agd: also revert hibmc changes

Reviewed-by: Sinclair Yeh <syeh@vmware.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-and-Tested-by: Roger.He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: add BO priorities for the LRUs

This way the driver can specify a priority for a BO which has the effect that
a BO is only evicted when all other BOs with a lower priority are evicted
first.

Reviewed-by: Sinclair Yeh <syeh@vmware.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Roger.He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: remove allow_errors parameter from ttm_bo_force_list_clean

Not allowing errors here is completely pointless and actually dangerous
cause trying to continue on an error can cause an endless loop.

Reviewed-by: Sinclair Yeh <syeh@vmware.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Roger.He <Hongbo.He@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: add evict parameter to ttm_bo_driver::move_notify

Ensure that the driver can listen to evictions even when they don't take the
path through ttm_bo_driver::move.

This is crucial for amdgpu, which relies on an eviction counter to skip
re-binding page tables when possible.

Signed-off-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Make sure BOs being swapped out are cacheable

The current caching state may not be tt_cached, even though the
placement contains TTM_PL_FLAG_CACHED, because placement can contain
multiple caching flags. Trying to swap out such a BO would trip up the

BUG_ON(ttm->caching_state != tt_cached);

in ttm_tt_swapout.

Cc: stable@vger.kernel.org
Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Thomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: Christian König <christian.koenig@amd.com>.
Reviewed-by: Sinclair Yeh <syeh@vmware.com>
Signed-off-by: Christian König <christian.koenig@amd.com>

locking/atomic, kref: Kill kref_sub()

By general sentiment kref_sub() is a bad interface, make it go away.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>

locking/atomic, kref: Add kref_read()

Since we need to change the implementation, stop exposing internals.

Provide kref_read() to read the current reference count; typically
used for debug messages.

Kills two anti-patterns:

atomic_read(&kref->refcount)
kref->refcount.counter

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>

drm/ttm: fix ttm_bo_wait

reservation_object_wait_timeout_rcu() should enable signaling even with a
zero timeout, but ttm_bo_wait() can also be called from atomic context and
then it is not a good idea to do this.

Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Gustavo Padovan <gustavo.padovan@collabora.co.uk>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Sumit Semwal <sumit.semwal@linaro.org>
[sumits: fix checkpatch warnings]
Link: http://patchwork.freedesktop.org/patch/msgid/1478553376-18575-3-git-send-email-alexander.deucher@amd.com

drm/ttm: make eviction decision a driver callback v2

This way the driver can decide if it is valuable to evict a BO or not.

The current implementation is added as default to all existing drivers.

v2: fix some typos found during internal testing

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

dma-buf: Rename struct fence to dma_fence

I plan to usurp the short name of struct fence for a core kernel struct,
and so I need to rename the specialised fence/timeline for DMA
operations to make room.

A consensus was reached in
https://lists.freedesktop.org/archives/dri-devel/2016-July/113083.html
that making clear this fence applies to DMA operations was a good thing.
Since then the patch has grown a bit as usage increases, so hopefully it
remains a good thing!

(v2...: rebase, rerun spatch)
v3: Compile on msm, spotted a manual fixup that I broke.
v4: Try again for msm, sorry Daniel

coccinelle script:
@@

@@
- struct fence
+ struct dma_fence
@@

@@
- struct fence_ops
+ struct dma_fence_ops
@@

@@
- struct fence_cb
+ struct dma_fence_cb
@@

@@
- struct fence_array
+ struct dma_fence_array
@@

@@
- enum fence_flag_bits
+ enum dma_fence_flag_bits
@@

@@
(
- fence_init
+ dma_fence_init
|
- fence_release
+ dma_fence_release
|
- fence_free
+ dma_fence_free
|
- fence_get
+ dma_fence_get
|
- fence_get_rcu
+ dma_fence_get_rcu
|
- fence_put
+ dma_fence_put
|
- fence_signal
+ dma_fence_signal
|
- fence_signal_locked
+ dma_fence_signal_locked
|
- fence_default_wait
+ dma_fence_default_wait
|
- fence_add_callback
+ dma_fence_add_callback
|
- fence_remove_callback
+ dma_fence_remove_callback
|
- fence_enable_sw_signaling
+ dma_fence_enable_sw_signaling
|
- fence_is_signaled_locked
+ dma_fence_is_signaled_locked
|
- fence_is_signaled
+ dma_fence_is_signaled
|
- fence_is_later
+ dma_fence_is_later
|
- fence_later
+ dma_fence_later
|
- fence_wait_timeout
+ dma_fence_wait_timeout
|
- fence_wait_any_timeout
+ dma_fence_wait_any_timeout
|
- fence_wait
+ dma_fence_wait
|
- fence_context_alloc
+ dma_fence_context_alloc
|
- fence_array_create
+ dma_fence_array_create
|
- to_fence_array
+ to_dma_fence_array
|
- fence_is_array
+ dma_fence_is_array
|
- trace_fence_emit
+ trace_dma_fence_emit
|
- FENCE_TRACE
+ DMA_FENCE_TRACE
|
- FENCE_WARN
+ DMA_FENCE_WARN
|
- FENCE_ERR
+ DMA_FENCE_ERR
)
(
...
)

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Gustavo Padovan <gustavo.padovan@collabora.co.uk>
Acked-by: Sumit Semwal <sumit.semwal@linaro.org>
Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: http://patchwork.freedesktop.org/patch/msgid/20161025120045.28839-1-chris@chris-wilson.co.uk

drm/ttm: use ffs in ttm_mem_type_from_place

A bit pointless to search for the first bit set manually.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Remove unused parameter evict from ttm_bo_move_memcpy

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Remove unused parameter evict from ttm_bo_move_ttm

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: Wait for a BO to become idle before unbinding it from GTT

Fixes hangs under memory pressure, e.g. running the piglit test
tex3d-maxsize concurrently with other tests.

Fixes: 17d33bc9d6ef ("drm/ttm: drop waiting for idle in ttm_bo_evict.")
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>