Avoid segments greater than maxsegsz during map merging

Found with hints from kettenis@

Tested on Sun-Fire-V215.
Also tested on U2 and UltraBook IIe by miod@

ok miod@

panic does not require a \n at the end. When one is provided, it looks wrong.

spelling

Unbreak DEBUG build: use %x not %lx with int

Unbreak DEBUG build: do not use undefined pa

introduced iommu_dvmamap_insert() using an undefined paddr_t pa under DEBUG;
since it cannot be introduced it (easily), remove it from the printf().

Unbreak DEBUG build: Format bus_{addr,size}_t as long

Unbreak DEBUG build: format time_t as long long

Feedback guenther

add support for bypassing iommu translation

managing the translation table entries (TTEs) on an iommu is not
free, and is in fact extremely expensive on some platforms. the
flip side of this is that forcing dma through TTEs does provide
some safety and can help during the development of drivers. however,
this has been less true in recent years than it used to be and is
less of a concern now, especially considering the performance
differences on some platforms.

devices have to create dmamaps with BUS_DMA_64BIT to bypass the
iommu because the memory window presented to hardware with direct
access to memory is at an extremely high address. there's no 32bit
bypass access to memory, it has to go through TTEs otherwise.

on an m4000 there are several orders of magnitude performance
difference between a driver with BUS_DMA_64BIT set and one without
it.

hilariously, sun used a pci bridge on a whole generation of machines
that had broken support for dma addresses over 40 bits (or around
there), so devices behind those pci bridges need to have their
dmamap_creates intercepted and any potential BUS_DMA_64BIT flags
cleared on the way to the iommu drivers. this affects at least v215,
v245, and v445, and probably u25s and u45s. it probably explains
why all their onboard nics and disk controllers feel super slow,
and why there was a meme at sun that bcopy was cheaper than dma
when moving packets on and off a nic.

ok kettenis@ deraadt@

tweak sparc64 membars as a step toward making them usable in userland.

specifically, dont rely on magic in ctlreg to implement membars. moving
that to atomic.h would add a lot of pollution to the namespace, so
move to passing the membar options to a single __membar macro.

this tweaks everything that was using the ctlreg backend to either use
an appropriate membar_foo(), or to use __membar() in the MD code.

ok kettenis@

Rename Debugger() into db_enter().

Using a name with the 'db_' prefix makes it invisible from the dynamic
profiler.

ok deraadt@, kettenis@, visa@

Some hardware (such as the onboard dc(4) of the Netra X1) has a broken DMA
engine that might attempt to read beyond the end of the buffer that was
programmed. The IOMMU catches this "DMA overrun" and throws an unrecoverable
error at us, at which point we have no choice but to panic. To avoid this
implement a BUS_DMA_OVERRUN flag that maps an additional scratch page at the
end of the vdma address range. DMA requests will spill over into this page,
which just returns zeroes.

Thanks to matthieu@ for giving me access to a machine with the problem.

ok deraadt@, beck@

Fix loading memory allocated with bus_dmamem_alloc(9). The old could would
always load all allocated pages instead of the size specified in the
bus_dmamap_load_raw(9) call. Also fixes the corner case where a specified
boundary is less than the page size, which would always create multiple
segments, even if the specified size was smaller than the boundary.

Fixes xhci(4) on sparc64.

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

uvm_km_valloc -> km_alloc

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

Format string fixes for bus_addr_t and bus_size_t, which are u_long everywhere.

Use extent_alloc_subregion_with_descr(9). This should make bus_dmamap_load(9)
and bus_dmamap_unload(9) "mpsafe".

Fix segmentation of buffers that straddle a boundary.

ok dlg@

Properly initialize the IOMMU control and status register for pyro(4).
Fixes DMA problems spotted on the v445.

When the sparc64 iommu fails to lookup a vaddr, it prints a message and
returns EFBIG. This cannot be recovered from and is usually indicative
of a damaged pmap. Therefor, panic right here instead.

This diff meant the difference between dropping into single-user mode versus
getting a traceable panic.

ok kettenis@

Do not use NULL in integer comparisons. No functional change.
ok matthew@ tedu@, also eyeballed by at least krw@ oga@ kettenis@ jsg@

cleanup more confusion regarding user.h before proc.h, or missing proc.h
ok tedu

if extent_alloc() fails, we don't clear the iomap properly before
returning an error. so next time we mess around, we may get annoying
printfs.

Fix this.

ok kettenis@

type pedantry.

the type we bind to an iommu or a GART is paddr_t, by definition, on the
other hand, the type we get out of it is not a vaddr_t, it's bus_addr_t.

fix up sparc64 iommu, amd64 iommu and the sg_dma backedn that uses it to
realise this.

ok kettenis@

Avoid clobbering error return values with the result from extent_free(), by
simply calling iommu_dvma_unload(). Solution suggested by oga@.

While there, also unwrap a line that isn't long enough to need wrapping.

Avoid clobbering error return values with the result from extent_free(), by
simply calling iommu_dvma_unload(). Solution suggested by oga@.

Convert the waitok field of uvm_pglistalloc to "flags", more will be added soon.

For the possibility of sleeping, the first two flags are UVM_PLA_WAITOK
and UVM_PLA_NOWAIT. It is an error not to show intention, so assert that
one of the two is provided. Switch over every caller in the tree to
using the appropriate flag.

ok art@, ariane@

In the rare case where after we've loaded the iomap into the hardware,
if we fail while assembling the dmamap due to the memory not fitting
into our constraints we'll return from the function with the iomap still
loaded, and more importantly with memory still allocated from the
extent(9). So in such a case, make sure we clean up after outselves.

In order to make this cleaner, remove an impossible condition check
(kettenis and myself are satisfied that it will never happen), and make
iomap_load_map void (it can't fail), so that we can only fail after both
the extent is allocated and the iomap is loaded, and not inbetween the
two.

I tested iommu, kettenis tested viommu.

ok kettenis@.

Trivial malloc + memset -> malloc(,,M_ZERO) conversion.

As a small bonus this now actually zeroes the whole struct, not just the
non-varying sized part.

pmap = (conditional ? user_way : pmap = pmap_kernel());

doesn't need the second pmap =. "ok if you tested it" kettenis@

Make IOMMU code use the generic _bus_dmamem_map() and _bus_dmamem_unmap().

Fix dmesg ugliness caused by not printing iotdb stuff.

Only print iotdb stuff when DEBUG.

Perform IOMMU cache flushes on Oberon.

move some iommu_dvmamap_validate_map()'s into DEBUG instead of DIAGNOSTIC, to
slightly improve performance
ok kettenis

use mutexes to protect the iommu's extent map and the streaming buffer,
so that busdma is most likely MP_SAFE now. (while there, fix an extent
map race... the ranges were selected outside splhigh)
ok kettenis

use the right capitalization for `SBus'

ok jmc@

Correctly set the TSB size on pyro(4).

Return EFBIG in case not enough segments are present to load a DMA request
instead of ENOMEM and remove/shortcut the additional debug printfs.
Some network drivers will try to load the mbuf chain and linearize the mbufs
if EFBIG is returned.
OK miod@ kettenis@

Deal with zero wraparound in DVMA range checks;
Fix default DVMA range for Schizo's with no "virtual-dma" property;
Add TSB size indicator to Schizo & Psycho debugging messages.

ok jason@

Standardize on EFBIG in bus_dmamap_load* if caller is too greedy; I had
done this 4.5 years ago already but regressions happened; reminded by
a similar commit in NetBSD (from mrg@); ok damien@ deraadt@ jason@

Allocate as many iommu page lists as necessary to match the size request
in bus_dmamap_create() again, but this time add a few extra pages to cope
with fragmented data (such as mbufs). Tested by many.

get rid of an unused variable.

ok deraadt@

clean dmesg output for iommu more; tested miod dlg

cleanup dmesg logging for iommu goo, make it follow the standard form

revert previous commit and unbreak sparc64.

okay deraadt

allow for dma maps larger than 64 pages; jason@ ok

Use list and queue macros where applicable to make the code easier to read;
no functional change.

Off-by-ones, requests for (x * PAGE_SIZE) + 1 bytes would not allocate the
last page. Found the hard way by chris@ and claudio@.

ok jason@ deraadt@

Pass -Wformat

Typos

It is important not to forget "membar(MemIssue)". Got missed
when extracting the minimal diff for the previous iommu change.

ok jason@

Much of the sbus, psycho, and schizo bus_dma code is the same, so let's
call the real implementation functions directly instead of duplicating
lots of code that only calls the real stuff anyway.

tested by miod@ henning@
ok jason@

The "min" and "max" functions cast to "u_int", which can lead to surprising
results. Use the MIN/MAX macros instead.

tested by miod@
ok jason@ millert@

There's an off-by-one in the diagnostic code that can cause grief,
especially for NIC drivers that aren't careful about handling load
failures.

Bug reported and diff tested by Holger Burde.

ok jason@.

The existing IOMMU code had a rounding problem that was most noticeable
on faster systems under heavy network load. This replaces some of the
unreadable iommu functions with something a little less dense and a lot
less crash prone.

The bus_dma function pointer/cookie handling was broken. Change them
to work like the stacked bus_space drivers (where "work" is the key
word).

Tested my many (thanks).

ok jason@ deraadt@

don't use home grown timeval comparison when timercmp() is available; pointed out by henric

Print the pa not ldxa(pa)

- kill several unnecessary static's
- fix debugging code so it doesn't generate crashes itself

Add support for the Sun Enterprise 450
Reduce the size of a GENERIC kernel by ~190k
Remove the nasty pointer/bus_space_handle_t casts
Adds debug bus_space code including the ability to trace
bus operations (it actually works now).

The following rules are now followed (and verfified by the debug
code):

1. A "bus_space_handle_t" may only be used with the
"bus_space_tag_t" that created it.
2. Only "bus_space_map()" may create "bus_space_handle_t"s.
3. A "bus_space_handle_t" may not be modified after it has
been created (other than being destroyed by "bus_space_unmap()").


Thanks to help from mcbride, marc, jason, drahn, to anyone that might
have slipped my mind at the moment.

ok jason@, deraadt@

Remove more '\n's from panic() statements. Both trailing and leading.

Diff generated by Chris Kuethe.

this removes the functionality of adding allocated
pages into the queue already containing allocated pages.
breaks i386:setup_buffers() because of this.

No more need to initialize the result list before uvm_pglistalloc.

Aasking to load an already loaded map is not a "warn and pray" kinda thing.

From NetBSD:
-Fix some corner cases in bus_dmamap_load_mbuf().
From Takeshi Nakayama <tn@catvmics.ne.jp>
-Fix off-by-one error in iommu_dvmamap_load_raw() where if a DMA segment
has just one byte on a page the page is never mapped into the IOMMU.

No need to cast &thing to paddr_t* when thing IS a paddr_t

Deal with 24bit dvma requests

Permit compiling with DEBUG; Joey Coleman <joeycoleman@acm.org>

First round of __P removal in sys

Look up the TTE and only do the streaming cache dance if the mapping is marked as streaming.

Remove some debugging code accidentally committed in previous

From NetBSD:
Fix calculation of dma segment length when the DVMA range is crossing the boundary.

Be sure to wait the whole 0.5 seconds (half a second!) for the streaming
cache to flush, not some fraction of it, before giving up.

Debugger() is #ifdef DDB

- don't call strbuf_flush_done for each segment, call it once per map (and
then only if necessary)
- When flushing a partial segment make sure we flush all of the necessary pages
(the case where len < NBPG would not correctly flush both pages if
va + off <-> va + off + len crossed a page boundary)

From NetBSD:
- Fix pr sparc64/15633: datafault at tlp_start causes panic
- Also clean up some additional 32-bit kernel printf issues.
- Handle the case where a DMA operation wraps from the end of a page to the
beginning of the same page properly by double-mapping that page.
- Move a brace so this will compile w/o DEBUG.
- Overhaul iommu_dvmamap_load_raw().
From Me:
- Verify the streaming cache exists before waiting on it to clear during flush.
- Force sync of flush status (hardware sets it to 1 when its done)
- Force sync flush of registers (bus_space_barrier)
- Flush ALL segments on BUS_DMA_POSTREAD operations (not just the first!)

branches: 1.7.2;
Yet another sync to NetBSD uvm.
Today we add a pmap argument to pmap_update() and allocate map entries for
kernel_map from kmem_map instead of using the static entries. This should
get rid of MAX_KMAPENT panics. Also some uvm_loan problems are fixed.

Replace inclusion of <vm/foo.h> with the correct <uvm/bar.h> when necessary.
(Look ma, I might have broken the tree)

branches: 1.5.4;
Pull in several changes from NetBSD:
- Clear the dmamap inside splhigh() protection.
- Manage both streaming caches on psycho/psycho+.
And fix a botch I created when merging iommu changes last time.

Merge with NetBSD:
better handling of boundary conditions
add iommu_extract debugging function

merge vm/vm_kern.h into uvm/uvm_extern.h; art@ ok

RCSids

compiles with minimal mods (From NetBSD)

panic does not require a \n at the end. When one is provided, it looks wrong.

spelling

Unbreak DEBUG build: use %x not %lx with int

Unbreak DEBUG build: do not use undefined pa

introduced iommu_dvmamap_insert() using an undefined paddr_t pa under DEBUG;
since it cannot be introduced it (easily), remove it from the printf().

Unbreak DEBUG build: Format bus_{addr,size}_t as long

Unbreak DEBUG build: format time_t as long long

Feedback guenther

add support for bypassing iommu translation

managing the translation table entries (TTEs) on an iommu is not
free, and is in fact extremely expensive on some platforms. the
flip side of this is that forcing dma through TTEs does provide
some safety and can help during the development of drivers. however,
this has been less true in recent years than it used to be and is
less of a concern now, especially considering the performance
differences on some platforms.

devices have to create dmamaps with BUS_DMA_64BIT to bypass the
iommu because the memory window presented to hardware with direct
access to memory is at an extremely high address. there's no 32bit
bypass access to memory, it has to go through TTEs otherwise.

on an m4000 there are several orders of magnitude performance
difference between a driver with BUS_DMA_64BIT set and one without
it.

hilariously, sun used a pci bridge on a whole generation of machines
that had broken support for dma addresses over 40 bits (or around
there), so devices behind those pci bridges need to have their
dmamap_creates intercepted and any potential BUS_DMA_64BIT flags
cleared on the way to the iommu drivers. this affects at least v215,
v245, and v445, and probably u25s and u45s. it probably explains
why all their onboard nics and disk controllers feel super slow,
and why there was a meme at sun that bcopy was cheaper than dma
when moving packets on and off a nic.

ok kettenis@ deraadt@

tweak sparc64 membars as a step toward making them usable in userland.

specifically, dont rely on magic in ctlreg to implement membars. moving
that to atomic.h would add a lot of pollution to the namespace, so
move to passing the membar options to a single __membar macro.

this tweaks everything that was using the ctlreg backend to either use
an appropriate membar_foo(), or to use __membar() in the MD code.

ok kettenis@

Rename Debugger() into db_enter().

Using a name with the 'db_' prefix makes it invisible from the dynamic
profiler.

ok deraadt@, kettenis@, visa@

Some hardware (such as the onboard dc(4) of the Netra X1) has a broken DMA
engine that might attempt to read beyond the end of the buffer that was
programmed. The IOMMU catches this "DMA overrun" and throws an unrecoverable
error at us, at which point we have no choice but to panic. To avoid this
implement a BUS_DMA_OVERRUN flag that maps an additional scratch page at the
end of the vdma address range. DMA requests will spill over into this page,
which just returns zeroes.

Thanks to matthieu@ for giving me access to a machine with the problem.

ok deraadt@, beck@

Fix loading memory allocated with bus_dmamem_alloc(9). The old could would
always load all allocated pages instead of the size specified in the
bus_dmamap_load_raw(9) call. Also fixes the corner case where a specified
boundary is less than the page size, which would always create multiple
segments, even if the specified size was smaller than the boundary.

Fixes xhci(4) on sparc64.

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

uvm_km_valloc -> km_alloc

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

Format string fixes for bus_addr_t and bus_size_t, which are u_long everywhere.

Use extent_alloc_subregion_with_descr(9). This should make bus_dmamap_load(9)
and bus_dmamap_unload(9) "mpsafe".

Fix segmentation of buffers that straddle a boundary.

ok dlg@

Properly initialize the IOMMU control and status register for pyro(4).
Fixes DMA problems spotted on the v445.

When the sparc64 iommu fails to lookup a vaddr, it prints a message and
returns EFBIG. This cannot be recovered from and is usually indicative
of a damaged pmap. Therefor, panic right here instead.

This diff meant the difference between dropping into single-user mode versus
getting a traceable panic.

ok kettenis@

Do not use NULL in integer comparisons. No functional change.
ok matthew@ tedu@, also eyeballed by at least krw@ oga@ kettenis@ jsg@

cleanup more confusion regarding user.h before proc.h, or missing proc.h
ok tedu

if extent_alloc() fails, we don't clear the iomap properly before
returning an error. so next time we mess around, we may get annoying
printfs.

Fix this.

ok kettenis@

type pedantry.

the type we bind to an iommu or a GART is paddr_t, by definition, on the
other hand, the type we get out of it is not a vaddr_t, it's bus_addr_t.

fix up sparc64 iommu, amd64 iommu and the sg_dma backedn that uses it to
realise this.

ok kettenis@

Avoid clobbering error return values with the result from extent_free(), by
simply calling iommu_dvma_unload(). Solution suggested by oga@.

While there, also unwrap a line that isn't long enough to need wrapping.

Avoid clobbering error return values with the result from extent_free(), by
simply calling iommu_dvma_unload(). Solution suggested by oga@.

Convert the waitok field of uvm_pglistalloc to "flags", more will be added soon.

For the possibility of sleeping, the first two flags are UVM_PLA_WAITOK
and UVM_PLA_NOWAIT. It is an error not to show intention, so assert that
one of the two is provided. Switch over every caller in the tree to
using the appropriate flag.

ok art@, ariane@

In the rare case where after we've loaded the iomap into the hardware,
if we fail while assembling the dmamap due to the memory not fitting
into our constraints we'll return from the function with the iomap still
loaded, and more importantly with memory still allocated from the
extent(9). So in such a case, make sure we clean up after outselves.

In order to make this cleaner, remove an impossible condition check
(kettenis and myself are satisfied that it will never happen), and make
iomap_load_map void (it can't fail), so that we can only fail after both
the extent is allocated and the iomap is loaded, and not inbetween the
two.

I tested iommu, kettenis tested viommu.

ok kettenis@.

Trivial malloc + memset -> malloc(,,M_ZERO) conversion.

As a small bonus this now actually zeroes the whole struct, not just the
non-varying sized part.

pmap = (conditional ? user_way : pmap = pmap_kernel());

doesn't need the second pmap =. "ok if you tested it" kettenis@

Make IOMMU code use the generic _bus_dmamem_map() and _bus_dmamem_unmap().

Fix dmesg ugliness caused by not printing iotdb stuff.

Only print iotdb stuff when DEBUG.

Perform IOMMU cache flushes on Oberon.

move some iommu_dvmamap_validate_map()'s into DEBUG instead of DIAGNOSTIC, to
slightly improve performance
ok kettenis

use mutexes to protect the iommu's extent map and the streaming buffer,
so that busdma is most likely MP_SAFE now. (while there, fix an extent
map race... the ranges were selected outside splhigh)
ok kettenis

use the right capitalization for `SBus'

ok jmc@

Correctly set the TSB size on pyro(4).

Return EFBIG in case not enough segments are present to load a DMA request
instead of ENOMEM and remove/shortcut the additional debug printfs.
Some network drivers will try to load the mbuf chain and linearize the mbufs
if EFBIG is returned.
OK miod@ kettenis@

Deal with zero wraparound in DVMA range checks;
Fix default DVMA range for Schizo's with no "virtual-dma" property;
Add TSB size indicator to Schizo & Psycho debugging messages.

ok jason@

Standardize on EFBIG in bus_dmamap_load* if caller is too greedy; I had
done this 4.5 years ago already but regressions happened; reminded by
a similar commit in NetBSD (from mrg@); ok damien@ deraadt@ jason@

Allocate as many iommu page lists as necessary to match the size request
in bus_dmamap_create() again, but this time add a few extra pages to cope
with fragmented data (such as mbufs). Tested by many.

get rid of an unused variable.

ok deraadt@

clean dmesg output for iommu more; tested miod dlg

cleanup dmesg logging for iommu goo, make it follow the standard form

revert previous commit and unbreak sparc64.

okay deraadt

allow for dma maps larger than 64 pages; jason@ ok

Use list and queue macros where applicable to make the code easier to read;
no functional change.

Off-by-ones, requests for (x * PAGE_SIZE) + 1 bytes would not allocate the
last page. Found the hard way by chris@ and claudio@.

ok jason@ deraadt@

Pass -Wformat

Typos

It is important not to forget "membar(MemIssue)". Got missed
when extracting the minimal diff for the previous iommu change.

ok jason@

Much of the sbus, psycho, and schizo bus_dma code is the same, so let's
call the real implementation functions directly instead of duplicating
lots of code that only calls the real stuff anyway.

tested by miod@ henning@
ok jason@

The "min" and "max" functions cast to "u_int", which can lead to surprising
results. Use the MIN/MAX macros instead.

tested by miod@
ok jason@ millert@

There's an off-by-one in the diagnostic code that can cause grief,
especially for NIC drivers that aren't careful about handling load
failures.

Bug reported and diff tested by Holger Burde.

ok jason@.

The existing IOMMU code had a rounding problem that was most noticeable
on faster systems under heavy network load. This replaces some of the
unreadable iommu functions with something a little less dense and a lot
less crash prone.

The bus_dma function pointer/cookie handling was broken. Change them
to work like the stacked bus_space drivers (where "work" is the key
word).

Tested my many (thanks).

ok jason@ deraadt@

don't use home grown timeval comparison when timercmp() is available; pointed out by henric

Print the pa not ldxa(pa)

- kill several unnecessary static's
- fix debugging code so it doesn't generate crashes itself

Add support for the Sun Enterprise 450
Reduce the size of a GENERIC kernel by ~190k
Remove the nasty pointer/bus_space_handle_t casts
Adds debug bus_space code including the ability to trace
bus operations (it actually works now).

The following rules are now followed (and verfified by the debug
code):

1. A "bus_space_handle_t" may only be used with the
"bus_space_tag_t" that created it.
2. Only "bus_space_map()" may create "bus_space_handle_t"s.
3. A "bus_space_handle_t" may not be modified after it has
been created (other than being destroyed by "bus_space_unmap()").


Thanks to help from mcbride, marc, jason, drahn, to anyone that might
have slipped my mind at the moment.

ok jason@, deraadt@

Remove more '\n's from panic() statements. Both trailing and leading.

Diff generated by Chris Kuethe.

this removes the functionality of adding allocated
pages into the queue already containing allocated pages.
breaks i386:setup_buffers() because of this.

No more need to initialize the result list before uvm_pglistalloc.

Aasking to load an already loaded map is not a "warn and pray" kinda thing.

From NetBSD:
-Fix some corner cases in bus_dmamap_load_mbuf().
From Takeshi Nakayama <tn@catvmics.ne.jp>
-Fix off-by-one error in iommu_dvmamap_load_raw() where if a DMA segment
has just one byte on a page the page is never mapped into the IOMMU.

No need to cast &thing to paddr_t* when thing IS a paddr_t

Deal with 24bit dvma requests

Permit compiling with DEBUG; Joey Coleman <joeycoleman@acm.org>

First round of __P removal in sys

Look up the TTE and only do the streaming cache dance if the mapping is marked as streaming.

Remove some debugging code accidentally committed in previous

From NetBSD:
Fix calculation of dma segment length when the DVMA range is crossing the boundary.

Be sure to wait the whole 0.5 seconds (half a second!) for the streaming
cache to flush, not some fraction of it, before giving up.

Debugger() is #ifdef DDB

- don't call strbuf_flush_done for each segment, call it once per map (and
then only if necessary)
- When flushing a partial segment make sure we flush all of the necessary pages
(the case where len < NBPG would not correctly flush both pages if
va + off <-> va + off + len crossed a page boundary)

From NetBSD:
- Fix pr sparc64/15633: datafault at tlp_start causes panic
- Also clean up some additional 32-bit kernel printf issues.
- Handle the case where a DMA operation wraps from the end of a page to the
beginning of the same page properly by double-mapping that page.
- Move a brace so this will compile w/o DEBUG.
- Overhaul iommu_dvmamap_load_raw().
From Me:
- Verify the streaming cache exists before waiting on it to clear during flush.
- Force sync of flush status (hardware sets it to 1 when its done)
- Force sync flush of registers (bus_space_barrier)
- Flush ALL segments on BUS_DMA_POSTREAD operations (not just the first!)

branches: 1.7.2;
Yet another sync to NetBSD uvm.
Today we add a pmap argument to pmap_update() and allocate map entries for
kernel_map from kmem_map instead of using the static entries. This should
get rid of MAX_KMAPENT panics. Also some uvm_loan problems are fixed.

Replace inclusion of <vm/foo.h> with the correct <uvm/bar.h> when necessary.
(Look ma, I might have broken the tree)

branches: 1.5.4;
Pull in several changes from NetBSD:
- Clear the dmamap inside splhigh() protection.
- Manage both streaming caches on psycho/psycho+.
And fix a botch I created when merging iommu changes last time.

Merge with NetBSD:
better handling of boundary conditions
add iommu_extract debugging function

merge vm/vm_kern.h into uvm/uvm_extern.h; art@ ok

RCSids

compiles with minimal mods (From NetBSD)

spelling

Unbreak DEBUG build: use %x not %lx with int

Unbreak DEBUG build: do not use undefined pa

introduced iommu_dvmamap_insert() using an undefined paddr_t pa under DEBUG;
since it cannot be introduced it (easily), remove it from the printf().

Unbreak DEBUG build: Format bus_{addr,size}_t as long

Unbreak DEBUG build: format time_t as long long

Feedback guenther

add support for bypassing iommu translation

managing the translation table entries (TTEs) on an iommu is not
free, and is in fact extremely expensive on some platforms. the
flip side of this is that forcing dma through TTEs does provide
some safety and can help during the development of drivers. however,
this has been less true in recent years than it used to be and is
less of a concern now, especially considering the performance
differences on some platforms.

devices have to create dmamaps with BUS_DMA_64BIT to bypass the
iommu because the memory window presented to hardware with direct
access to memory is at an extremely high address. there's no 32bit
bypass access to memory, it has to go through TTEs otherwise.

on an m4000 there are several orders of magnitude performance
difference between a driver with BUS_DMA_64BIT set and one without
it.

hilariously, sun used a pci bridge on a whole generation of machines
that had broken support for dma addresses over 40 bits (or around
there), so devices behind those pci bridges need to have their
dmamap_creates intercepted and any potential BUS_DMA_64BIT flags
cleared on the way to the iommu drivers. this affects at least v215,
v245, and v445, and probably u25s and u45s. it probably explains
why all their onboard nics and disk controllers feel super slow,
and why there was a meme at sun that bcopy was cheaper than dma
when moving packets on and off a nic.

ok kettenis@ deraadt@

tweak sparc64 membars as a step toward making them usable in userland.

specifically, dont rely on magic in ctlreg to implement membars. moving
that to atomic.h would add a lot of pollution to the namespace, so
move to passing the membar options to a single __membar macro.

this tweaks everything that was using the ctlreg backend to either use
an appropriate membar_foo(), or to use __membar() in the MD code.

ok kettenis@

Rename Debugger() into db_enter().

Using a name with the 'db_' prefix makes it invisible from the dynamic
profiler.

ok deraadt@, kettenis@, visa@

Some hardware (such as the onboard dc(4) of the Netra X1) has a broken DMA
engine that might attempt to read beyond the end of the buffer that was
programmed. The IOMMU catches this "DMA overrun" and throws an unrecoverable
error at us, at which point we have no choice but to panic. To avoid this
implement a BUS_DMA_OVERRUN flag that maps an additional scratch page at the
end of the vdma address range. DMA requests will spill over into this page,
which just returns zeroes.

Thanks to matthieu@ for giving me access to a machine with the problem.

ok deraadt@, beck@

Fix loading memory allocated with bus_dmamem_alloc(9). The old could would
always load all allocated pages instead of the size specified in the
bus_dmamap_load_raw(9) call. Also fixes the corner case where a specified
boundary is less than the page size, which would always create multiple
segments, even if the specified size was smaller than the boundary.

Fixes xhci(4) on sparc64.

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

uvm_km_valloc -> km_alloc

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

Format string fixes for bus_addr_t and bus_size_t, which are u_long everywhere.

Use extent_alloc_subregion_with_descr(9). This should make bus_dmamap_load(9)
and bus_dmamap_unload(9) "mpsafe".

Fix segmentation of buffers that straddle a boundary.

ok dlg@

Properly initialize the IOMMU control and status register for pyro(4).
Fixes DMA problems spotted on the v445.

When the sparc64 iommu fails to lookup a vaddr, it prints a message and
returns EFBIG. This cannot be recovered from and is usually indicative
of a damaged pmap. Therefor, panic right here instead.

This diff meant the difference between dropping into single-user mode versus
getting a traceable panic.

ok kettenis@

Do not use NULL in integer comparisons. No functional change.
ok matthew@ tedu@, also eyeballed by at least krw@ oga@ kettenis@ jsg@

cleanup more confusion regarding user.h before proc.h, or missing proc.h
ok tedu

if extent_alloc() fails, we don't clear the iomap properly before
returning an error. so next time we mess around, we may get annoying
printfs.

Fix this.

ok kettenis@

type pedantry.

the type we bind to an iommu or a GART is paddr_t, by definition, on the
other hand, the type we get out of it is not a vaddr_t, it's bus_addr_t.

fix up sparc64 iommu, amd64 iommu and the sg_dma backedn that uses it to
realise this.

ok kettenis@

Avoid clobbering error return values with the result from extent_free(), by
simply calling iommu_dvma_unload(). Solution suggested by oga@.

While there, also unwrap a line that isn't long enough to need wrapping.

Avoid clobbering error return values with the result from extent_free(), by
simply calling iommu_dvma_unload(). Solution suggested by oga@.

Convert the waitok field of uvm_pglistalloc to "flags", more will be added soon.

For the possibility of sleeping, the first two flags are UVM_PLA_WAITOK
and UVM_PLA_NOWAIT. It is an error not to show intention, so assert that
one of the two is provided. Switch over every caller in the tree to
using the appropriate flag.

ok art@, ariane@

In the rare case where after we've loaded the iomap into the hardware,
if we fail while assembling the dmamap due to the memory not fitting
into our constraints we'll return from the function with the iomap still
loaded, and more importantly with memory still allocated from the
extent(9). So in such a case, make sure we clean up after outselves.

In order to make this cleaner, remove an impossible condition check
(kettenis and myself are satisfied that it will never happen), and make
iomap_load_map void (it can't fail), so that we can only fail after both
the extent is allocated and the iomap is loaded, and not inbetween the
two.

I tested iommu, kettenis tested viommu.

ok kettenis@.

Trivial malloc + memset -> malloc(,,M_ZERO) conversion.

As a small bonus this now actually zeroes the whole struct, not just the
non-varying sized part.

pmap = (conditional ? user_way : pmap = pmap_kernel());

doesn't need the second pmap =. "ok if you tested it" kettenis@

Make IOMMU code use the generic _bus_dmamem_map() and _bus_dmamem_unmap().

Fix dmesg ugliness caused by not printing iotdb stuff.

Only print iotdb stuff when DEBUG.

Perform IOMMU cache flushes on Oberon.

move some iommu_dvmamap_validate_map()'s into DEBUG instead of DIAGNOSTIC, to
slightly improve performance
ok kettenis

use mutexes to protect the iommu's extent map and the streaming buffer,
so that busdma is most likely MP_SAFE now. (while there, fix an extent
map race... the ranges were selected outside splhigh)
ok kettenis

use the right capitalization for `SBus'

ok jmc@

Correctly set the TSB size on pyro(4).

Return EFBIG in case not enough segments are present to load a DMA request
instead of ENOMEM and remove/shortcut the additional debug printfs.
Some network drivers will try to load the mbuf chain and linearize the mbufs
if EFBIG is returned.
OK miod@ kettenis@

Deal with zero wraparound in DVMA range checks;
Fix default DVMA range for Schizo's with no "virtual-dma" property;
Add TSB size indicator to Schizo & Psycho debugging messages.

ok jason@

Standardize on EFBIG in bus_dmamap_load* if caller is too greedy; I had
done this 4.5 years ago already but regressions happened; reminded by
a similar commit in NetBSD (from mrg@); ok damien@ deraadt@ jason@

Allocate as many iommu page lists as necessary to match the size request
in bus_dmamap_create() again, but this time add a few extra pages to cope
with fragmented data (such as mbufs). Tested by many.

get rid of an unused variable.

ok deraadt@

clean dmesg output for iommu more; tested miod dlg

cleanup dmesg logging for iommu goo, make it follow the standard form

revert previous commit and unbreak sparc64.

okay deraadt

allow for dma maps larger than 64 pages; jason@ ok

Use list and queue macros where applicable to make the code easier to read;
no functional change.

Off-by-ones, requests for (x * PAGE_SIZE) + 1 bytes would not allocate the
last page. Found the hard way by chris@ and claudio@.

ok jason@ deraadt@

Pass -Wformat

Typos

It is important not to forget "membar(MemIssue)". Got missed
when extracting the minimal diff for the previous iommu change.

ok jason@

Much of the sbus, psycho, and schizo bus_dma code is the same, so let's
call the real implementation functions directly instead of duplicating
lots of code that only calls the real stuff anyway.

tested by miod@ henning@
ok jason@

The "min" and "max" functions cast to "u_int", which can lead to surprising
results. Use the MIN/MAX macros instead.

tested by miod@
ok jason@ millert@

There's an off-by-one in the diagnostic code that can cause grief,
especially for NIC drivers that aren't careful about handling load
failures.

Bug reported and diff tested by Holger Burde.

ok jason@.

The existing IOMMU code had a rounding problem that was most noticeable
on faster systems under heavy network load. This replaces some of the
unreadable iommu functions with something a little less dense and a lot
less crash prone.

The bus_dma function pointer/cookie handling was broken. Change them
to work like the stacked bus_space drivers (where "work" is the key
word).

Tested my many (thanks).

ok jason@ deraadt@

don't use home grown timeval comparison when timercmp() is available; pointed out by henric

Print the pa not ldxa(pa)

- kill several unnecessary static's
- fix debugging code so it doesn't generate crashes itself

Add support for the Sun Enterprise 450
Reduce the size of a GENERIC kernel by ~190k
Remove the nasty pointer/bus_space_handle_t casts
Adds debug bus_space code including the ability to trace
bus operations (it actually works now).

The following rules are now followed (and verfified by the debug
code):

1. A "bus_space_handle_t" may only be used with the
"bus_space_tag_t" that created it.
2. Only "bus_space_map()" may create "bus_space_handle_t"s.
3. A "bus_space_handle_t" may not be modified after it has
been created (other than being destroyed by "bus_space_unmap()").


Thanks to help from mcbride, marc, jason, drahn, to anyone that might
have slipped my mind at the moment.

ok jason@, deraadt@

Remove more '\n's from panic() statements. Both trailing and leading.

Diff generated by Chris Kuethe.

this removes the functionality of adding allocated
pages into the queue already containing allocated pages.
breaks i386:setup_buffers() because of this.

No more need to initialize the result list before uvm_pglistalloc.

Aasking to load an already loaded map is not a "warn and pray" kinda thing.

From NetBSD:
-Fix some corner cases in bus_dmamap_load_mbuf().
From Takeshi Nakayama <tn@catvmics.ne.jp>
-Fix off-by-one error in iommu_dvmamap_load_raw() where if a DMA segment
has just one byte on a page the page is never mapped into the IOMMU.

No need to cast &thing to paddr_t* when thing IS a paddr_t

Deal with 24bit dvma requests

Permit compiling with DEBUG; Joey Coleman <joeycoleman@acm.org>

First round of __P removal in sys

Look up the TTE and only do the streaming cache dance if the mapping is marked as streaming.

Remove some debugging code accidentally committed in previous

From NetBSD:
Fix calculation of dma segment length when the DVMA range is crossing the boundary.

Be sure to wait the whole 0.5 seconds (half a second!) for the streaming
cache to flush, not some fraction of it, before giving up.

Debugger() is #ifdef DDB

- don't call strbuf_flush_done for each segment, call it once per map (and
then only if necessary)
- When flushing a partial segment make sure we flush all of the necessary pages
(the case where len < NBPG would not correctly flush both pages if
va + off <-> va + off + len crossed a page boundary)

From NetBSD:
- Fix pr sparc64/15633: datafault at tlp_start causes panic
- Also clean up some additional 32-bit kernel printf issues.
- Handle the case where a DMA operation wraps from the end of a page to the
beginning of the same page properly by double-mapping that page.
- Move a brace so this will compile w/o DEBUG.
- Overhaul iommu_dvmamap_load_raw().
From Me:
- Verify the streaming cache exists before waiting on it to clear during flush.
- Force sync of flush status (hardware sets it to 1 when its done)
- Force sync flush of registers (bus_space_barrier)
- Flush ALL segments on BUS_DMA_POSTREAD operations (not just the first!)

branches: 1.7.2;
Yet another sync to NetBSD uvm.
Today we add a pmap argument to pmap_update() and allocate map entries for
kernel_map from kmem_map instead of using the static entries. This should
get rid of MAX_KMAPENT panics. Also some uvm_loan problems are fixed.

Replace inclusion of <vm/foo.h> with the correct <uvm/bar.h> when necessary.
(Look ma, I might have broken the tree)

branches: 1.5.4;
Pull in several changes from NetBSD:
- Clear the dmamap inside splhigh() protection.
- Manage both streaming caches on psycho/psycho+.
And fix a botch I created when merging iommu changes last time.

Merge with NetBSD:
better handling of boundary conditions
add iommu_extract debugging function

merge vm/vm_kern.h into uvm/uvm_extern.h; art@ ok

RCSids

compiles with minimal mods (From NetBSD)

Unbreak DEBUG build: use %x not %lx with int

Unbreak DEBUG build: do not use undefined pa

introduced iommu_dvmamap_insert() using an undefined paddr_t pa under DEBUG;
since it cannot be introduced it (easily), remove it from the printf().

Unbreak DEBUG build: Format bus_{addr,size}_t as long

Unbreak DEBUG build: format time_t as long long

Feedback guenther

add support for bypassing iommu translation

managing the translation table entries (TTEs) on an iommu is not
free, and is in fact extremely expensive on some platforms. the
flip side of this is that forcing dma through TTEs does provide
some safety and can help during the development of drivers. however,
this has been less true in recent years than it used to be and is
less of a concern now, especially considering the performance
differences on some platforms.

devices have to create dmamaps with BUS_DMA_64BIT to bypass the
iommu because the memory window presented to hardware with direct
access to memory is at an extremely high address. there's no 32bit
bypass access to memory, it has to go through TTEs otherwise.

on an m4000 there are several orders of magnitude performance
difference between a driver with BUS_DMA_64BIT set and one without
it.

hilariously, sun used a pci bridge on a whole generation of machines
that had broken support for dma addresses over 40 bits (or around
there), so devices behind those pci bridges need to have their
dmamap_creates intercepted and any potential BUS_DMA_64BIT flags
cleared on the way to the iommu drivers. this affects at least v215,
v245, and v445, and probably u25s and u45s. it probably explains
why all their onboard nics and disk controllers feel super slow,
and why there was a meme at sun that bcopy was cheaper than dma
when moving packets on and off a nic.

ok kettenis@ deraadt@

tweak sparc64 membars as a step toward making them usable in userland.

specifically, dont rely on magic in ctlreg to implement membars. moving
that to atomic.h would add a lot of pollution to the namespace, so
move to passing the membar options to a single __membar macro.

this tweaks everything that was using the ctlreg backend to either use
an appropriate membar_foo(), or to use __membar() in the MD code.

ok kettenis@

Rename Debugger() into db_enter().

Using a name with the 'db_' prefix makes it invisible from the dynamic
profiler.

ok deraadt@, kettenis@, visa@

Some hardware (such as the onboard dc(4) of the Netra X1) has a broken DMA
engine that might attempt to read beyond the end of the buffer that was
programmed. The IOMMU catches this "DMA overrun" and throws an unrecoverable
error at us, at which point we have no choice but to panic. To avoid this
implement a BUS_DMA_OVERRUN flag that maps an additional scratch page at the
end of the vdma address range. DMA requests will spill over into this page,
which just returns zeroes.

Thanks to matthieu@ for giving me access to a machine with the problem.

ok deraadt@, beck@

Fix loading memory allocated with bus_dmamem_alloc(9). The old could would
always load all allocated pages instead of the size specified in the
bus_dmamap_load_raw(9) call. Also fixes the corner case where a specified
boundary is less than the page size, which would always create multiple
segments, even if the specified size was smaller than the boundary.

Fixes xhci(4) on sparc64.

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

uvm_km_valloc -> km_alloc

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

Format string fixes for bus_addr_t and bus_size_t, which are u_long everywhere.

Use extent_alloc_subregion_with_descr(9). This should make bus_dmamap_load(9)
and bus_dmamap_unload(9) "mpsafe".

Fix segmentation of buffers that straddle a boundary.

ok dlg@

Properly initialize the IOMMU control and status register for pyro(4).
Fixes DMA problems spotted on the v445.

When the sparc64 iommu fails to lookup a vaddr, it prints a message and
returns EFBIG. This cannot be recovered from and is usually indicative
of a damaged pmap. Therefor, panic right here instead.

This diff meant the difference between dropping into single-user mode versus
getting a traceable panic.

ok kettenis@

Do not use NULL in integer comparisons. No functional change.
ok matthew@ tedu@, also eyeballed by at least krw@ oga@ kettenis@ jsg@

cleanup more confusion regarding user.h before proc.h, or missing proc.h
ok tedu

if extent_alloc() fails, we don't clear the iomap properly before
returning an error. so next time we mess around, we may get annoying
printfs.

Fix this.

ok kettenis@

type pedantry.

the type we bind to an iommu or a GART is paddr_t, by definition, on the
other hand, the type we get out of it is not a vaddr_t, it's bus_addr_t.

fix up sparc64 iommu, amd64 iommu and the sg_dma backedn that uses it to
realise this.

ok kettenis@

Avoid clobbering error return values with the result from extent_free(), by
simply calling iommu_dvma_unload(). Solution suggested by oga@.

While there, also unwrap a line that isn't long enough to need wrapping.

Avoid clobbering error return values with the result from extent_free(), by
simply calling iommu_dvma_unload(). Solution suggested by oga@.

Convert the waitok field of uvm_pglistalloc to "flags", more will be added soon.

For the possibility of sleeping, the first two flags are UVM_PLA_WAITOK
and UVM_PLA_NOWAIT. It is an error not to show intention, so assert that
one of the two is provided. Switch over every caller in the tree to
using the appropriate flag.

ok art@, ariane@

In the rare case where after we've loaded the iomap into the hardware,
if we fail while assembling the dmamap due to the memory not fitting
into our constraints we'll return from the function with the iomap still
loaded, and more importantly with memory still allocated from the
extent(9). So in such a case, make sure we clean up after outselves.

In order to make this cleaner, remove an impossible condition check
(kettenis and myself are satisfied that it will never happen), and make
iomap_load_map void (it can't fail), so that we can only fail after both
the extent is allocated and the iomap is loaded, and not inbetween the
two.

I tested iommu, kettenis tested viommu.

ok kettenis@.

Trivial malloc + memset -> malloc(,,M_ZERO) conversion.

As a small bonus this now actually zeroes the whole struct, not just the
non-varying sized part.

pmap = (conditional ? user_way : pmap = pmap_kernel());

doesn't need the second pmap =. "ok if you tested it" kettenis@

Make IOMMU code use the generic _bus_dmamem_map() and _bus_dmamem_unmap().

Fix dmesg ugliness caused by not printing iotdb stuff.

Only print iotdb stuff when DEBUG.

Perform IOMMU cache flushes on Oberon.

move some iommu_dvmamap_validate_map()'s into DEBUG instead of DIAGNOSTIC, to
slightly improve performance
ok kettenis

use mutexes to protect the iommu's extent map and the streaming buffer,
so that busdma is most likely MP_SAFE now. (while there, fix an extent
map race... the ranges were selected outside splhigh)
ok kettenis

use the right capitalization for `SBus'

ok jmc@

Correctly set the TSB size on pyro(4).

Return EFBIG in case not enough segments are present to load a DMA request
instead of ENOMEM and remove/shortcut the additional debug printfs.
Some network drivers will try to load the mbuf chain and linearize the mbufs
if EFBIG is returned.
OK miod@ kettenis@

Deal with zero wraparound in DVMA range checks;
Fix default DVMA range for Schizo's with no "virtual-dma" property;
Add TSB size indicator to Schizo & Psycho debugging messages.

ok jason@

Standardize on EFBIG in bus_dmamap_load* if caller is too greedy; I had
done this 4.5 years ago already but regressions happened; reminded by
a similar commit in NetBSD (from mrg@); ok damien@ deraadt@ jason@

Allocate as many iommu page lists as necessary to match the size request
in bus_dmamap_create() again, but this time add a few extra pages to cope
with fragmented data (such as mbufs). Tested by many.

get rid of an unused variable.

ok deraadt@

clean dmesg output for iommu more; tested miod dlg

cleanup dmesg logging for iommu goo, make it follow the standard form

revert previous commit and unbreak sparc64.

okay deraadt

allow for dma maps larger than 64 pages; jason@ ok

Use list and queue macros where applicable to make the code easier to read;
no functional change.

Off-by-ones, requests for (x * PAGE_SIZE) + 1 bytes would not allocate the
last page. Found the hard way by chris@ and claudio@.

ok jason@ deraadt@

Pass -Wformat

Typos

It is important not to forget "membar(MemIssue)". Got missed
when extracting the minimal diff for the previous iommu change.

ok jason@

Much of the sbus, psycho, and schizo bus_dma code is the same, so let's
call the real implementation functions directly instead of duplicating
lots of code that only calls the real stuff anyway.

tested by miod@ henning@
ok jason@

The "min" and "max" functions cast to "u_int", which can lead to surprising
results. Use the MIN/MAX macros instead.

tested by miod@
ok jason@ millert@

There's an off-by-one in the diagnostic code that can cause grief,
especially for NIC drivers that aren't careful about handling load
failures.

Bug reported and diff tested by Holger Burde.

ok jason@.

The existing IOMMU code had a rounding problem that was most noticeable
on faster systems under heavy network load. This replaces some of the
unreadable iommu functions with something a little less dense and a lot
less crash prone.

The bus_dma function pointer/cookie handling was broken. Change them
to work like the stacked bus_space drivers (where "work" is the key
word).

Tested my many (thanks).

ok jason@ deraadt@

don't use home grown timeval comparison when timercmp() is available; pointed out by henric

Print the pa not ldxa(pa)

- kill several unnecessary static's
- fix debugging code so it doesn't generate crashes itself

Add support for the Sun Enterprise 450
Reduce the size of a GENERIC kernel by ~190k
Remove the nasty pointer/bus_space_handle_t casts
Adds debug bus_space code including the ability to trace
bus operations (it actually works now).

The following rules are now followed (and verfified by the debug
code):

1. A "bus_space_handle_t" may only be used with the
"bus_space_tag_t" that created it.
2. Only "bus_space_map()" may create "bus_space_handle_t"s.
3. A "bus_space_handle_t" may not be modified after it has
been created (other than being destroyed by "bus_space_unmap()").


Thanks to help from mcbride, marc, jason, drahn, to anyone that might
have slipped my mind at the moment.

ok jason@, deraadt@

Remove more '\n's from panic() statements. Both trailing and leading.

Diff generated by Chris Kuethe.

this removes the functionality of adding allocated
pages into the queue already containing allocated pages.
breaks i386:setup_buffers() because of this.

No more need to initialize the result list before uvm_pglistalloc.

Aasking to load an already loaded map is not a "warn and pray" kinda thing.

From NetBSD:
-Fix some corner cases in bus_dmamap_load_mbuf().
From Takeshi Nakayama <tn@catvmics.ne.jp>
-Fix off-by-one error in iommu_dvmamap_load_raw() where if a DMA segment
has just one byte on a page the page is never mapped into the IOMMU.

No need to cast &thing to paddr_t* when thing IS a paddr_t

Deal with 24bit dvma requests

Permit compiling with DEBUG; Joey Coleman <joeycoleman@acm.org>

First round of __P removal in sys

Look up the TTE and only do the streaming cache dance if the mapping is marked as streaming.

Remove some debugging code accidentally committed in previous

From NetBSD:
Fix calculation of dma segment length when the DVMA range is crossing the boundary.

Be sure to wait the whole 0.5 seconds (half a second!) for the streaming
cache to flush, not some fraction of it, before giving up.

Debugger() is #ifdef DDB

- don't call strbuf_flush_done for each segment, call it once per map (and
then only if necessary)
- When flushing a partial segment make sure we flush all of the necessary pages
(the case where len < NBPG would not correctly flush both pages if
va + off <-> va + off + len crossed a page boundary)

From NetBSD:
- Fix pr sparc64/15633: datafault at tlp_start causes panic
- Also clean up some additional 32-bit kernel printf issues.
- Handle the case where a DMA operation wraps from the end of a page to the
beginning of the same page properly by double-mapping that page.
- Move a brace so this will compile w/o DEBUG.
- Overhaul iommu_dvmamap_load_raw().
From Me:
- Verify the streaming cache exists before waiting on it to clear during flush.
- Force sync of flush status (hardware sets it to 1 when its done)
- Force sync flush of registers (bus_space_barrier)
- Flush ALL segments on BUS_DMA_POSTREAD operations (not just the first!)

branches: 1.7.2;
Yet another sync to NetBSD uvm.
Today we add a pmap argument to pmap_update() and allocate map entries for
kernel_map from kmem_map instead of using the static entries. This should
get rid of MAX_KMAPENT panics. Also some uvm_loan problems are fixed.

Replace inclusion of <vm/foo.h> with the correct <uvm/bar.h> when necessary.
(Look ma, I might have broken the tree)

branches: 1.5.4;
Pull in several changes from NetBSD:
- Clear the dmamap inside splhigh() protection.
- Manage both streaming caches on psycho/psycho+.
And fix a botch I created when merging iommu changes last time.

Merge with NetBSD:
better handling of boundary conditions
add iommu_extract debugging function

merge vm/vm_kern.h into uvm/uvm_extern.h; art@ ok

RCSids

compiles with minimal mods (From NetBSD)

add support for bypassing iommu translation

managing the translation table entries (TTEs) on an iommu is not
free, and is in fact extremely expensive on some platforms. the
flip side of this is that forcing dma through TTEs does provide
some safety and can help during the development of drivers. however,
this has been less true in recent years than it used to be and is
less of a concern now, especially considering the performance
differences on some platforms.

devices have to create dmamaps with BUS_DMA_64BIT to bypass the
iommu because the memory window presented to hardware with direct
access to memory is at an extremely high address. there's no 32bit
bypass access to memory, it has to go through TTEs otherwise.

on an m4000 there are several orders of magnitude performance
difference between a driver with BUS_DMA_64BIT set and one without
it.

hilariously, sun used a pci bridge on a whole generation of machines
that had broken support for dma addresses over 40 bits (or around
there), so devices behind those pci bridges need to have their
dmamap_creates intercepted and any potential BUS_DMA_64BIT flags
cleared on the way to the iommu drivers. this affects at least v215,
v245, and v445, and probably u25s and u45s. it probably explains
why all their onboard nics and disk controllers feel super slow,
and why there was a meme at sun that bcopy was cheaper than dma
when moving packets on and off a nic.

ok kettenis@ deraadt@

tweak sparc64 membars as a step toward making them usable in userland.

specifically, dont rely on magic in ctlreg to implement membars. moving
that to atomic.h would add a lot of pollution to the namespace, so
move to passing the membar options to a single __membar macro.

this tweaks everything that was using the ctlreg backend to either use
an appropriate membar_foo(), or to use __membar() in the MD code.

ok kettenis@

Rename Debugger() into db_enter().

Using a name with the 'db_' prefix makes it invisible from the dynamic
profiler.

ok deraadt@, kettenis@, visa@

Some hardware (such as the onboard dc(4) of the Netra X1) has a broken DMA
engine that might attempt to read beyond the end of the buffer that was
programmed. The IOMMU catches this "DMA overrun" and throws an unrecoverable
error at us, at which point we have no choice but to panic. To avoid this
implement a BUS_DMA_OVERRUN flag that maps an additional scratch page at the
end of the vdma address range. DMA requests will spill over into this page,
which just returns zeroes.

Thanks to matthieu@ for giving me access to a machine with the problem.

ok deraadt@, beck@

Fix loading memory allocated with bus_dmamem_alloc(9). The old could would
always load all allocated pages instead of the size specified in the
bus_dmamap_load_raw(9) call. Also fixes the corner case where a specified
boundary is less than the page size, which would always create multiple
segments, even if the specified size was smaller than the boundary.

Fixes xhci(4) on sparc64.

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

uvm_km_valloc -> km_alloc

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

Format string fixes for bus_addr_t and bus_size_t, which are u_long everywhere.

Use extent_alloc_subregion_with_descr(9). This should make bus_dmamap_load(9)
and bus_dmamap_unload(9) "mpsafe".

Fix segmentation of buffers that straddle a boundary.

ok dlg@

Properly initialize the IOMMU control and status register for pyro(4).
Fixes DMA problems spotted on the v445.

When the sparc64 iommu fails to lookup a vaddr, it prints a message and
returns EFBIG. This cannot be recovered from and is usually indicative
of a damaged pmap. Therefor, panic right here instead.

This diff meant the difference between dropping into single-user mode versus
getting a traceable panic.

ok kettenis@

Do not use NULL in integer comparisons. No functional change.
ok matthew@ tedu@, also eyeballed by at least krw@ oga@ kettenis@ jsg@

cleanup more confusion regarding user.h before proc.h, or missing proc.h
ok tedu

if extent_alloc() fails, we don't clear the iomap properly before
returning an error. so next time we mess around, we may get annoying
printfs.

Fix this.

ok kettenis@

type pedantry.

the type we bind to an iommu or a GART is paddr_t, by definition, on the
other hand, the type we get out of it is not a vaddr_t, it's bus_addr_t.

fix up sparc64 iommu, amd64 iommu and the sg_dma backedn that uses it to
realise this.

ok kettenis@

Avoid clobbering error return values with the result from extent_free(), by
simply calling iommu_dvma_unload(). Solution suggested by oga@.

While there, also unwrap a line that isn't long enough to need wrapping.

Avoid clobbering error return values with the result from extent_free(), by
simply calling iommu_dvma_unload(). Solution suggested by oga@.

Convert the waitok field of uvm_pglistalloc to "flags", more will be added soon.

For the possibility of sleeping, the first two flags are UVM_PLA_WAITOK
and UVM_PLA_NOWAIT. It is an error not to show intention, so assert that
one of the two is provided. Switch over every caller in the tree to
using the appropriate flag.

ok art@, ariane@

In the rare case where after we've loaded the iomap into the hardware,
if we fail while assembling the dmamap due to the memory not fitting
into our constraints we'll return from the function with the iomap still
loaded, and more importantly with memory still allocated from the
extent(9). So in such a case, make sure we clean up after outselves.

In order to make this cleaner, remove an impossible condition check
(kettenis and myself are satisfied that it will never happen), and make
iomap_load_map void (it can't fail), so that we can only fail after both
the extent is allocated and the iomap is loaded, and not inbetween the
two.

I tested iommu, kettenis tested viommu.

ok kettenis@.

Trivial malloc + memset -> malloc(,,M_ZERO) conversion.

As a small bonus this now actually zeroes the whole struct, not just the
non-varying sized part.

pmap = (conditional ? user_way : pmap = pmap_kernel());

doesn't need the second pmap =. "ok if you tested it" kettenis@

Make IOMMU code use the generic _bus_dmamem_map() and _bus_dmamem_unmap().

Fix dmesg ugliness caused by not printing iotdb stuff.

Only print iotdb stuff when DEBUG.

Perform IOMMU cache flushes on Oberon.

move some iommu_dvmamap_validate_map()'s into DEBUG instead of DIAGNOSTIC, to
slightly improve performance
ok kettenis

use mutexes to protect the iommu's extent map and the streaming buffer,
so that busdma is most likely MP_SAFE now. (while there, fix an extent
map race... the ranges were selected outside splhigh)
ok kettenis

use the right capitalization for `SBus'

ok jmc@

Correctly set the TSB size on pyro(4).

Return EFBIG in case not enough segments are present to load a DMA request
instead of ENOMEM and remove/shortcut the additional debug printfs.
Some network drivers will try to load the mbuf chain and linearize the mbufs
if EFBIG is returned.
OK miod@ kettenis@

Deal with zero wraparound in DVMA range checks;
Fix default DVMA range for Schizo's with no "virtual-dma" property;
Add TSB size indicator to Schizo & Psycho debugging messages.

ok jason@

Standardize on EFBIG in bus_dmamap_load* if caller is too greedy; I had
done this 4.5 years ago already but regressions happened; reminded by
a similar commit in NetBSD (from mrg@); ok damien@ deraadt@ jason@

Allocate as many iommu page lists as necessary to match the size request
in bus_dmamap_create() again, but this time add a few extra pages to cope
with fragmented data (such as mbufs). Tested by many.

get rid of an unused variable.

ok deraadt@

clean dmesg output for iommu more; tested miod dlg

cleanup dmesg logging for iommu goo, make it follow the standard form

revert previous commit and unbreak sparc64.

okay deraadt

allow for dma maps larger than 64 pages; jason@ ok

Use list and queue macros where applicable to make the code easier to read;
no functional change.

Off-by-ones, requests for (x * PAGE_SIZE) + 1 bytes would not allocate the
last page. Found the hard way by chris@ and claudio@.

ok jason@ deraadt@

Pass -Wformat

Typos

It is important not to forget "membar(MemIssue)". Got missed
when extracting the minimal diff for the previous iommu change.

ok jason@

Much of the sbus, psycho, and schizo bus_dma code is the same, so let's
call the real implementation functions directly instead of duplicating
lots of code that only calls the real stuff anyway.

tested by miod@ henning@
ok jason@

The "min" and "max" functions cast to "u_int", which can lead to surprising
results. Use the MIN/MAX macros instead.

tested by miod@
ok jason@ millert@

There's an off-by-one in the diagnostic code that can cause grief,
especially for NIC drivers that aren't careful about handling load
failures.

Bug reported and diff tested by Holger Burde.

ok jason@.

The existing IOMMU code had a rounding problem that was most noticeable
on faster systems under heavy network load. This replaces some of the
unreadable iommu functions with something a little less dense and a lot
less crash prone.

The bus_dma function pointer/cookie handling was broken. Change them
to work like the stacked bus_space drivers (where "work" is the key
word).

Tested my many (thanks).

ok jason@ deraadt@

don't use home grown timeval comparison when timercmp() is available; pointed out by henric

Print the pa not ldxa(pa)

- kill several unnecessary static's
- fix debugging code so it doesn't generate crashes itself

Add support for the Sun Enterprise 450
Reduce the size of a GENERIC kernel by ~190k
Remove the nasty pointer/bus_space_handle_t casts
Adds debug bus_space code including the ability to trace
bus operations (it actually works now).

The following rules are now followed (and verfified by the debug
code):

1. A "bus_space_handle_t" may only be used with the
"bus_space_tag_t" that created it.
2. Only "bus_space_map()" may create "bus_space_handle_t"s.
3. A "bus_space_handle_t" may not be modified after it has
been created (other than being destroyed by "bus_space_unmap()").


Thanks to help from mcbride, marc, jason, drahn, to anyone that might
have slipped my mind at the moment.

ok jason@, deraadt@

Remove more '\n's from panic() statements. Both trailing and leading.

Diff generated by Chris Kuethe.

this removes the functionality of adding allocated
pages into the queue already containing allocated pages.
breaks i386:setup_buffers() because of this.

No more need to initialize the result list before uvm_pglistalloc.

Aasking to load an already loaded map is not a "warn and pray" kinda thing.

From NetBSD:
-Fix some corner cases in bus_dmamap_load_mbuf().
From Takeshi Nakayama <tn@catvmics.ne.jp>
-Fix off-by-one error in iommu_dvmamap_load_raw() where if a DMA segment
has just one byte on a page the page is never mapped into the IOMMU.

No need to cast &thing to paddr_t* when thing IS a paddr_t

Deal with 24bit dvma requests

Permit compiling with DEBUG; Joey Coleman <joeycoleman@acm.org>

First round of __P removal in sys

Look up the TTE and only do the streaming cache dance if the mapping is marked as streaming.

Remove some debugging code accidentally committed in previous

From NetBSD:
Fix calculation of dma segment length when the DVMA range is crossing the boundary.

Be sure to wait the whole 0.5 seconds (half a second!) for the streaming
cache to flush, not some fraction of it, before giving up.

Debugger() is #ifdef DDB

- don't call strbuf_flush_done for each segment, call it once per map (and
then only if necessary)
- When flushing a partial segment make sure we flush all of the necessary pages
(the case where len < NBPG would not correctly flush both pages if
va + off <-> va + off + len crossed a page boundary)

From NetBSD:
- Fix pr sparc64/15633: datafault at tlp_start causes panic
- Also clean up some additional 32-bit kernel printf issues.
- Handle the case where a DMA operation wraps from the end of a page to the
beginning of the same page properly by double-mapping that page.
- Move a brace so this will compile w/o DEBUG.
- Overhaul iommu_dvmamap_load_raw().
From Me:
- Verify the streaming cache exists before waiting on it to clear during flush.
- Force sync of flush status (hardware sets it to 1 when its done)
- Force sync flush of registers (bus_space_barrier)
- Flush ALL segments on BUS_DMA_POSTREAD operations (not just the first!)

branches: 1.7.2;
Yet another sync to NetBSD uvm.
Today we add a pmap argument to pmap_update() and allocate map entries for
kernel_map from kmem_map instead of using the static entries. This should
get rid of MAX_KMAPENT panics. Also some uvm_loan problems are fixed.

Replace inclusion of <vm/foo.h> with the correct <uvm/bar.h> when necessary.
(Look ma, I might have broken the tree)

branches: 1.5.4;
Pull in several changes from NetBSD:
- Clear the dmamap inside splhigh() protection.
- Manage both streaming caches on psycho/psycho+.
And fix a botch I created when merging iommu changes last time.

Merge with NetBSD:
better handling of boundary conditions
add iommu_extract debugging function

merge vm/vm_kern.h into uvm/uvm_extern.h; art@ ok

RCSids

compiles with minimal mods (From NetBSD)

tweak sparc64 membars as a step toward making them usable in userland.

specifically, dont rely on magic in ctlreg to implement membars. moving
that to atomic.h would add a lot of pollution to the namespace, so
move to passing the membar options to a single __membar macro.

this tweaks everything that was using the ctlreg backend to either use
an appropriate membar_foo(), or to use __membar() in the MD code.

ok kettenis@

Rename Debugger() into db_enter().

Using a name with the 'db_' prefix makes it invisible from the dynamic
profiler.

ok deraadt@, kettenis@, visa@

Some hardware (such as the onboard dc(4) of the Netra X1) has a broken DMA
engine that might attempt to read beyond the end of the buffer that was
programmed. The IOMMU catches this "DMA overrun" and throws an unrecoverable
error at us, at which point we have no choice but to panic. To avoid this
implement a BUS_DMA_OVERRUN flag that maps an additional scratch page at the
end of the vdma address range. DMA requests will spill over into this page,
which just returns zeroes.

Thanks to matthieu@ for giving me access to a machine with the problem.

ok deraadt@, beck@

Fix loading memory allocated with bus_dmamem_alloc(9). The old could would
always load all allocated pages instead of the size specified in the
bus_dmamap_load_raw(9) call. Also fixes the corner case where a specified
boundary is less than the page size, which would always create multiple
segments, even if the specified size was smaller than the boundary.

Fixes xhci(4) on sparc64.

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

uvm_km_valloc -> km_alloc

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.