app_server: fix integer overflow in ServerBitmap

Integer overflow caused bitmap buffer creation of wrong size and out of
bounds access when large bitmap was created. Now allocation failure is
reported for large bitmaps.

This prevents app_server from crashing while playing YouTube in WebPositive.

Fixes #16489.

Change-Id: I297aa6e3b79b32a486d297f1239a1fd4397a8a36
Reviewed-on: https://review.haiku-os.org/c/haiku/+/4209
Reviewed-by: Sergei Reznikov <diver@gelios.net>
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>

app_server memory management: use ObjectDeleter to mark ownership

Make object ownership explicit by use of ObjectDeleter where possible.

Change-Id: I499a00aa3390d1510ae284419e73faffa5166430
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2695
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>
Reviewed-by: Alex von Gluck IV <kallisti5@unixzen.com>

Make it possible to reconnect BBitmap to the app_server.
* maintain a list of all BBitmaps
* refactor the client memory allocator class, its possible now to just clone existing client area

* Fixed race conditions in the server's bitmap/picture handling: the objects
are now removed from the maps as soon as the client deletes them. This also
makes the "client reference" mechanism superfluous I introduced earlier.
* ServerApp::SetCurrentCursor() must always call Desktop::SetCursor(), since it
is also called whenever the current application changes. This fixes the cursor
almost never changing.
* Renamed ServerPicture::Usurp()/StepDown() to PushPicture(), and PopPicture().
* Also, they now acquire a reference to the picture in question (ie. the picture
you get from PopPicture() also owns a reference you need to free).
* ServerApp::CreatePicture() may fail, too. This case is now handled in the code
that calls it.
* Previously, the ServerWindow tried to process up to 70 messages in one go.
That obviously caused bug #4709. Now, we have the additional requirement to
not hold the desktop lock for longer than 25 ms. I haven't tested it with
Kaleidoscope yet, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35472 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Bitmaps and pictures now maintain their client reference independently;
clients can no longer release more references than they own.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34130 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Use BReferenceable instead of Referenceable. Didn't notice there was a
BReferenceable in the first place, thanks Ingo!


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33880 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Refactored ServerBitmap a bit: it now inherits from Referenceable instead of
roling its own solution.
* Also removed BitmapManager::DeleteBitmap() - you only call
ServerBitmap::RemoveReference(), and that one will notify the manager if
needed.
* Some more cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33878 a95241bf-73f2-0310-859d-f6bbb57e9c96

* ServerApp's bitmap and picture handling was completely broken, as it ignored
concurrency as well as reference counting, causing occasional crashes and
memory corruption.
* ServerPicture now subclasses Referenceable, and will notify its owner when
it's going to be deleted. This might bring some regressions, although I
couldn't spot anything wrong yet.
* ServerBitmap will now also notify its owner when it's going to be deleted as
well.
* Switched from the former picture/bitmap lists to a std::map. This also solves
the issue of not checking whether or not the bitmap/picture actually belongs
to the ServerApp (before, all apps could access and delete all
pictures/bitmaps)
* Introduced a new fMapLocker that guards the new maps.
* ServerWindow now uses GetBitmap()/GetPicture(), and gives up its reference
after use.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33876 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Some style cleanups, no functional change.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31759 a95241bf-73f2-0310-859d-f6bbb57e9c96

Export get_bytes_per_row() from InterfacePrivate.h, and use it in ServerBitmap
in place of the own rolled implementation. Comment typo fix.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31757 a95241bf-73f2-0310-859d-f6bbb57e9c96

Got rid of fBitsPerPixel (and BitsPerPixel()) in ServerBitmap,
since they were nowhere used.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31677 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced new BBitmap flag B_BITMAP_SCALE_BILINEAR.
* When drawing BBitmaps with scaling in the app_server, use a bilinear
filter when a bitmap has this flag set. (Hope nobody objects, otherwise
I can revert or improve this. Performance can certainly be improved, since
the AGG implementation is too generic. But that goes for the nearest
neighbor implementation as well.)
* Flags are uint32, fix app_server side code to declare them correctly. Use
appropriate link methods in BBitmap and ServerApp.
* Enable the BeOS compatibility mode for B_RGB32 (works just like B_RGBA32
in B_OP_ALPHA mode).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26649 a95241bf-73f2-0310-859d-f6bbb57e9c96

* get rid of RGBColor usage where it is not needed, this simplified many things,
possibly making them a little faster too
* mess with decorator button size calculation to make the whole layout scale
more agreeable with the font size (no more fixed offsets/insets), but it
is work in progress
* DefaultDecorator no longer allocated the border color array, it is part of
the object now
* small memory footprint optimizations in ViewLayer, Decorator and WindowLayer


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22003 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Implemented the overlay suspend/resume protocol on mode changes; not really tested
yet. Also, BBitmap::LockBits() should probably fail when the Bits() are NULL.
* The downside is that many more classes now know of each other.
* Cleaned up the work divided between the BitmapManager and the Overlay class.
* Fixed a memory leak in AS_CREATE_BITMAP in case the bitmap could not be added to
the ServerApp's bitmap list.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@21512 a95241bf-73f2-0310-859d-f6bbb57e9c96

No longer invalidates the view when an overlay bitmap is updated.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@17319 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Some refactoring: renamed OverlayCookie to Overlay and put it in its own
source file.
* An overlay is now also hidden in case its is removed from the window.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@17209 a95241bf-73f2-0310-859d-f6bbb57e9c96

* We now have working video overlay - even though the overlay_token handling
is currently broken, mode switches probably fail or result in sudden death
(didn't try) it's good enough for Radeon cards and VLC (might work with
others as well).
* Implemented follow modes for view bitmaps (wasn't taken into account at
all before).
* Switched to a darker overlay color for now (dunno what exactly makes a
good candidate there, but this one is good enough for now).
* Added TODO about a race condition in AS_LAYER_SET_VIEW_BITMAP.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@17208 a95241bf-73f2-0310-859d-f6bbb57e9c96

* With Rudolf's information about relocating overlays, I changed the way memory
is managed for those bitmaps:
- the shared client memory mechanism is used to allocate a small overlay_client_data
structure that contains the actual buffer and a semaphore that you have acquire in
order to access it.
- LockBits()/UnlockBits() now have a function: you need to call them before accessing
the overlay buffer, and you need to keep that lock until you're done with it.
* The overlay cookie is now an extra member of the ServerBitmap class.
* Removed fInitialized from ServerBitmap - IsValid() now just checks the buffer associated
with the bitmap.
* ViewLayer::Draw() will now handle overlay bitmaps specially and will draw the overlay
color instead of any contents (this is currently in ugly pink, but will become some
dark color later on).
* All what's missing from actually being able to use overlays now is to configure
them so that they are shown on screen. VLC will now show an empty pink window when
overlay video is enabled.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@17201 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Refined overlay support a bit: we now allow as many overlay bitmaps to be
created as the graphics driver does.
* Also, B_BITMAP_RESERVE_OVERLAY_CHANNEL should now work as expected.
* We're no longer using the B_OVERLAY_COUNT hook anymore - that one really
looks like a misconception to me; I don't see how it can be useful.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@17196 a95241bf-73f2-0310-859d-f6bbb57e9c96

Implemented some more overlay support - the overlay bitmap is now allocated
via the graphics driver (but not yet shown on screen).
I probably got the meaning of the "overlay count" wrong - I guess that you
can allocate any number of overlay bitmaps, but can only see "overlay count"
on screen at a time (right now, I only allow to create "overlay count" bitmaps).
Stephan?


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@17193 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Removed headers/private/servers/app - everything is in src/servers/app now.
* Removed DisplaySupport.h, wasn't needed anymore.
* Removed private color set functions from InterfaceDefs.cpp - we might want
something similar, but definitely not like that.
* Minor cleanup, added some missing licenses.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@16831 a95241bf-73f2-0310-859d-f6bbb57e9c96

Make it possible to reconnect BBitmap to the app_server.
* maintain a list of all BBitmaps
* refactor the client memory allocator class, its possible now to just clone existing client area

* Fixed race conditions in the server's bitmap/picture handling: the objects
are now removed from the maps as soon as the client deletes them. This also
makes the "client reference" mechanism superfluous I introduced earlier.
* ServerApp::SetCurrentCursor() must always call Desktop::SetCursor(), since it
is also called whenever the current application changes. This fixes the cursor
almost never changing.
* Renamed ServerPicture::Usurp()/StepDown() to PushPicture(), and PopPicture().
* Also, they now acquire a reference to the picture in question (ie. the picture
you get from PopPicture() also owns a reference you need to free).
* ServerApp::CreatePicture() may fail, too. This case is now handled in the code
that calls it.
* Previously, the ServerWindow tried to process up to 70 messages in one go.
That obviously caused bug #4709. Now, we have the additional requirement to
not hold the desktop lock for longer than 25 ms. I haven't tested it with
Kaleidoscope yet, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35472 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Bitmaps and pictures now maintain their client reference independently;
clients can no longer release more references than they own.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34130 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Use BReferenceable instead of Referenceable. Didn't notice there was a
BReferenceable in the first place, thanks Ingo!


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33880 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Refactored ServerBitmap a bit: it now inherits from Referenceable instead of
roling its own solution.
* Also removed BitmapManager::DeleteBitmap() - you only call
ServerBitmap::RemoveReference(), and that one will notify the manager if
needed.
* Some more cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33878 a95241bf-73f2-0310-859d-f6bbb57e9c96

* ServerApp's bitmap and picture handling was completely broken, as it ignored
concurrency as well as reference counting, causing occasional crashes and
memory corruption.
* ServerPicture now subclasses Referenceable, and will notify its owner when
it's going to be deleted. This might bring some regressions, although I
couldn't spot anything wrong yet.
* ServerBitmap will now also notify its owner when it's going to be deleted as
well.
* Switched from the former picture/bitmap lists to a std::map. This also solves
the issue of not checking whether or not the bitmap/picture actually belongs
to the ServerApp (before, all apps could access and delete all
pictures/bitmaps)
* Introduced a new fMapLocker that guards the new maps.
* ServerWindow now uses GetBitmap()/GetPicture(), and gives up its reference
after use.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33876 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Some style cleanups, no functional change.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31759 a95241bf-73f2-0310-859d-f6bbb57e9c96

Export get_bytes_per_row() from InterfacePrivate.h, and use it in ServerBitmap
in place of the own rolled implementation. Comment typo fix.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31757 a95241bf-73f2-0310-859d-f6bbb57e9c96

Got rid of fBitsPerPixel (and BitsPerPixel()) in ServerBitmap,
since they were nowhere used.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31677 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced new BBitmap flag B_BITMAP_SCALE_BILINEAR.
* When drawing BBitmaps with scaling in the app_server, use a bilinear
filter when a bitmap has this flag set. (Hope nobody objects, otherwise
I can revert or improve this. Performance can certainly be improved, since
the AGG implementation is too generic. But that goes for the nearest
neighbor implementation as well.)
* Flags are uint32, fix app_server side code to declare them correctly. Use
appropriate link methods in BBitmap and ServerApp.
* Enable the BeOS compatibility mode for B_RGB32 (works just like B_RGBA32
in B_OP_ALPHA mode).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26649 a95241bf-73f2-0310-859d-f6bbb57e9c96

* get rid of RGBColor usage where it is not needed, this simplified many things,
possibly making them a little faster too
* mess with decorator button size calculation to make the whole layout scale
more agreeable with the font size (no more fixed offsets/insets), but it
is work in progress
* DefaultDecorator no longer allocated the border color array, it is part of
the object now
* small memory footprint optimizations in ViewLayer, Decorator and WindowLayer


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22003 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Implemented the overlay suspend/resume protocol on mode changes; not really tested
yet. Also, BBitmap::LockBits() should probably fail when the Bits() are NULL.
* The downside is that many more classes now know of each other.
* Cleaned up the work divided between the BitmapManager and the Overlay class.
* Fixed a memory leak in AS_CREATE_BITMAP in case the bitmap could not be added to
the ServerApp's bitmap list.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@21512 a95241bf-73f2-0310-859d-f6bbb57e9c96

No longer invalidates the view when an overlay bitmap is updated.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@17319 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Some refactoring: renamed OverlayCookie to Overlay and put it in its own
source file.
* An overlay is now also hidden in case its is removed from the window.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@17209 a95241bf-73f2-0310-859d-f6bbb57e9c96

* We now have working video overlay - even though the overlay_token handling
is currently broken, mode switches probably fail or result in sudden death
(didn't try) it's good enough for Radeon cards and VLC (might work with
others as well).
* Implemented follow modes for view bitmaps (wasn't taken into account at
all before).
* Switched to a darker overlay color for now (dunno what exactly makes a
good candidate there, but this one is good enough for now).
* Added TODO about a race condition in AS_LAYER_SET_VIEW_BITMAP.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@17208 a95241bf-73f2-0310-859d-f6bbb57e9c96

* With Rudolf's information about relocating overlays, I changed the way memory
is managed for those bitmaps:
- the shared client memory mechanism is used to allocate a small overlay_client_data
structure that contains the actual buffer and a semaphore that you have acquire in
order to access it.
- LockBits()/UnlockBits() now have a function: you need to call them before accessing
the overlay buffer, and you need to keep that lock until you're done with it.
* The overlay cookie is now an extra member of the ServerBitmap class.
* Removed fInitialized from ServerBitmap - IsValid() now just checks the buffer associated
with the bitmap.
* ViewLayer::Draw() will now handle overlay bitmaps specially and will draw the overlay
color instead of any contents (this is currently in ugly pink, but will become some
dark color later on).
* All what's missing from actually being able to use overlays now is to configure
them so that they are shown on screen. VLC will now show an empty pink window when
overlay video is enabled.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@17201 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Refined overlay support a bit: we now allow as many overlay bitmaps to be
created as the graphics driver does.
* Also, B_BITMAP_RESERVE_OVERLAY_CHANNEL should now work as expected.
* We're no longer using the B_OVERLAY_COUNT hook anymore - that one really
looks like a misconception to me; I don't see how it can be useful.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@17196 a95241bf-73f2-0310-859d-f6bbb57e9c96

Implemented some more overlay support - the overlay bitmap is now allocated
via the graphics driver (but not yet shown on screen).
I probably got the meaning of the "overlay count" wrong - I guess that you
can allocate any number of overlay bitmaps, but can only see "overlay count"
on screen at a time (right now, I only allow to create "overlay count" bitmaps).
Stephan?


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@17193 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Removed headers/private/servers/app - everything is in src/servers/app now.
* Removed DisplaySupport.h, wasn't needed anymore.
* Removed private color set functions from InterfaceDefs.cpp - we might want
something similar, but definitely not like that.
* Minor cleanup, added some missing licenses.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@16831 a95241bf-73f2-0310-859d-f6bbb57e9c96