[core, hle, video_core, memory] Improve multi-process, display layers, dynamic shader cache and rework overlay (#4238)

- [x] I have read and followed the [Contribution Guidelines](https://git.eden-emu.dev/eden-emu/eden/src/branch/master/CONTRIBUTING.md#code-contributions).
- [x] I have read and followed the [AI Policy](https://git.eden-emu.dev/eden-emu/eden/src/branch/master/docs/policies/AI.md)
- [x] I have read and followed the [Coding Guidelines](https://git.eden-emu.dev/eden-emu/eden/src/branch/master/docs/policies/Coding.md) to the best of my ability.

-------------------

- Dynamic shader cache reloading capability for qlaunch
- Multi-process improvements (thanks to @frank1734), instead of just using the main application we now respect caller process (also did it for HID devices while I was at it)
- Layer stack masks & shared buffer screenshot for video core, added different masks (screenshot, recording, etc.) this was discovered as an issue due to how in qlaunch the transition was not right between applications. May not be perfect but also fixes screenshots while using qlaunch
- Reworked overlay display management (input and visibility) - instead of random numbers as I've previously did, I decided to add an AppletZIndex enum for better readability. Also split capability of input by touch and gamepad.
- etc.

Reviewed-on: https://git.eden-emu.dev/eden-emu/eden/pulls/4238
Reviewed-by: Samuel <lizzie@eden-emu.dev>
Reviewed-by: CamilleLaVey <camillelavey99@gmail.com>
Reviewed-by: MaranBr <maranbr@eden-emu.dev>

2 files changed, 15 insertions, 3 deletions

diff --git a/src/video_core/renderer_opengl/renderer_opengl.cpp b/src/video_core/renderer_opengl/renderer_opengl.cpp
index 1c72147113..f9ec32e652 100644
--- a/src/video_core/renderer_opengl/renderer_opengl.cpp
+++ b/src/video_core/renderer_opengl/renderer_opengl.cpp
@@ -199,7 +199,10 @@ void RendererOpenGL::RenderScreenshot(std::span<const Tegra::FramebufferConfig>
         return;
     }
 
-    RenderToBuffer(framebuffers, renderer_settings.screenshot_framebuffer_layout,
+    const auto screenshot_layers = Tegra::FilterLayerStack(
+        framebuffers, renderer_settings.screenshot_layer_stack, screenshot_layer_scratch);
+
+    RenderToBuffer(screenshot_layers, renderer_settings.screenshot_framebuffer_layout,
                    renderer_settings.screenshot_bits);
 
     renderer_settings.screenshot_complete_callback(true);
@@ -208,6 +211,12 @@ void RendererOpenGL::RenderScreenshot(std::span<const Tegra::FramebufferConfig>
 
 void RendererOpenGL::RenderAppletCaptureLayer(
     std::span<const Tegra::FramebufferConfig> framebuffers) {
+    const auto capture_layers = Tegra::FilterLayerStack(
+        framebuffers, Service::Nvnflinger::LayerStackId::LastFrame, applet_capture_layers);
+
+    if (capture_layers.empty())
+        return;
+
     GLint old_read_fb;
     GLint old_draw_fb;
     glGetIntegerv(GL_READ_FRAMEBUFFER_BINDING, &old_read_fb);
@@ -217,7 +226,7 @@ void RendererOpenGL::RenderAppletCaptureLayer(
     glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER,
                               capture_renderbuffer.handle);
 
-    blit_applet->DrawScreen(framebuffers, VideoCore::Capture::Layout, true);
+    blit_applet->DrawScreen(capture_layers, VideoCore::Capture::Layout, true);
 
     glBindFramebuffer(GL_READ_FRAMEBUFFER, old_read_fb);
     glBindFramebuffer(GL_DRAW_FRAMEBUFFER, old_draw_fb);
diff --git a/src/video_core/renderer_opengl/renderer_opengl.h b/src/video_core/renderer_opengl/renderer_opengl.h
index f694dc413a..5a6757e805 100644
--- a/src/video_core/renderer_opengl/renderer_opengl.h
+++ b/src/video_core/renderer_opengl/renderer_opengl.h
@@ -1,4 +1,4 @@
-// SPDX-FileCopyrightText: Copyright 2025 Eden Emulator Project
+// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project
 // SPDX-License-Identifier: GPL-3.0-or-later
 
 // SPDX-FileCopyrightText: 2014 Citra Emulator Project
@@ -61,6 +61,9 @@ private:
     void RenderScreenshot(std::span<const Tegra::FramebufferConfig> framebuffers);
     void RenderAppletCaptureLayer(std::span<const Tegra::FramebufferConfig> framebuffers);
 
+    std::vector<Tegra::FramebufferConfig> applet_capture_layers;
+    std::vector<Tegra::FramebufferConfig> screenshot_layer_scratch;
+
     Core::Frontend::EmuWindow& emu_window;
     Tegra::MaxwellDeviceMemoryManager& device_memory;
     Tegra::GPU& gpu;