Linux and AMD: BC3 flashing on RADV, ASTC decode stalls, lower RAM usage, a buffer cache crash (#260)

* host_memory: punch holes in the backing on Linux instead of zero filling freed memory

Freeing guest memory zero filled the memfd backing with memset, which
commits every page it touches. fallocate(PUNCH_HOLE) hands the pages back
to the kernel instead, and they read as zero afterwards. Only the partial
pages at the ends are still cleared by hand.

Mario Kart World on Linux (RX 9070 XT, 31 GB): peak RSS 23.1 -> 15.9 GB.
EDEN_PUNCH_HOLE=0 restores the memset.

Assisted-by: Claude Opus 5.5 (debugging, profiling and code drafts)

* fiber: don't zero fill the stacks, untouched pages stay unbacked

The value initialised stack arrays were zero filled when each fiber was
made, which commits the whole stack. Left uninitialised, the pages a
fiber never touches stay unbacked. Together with the descriptor ring
change, about 1.2 GB less RSS in Mario Kart World.

Assisted-by: Claude Opus 5.5 (debugging, profiling and code drafts)

* vulkan: don't zero fill the descriptor update ring

Same as the fiber stacks: make_unique value initialised the whole ring
for every frame in flight. Entries are always written before they are
read, so allocate it without initialising it.

Assisted-by: Claude Opus 5.5 (debugging, profiling and code drafts)

* vulkan: wait for the GPU after an ASTC transcode decode like the other ASTC decodes

TranscodeAstcToBcn (astc_recompression Bc1/Bc3 with GPU decode) was the
only ASTC decode recorded with wait = false. On RADV (RX 9070 XT, Mesa)
Mario Kart World with BC3 flashed heavily all over the screen, while the
uncompressed GPU decode, which waits, did not. CPU decode with BC3 does
not flash either. Waiting after the transcode removes the flashing; full
pipeline barriers around its copy did not, so the wait hides a race
somewhere else rather than fixing one in the transcode itself.

Assisted-by: Claude Opus 5.5 (debugging, profiling and code drafts)

* vulkan: submit after an ASTC decode instead of waiting for the GPU by default

Every ASTC decode ended with scheduler.Finish(). Over a 19 minute Mario
Kart World Grand Prix that stalled the GPU thread 30.7 s in total, up to
1.2 s out of every 5 s while new textures stream in (track sections,
race starts), which is where the hitches were. Flushing keeps the
ordering that stops the BC3 flashing on RADV without the CPU waiting:
no flashing in play, and frames over 100 ms went from dozens to 3 in
about 7 minutes. EDEN_NX2_ASTC_FLUSH=0 restores the wait.

Assisted-by: Claude Opus 5.5 (debugging, profiling and code drafts)

* texture_cache: over the expected use, evict costly images unused for 1800 frames first

The collector never deletes CostlyLoad images outside aggressive mode, so
once a cup moves on, the previous tracks' textures stay resident and the
cache sits over its expected use deleting the cheap images still in use.
When over the expected use, costly images unused for 1800 frames (half a
minute) now go first.

Mario Kart World Grand Prix on an RX 9070 XT: frames over 20 ms 8.8% ->
4.0%, 58.7 fps average. EDEN_NX2_TC_OLD_AGE=<frames> sets the age, 0
turns it off.

Assisted-by: Claude Opus 5.5 (debugging, profiling and code drafts)

* buffer_cache: no buffer for device addresses past the page table

FindBuffer indexed page_table, which covers 2^36 bytes, with any address.
Mario Kart World crashed after about 4.5 minutes with a uniform buffer at
0x72253b2f2890 (FindBuffer <- UpdateUniformBuffers <- DrawEnd from a
macro), reading far past the table. Treat such an address as no buffer.
Where that address comes from is still unknown; this only stops the
crash.

Assisted-by: Claude Opus 5.5 (debugging, profiling and code drafts)

8 files changed, 50 insertions, 15 deletions

diff --git a/src/common/fiber.cpp b/src/common/fiber.cpp
index 0e7f28c5b9..08558a0026 100644
--- a/src/common/fiber.cpp
+++ b/src/common/fiber.cpp
@@ -25,8 +25,8 @@ struct Fiber::FiberImpl {
     FiberImpl() {}
 
     u32 canary_1 = CANARY_VALUE;
-    std::array<u8, DEFAULT_STACK_SIZE> stack{};
-    std::array<u8, DEFAULT_STACK_SIZE> rewind_stack{};
+    std::array<u8, DEFAULT_STACK_SIZE> stack;
+    std::array<u8, DEFAULT_STACK_SIZE> rewind_stack;
     u32 canary_2 = CANARY_VALUE;
 
     boost::context::detail::fcontext_t context{};
diff --git a/src/common/host_memory.cpp b/src/common/host_memory.cpp
index 75a9da80e9..147eba8477 100644
--- a/src/common/host_memory.cpp
+++ b/src/common/host_memory.cpp
@@ -51,6 +51,7 @@
 
 #endif // ^^^ POSIX ^^^
 
+#include <cstdlib>
 #include <mutex>
 #include <random>
 
@@ -517,6 +518,14 @@ public:
         , virtual_size{virtual_size_}
     {}
 
+#ifdef __linux__
+    bool PunchHole(size_t offset, size_t length) {
+        return fd >= 0 &&
+               fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, static_cast<off_t>(offset),
+                         static_cast<off_t>(length)) == 0;
+    }
+#endif
+
     bool Init() {
         long page_size = sysconf(_SC_PAGESIZE);
         ASSERT_MSG(page_size == 0x1000, "page size {:#x} is incompatible with 4K paging", page_size);
@@ -791,6 +800,19 @@ void HostMemory::Protect(size_t virtual_offset, size_t length, MemoryPermission
 }
 
 void HostMemory::ClearBackingRegion(size_t physical_offset, size_t length, u32 fill_value) {
+#ifdef __linux__
+    const size_t first = AlignUp(physical_offset, PageAlignment);
+    const size_t last = AlignDown(physical_offset + length, PageAlignment);
+    static const bool punch = [] {
+        const char* value = std::getenv("EDEN_PUNCH_HOLE");
+        return value == nullptr || value[0] != '0';
+    }();
+    if (punch && fill_value == 0 && impl && last > first && impl->PunchHole(first, last - first)) {
+        std::memset(backing_base + physical_offset, 0, first - physical_offset);
+        std::memset(backing_base + last, 0, physical_offset + length - last);
+        return;
+    }
+#endif
     std::memset(backing_base + physical_offset, fill_value, length);
 }
 
diff --git a/src/video_core/buffer_cache/buffer_cache.h b/src/video_core/buffer_cache/buffer_cache.h
index e09a435dcc..a3358048dc 100644
--- a/src/video_core/buffer_cache/buffer_cache.h
+++ b/src/video_core/buffer_cache/buffer_cache.h
@@ -1651,10 +1651,10 @@ void BufferCache<P>::MarkWrittenBuffer(BufferId buffer_id, DAddr device_addr, u3
 
 template <class P>
 BufferId BufferCache<P>::FindBuffer(DAddr device_addr, u32 size, bool sparse_compatible) {
-    if (device_addr == 0) {
+    const u64 page = device_addr >> CACHING_PAGEBITS;
+    if (device_addr == 0 || page >= page_table.size()) {
         return NULL_BUFFER_ID;
     }
-    const u64 page = device_addr >> CACHING_PAGEBITS;
     const BufferId buffer_id = page_table[page];
     if (buffer_id) {
         Buffer& buffer = slot_buffers[buffer_id];
diff --git a/src/video_core/renderer_vulkan/vk_compute_pass.cpp b/src/video_core/renderer_vulkan/vk_compute_pass.cpp
index 691be728d2..cb8dcde42f 100644
--- a/src/video_core/renderer_vulkan/vk_compute_pass.cpp
+++ b/src/video_core/renderer_vulkan/vk_compute_pass.cpp
@@ -801,11 +801,11 @@ void ASTCDecoderPass::AssembleInto(const VideoCommon::ImageInfo& info, u32 skip,
         cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT,
                        vk::PIPELINE_STAGE_GRAPHICS_COMPUTE, 0, image_barrier);
     });
-    // Each decode ends with the GPU thread waiting for the host GPU to go idle. Measured here
-    // (count and time per 5 s); EDEN_NX2_ASTC_FLUSH=1 submits without waiting instead.
+    // Each decode ends with a submit to the host GPU. Measured here
+    // (count and time per 5 s); EDEN_NX2_ASTC_FLUSH=0 waits for the host GPU to go idle instead.
     static const bool flush_only = [] {
         const char* value = std::getenv("EDEN_NX2_ASTC_FLUSH");
-        return value != nullptr && value[0] == '1';
+        return value == nullptr || value[0] != '0';
     }();
     if (!target.wait) {
         return;
diff --git a/src/video_core/renderer_vulkan/vk_texture_cache.cpp b/src/video_core/renderer_vulkan/vk_texture_cache.cpp
index 4e428229f6..b0e5eacb32 100644
--- a/src/video_core/renderer_vulkan/vk_texture_cache.cpp
+++ b/src/video_core/renderer_vulkan/vk_texture_cache.cpp
@@ -4267,7 +4267,7 @@ void TextureCacheRuntime::TranscodeAstcToBcn(
             .aspect_mask = VK_IMAGE_ASPECT_COLOR_BIT,
             .initialized = false,
             .level_view = [&pending](s32 level) { return *pending.views[level]; },
-            .wait = false,
+            .wait = true,
         },
         map, swizzles);
 
diff --git a/src/video_core/renderer_vulkan/vk_update_descriptor.cpp b/src/video_core/renderer_vulkan/vk_update_descriptor.cpp
index 742a8d9969..29f3ee71a1 100644
--- a/src/video_core/renderer_vulkan/vk_update_descriptor.cpp
+++ b/src/video_core/renderer_vulkan/vk_update_descriptor.cpp
@@ -20,10 +20,10 @@ UpdateDescriptorQueue::UpdateDescriptorQueue(const Device& device_, size_t frame
                                              bool supports_descriptor_buffer_)
     : device{device_}, frame_payload_size{frame_payload_size_},
       supports_descriptor_buffer{supports_descriptor_buffer_},
-      payload(frame_payload_size_ * FRAMES_IN_FLIGHT)
+      payload(std::make_unique_for_overwrite<DescriptorUpdateEntry[]>(frame_payload_size_ * FRAMES_IN_FLIGHT))
 {
-    payload_start = payload.data();
-    payload_cursor = payload.data();
+    payload_start = payload.get();
+    payload_cursor = payload.get();
 }
 
 UpdateDescriptorQueue::~UpdateDescriptorQueue() = default;
@@ -32,7 +32,7 @@ void UpdateDescriptorQueue::TickFrame() {
     if (++frame_index >= FRAMES_IN_FLIGHT) {
         frame_index = 0;
     }
-    payload_start = payload.data() + frame_index * frame_payload_size;
+    payload_start = payload.get() + frame_index * frame_payload_size;
     payload_cursor = payload_start;
 }
 
diff --git a/src/video_core/renderer_vulkan/vk_update_descriptor.h b/src/video_core/renderer_vulkan/vk_update_descriptor.h
index 9f30e65dbd..4d393376b4 100644
--- a/src/video_core/renderer_vulkan/vk_update_descriptor.h
+++ b/src/video_core/renderer_vulkan/vk_update_descriptor.h
@@ -6,6 +6,7 @@
 
 #pragma once
 
+#include <memory>
 #include <variant>
 #include <vector>
 #include "video_core/vulkan_common/vulkan_wrapper.h"
@@ -122,7 +123,7 @@ private:
     DescriptorUpdateEntry* payload_cursor = nullptr;
     DescriptorUpdateEntry* payload_start = nullptr;
     const DescriptorUpdateEntry* upload_start = nullptr;
-    std::vector<DescriptorUpdateEntry> payload;
+    std::unique_ptr<DescriptorUpdateEntry[]> payload;
 };
 
 // TODO: should these be separate classes instead?
diff --git a/src/video_core/texture_cache/texture_cache.h b/src/video_core/texture_cache/texture_cache.h
index 32775b8719..55ec4c64df 100644
--- a/src/video_core/texture_cache/texture_cache.h
+++ b/src/video_core/texture_cache/texture_cache.h
@@ -125,6 +125,7 @@ template <class P>
 void TextureCache<P>::RunGarbageCollector() {
     bool high_priority_mode = false;
     bool aggressive_mode = false;
+    bool evict_old_costly = false;
     u64 ticks_to_destroy = 0;
     size_t num_iterations = 0;
     const auto Configure = [&](bool allow_aggressive) {
@@ -139,7 +140,8 @@ void TextureCache<P>::RunGarbageCollector() {
         ticks_to_destroy = (std::max)(ticks_to_destroy, min_age);
         num_iterations = aggressive_mode ? 40 : (high_priority_mode ? 20 : 10);
     };
-    const auto Cleanup = [this, &num_iterations, &high_priority_mode, &aggressive_mode](ImageId image_id) {
+    const auto Cleanup = [this, &num_iterations, &high_priority_mode, &aggressive_mode,
+                          &evict_old_costly](ImageId image_id) {
         if (num_iterations == 0) {
             return true;
         }
@@ -149,7 +151,8 @@ void TextureCache<P>::RunGarbageCollector() {
             return false;
         }
         const bool must_download = IsDownloadable(image) && False(image.flags & ImageFlagBits::BadOverlap);
-        if ((!aggressive_mode && True(image.flags & ImageFlagBits::CostlyLoad)) || (!high_priority_mode && must_download)) {
+        if ((!aggressive_mode && !evict_old_costly && True(image.flags & ImageFlagBits::CostlyLoad)) ||
+            (!high_priority_mode && must_download)) {
             return false;
         }
         if (must_download) {
@@ -176,6 +179,15 @@ void TextureCache<P>::RunGarbageCollector() {
         return false;
     };
     Configure(false);
+    static const u64 old_age = [] {
+        const char* value = std::getenv("EDEN_NX2_TC_OLD_AGE");
+        return value != nullptr ? std::strtoull(value, nullptr, 10) : 1800ULL;
+    }();
+    if (high_priority_mode && old_age != 0 && frame_tick > old_age) {
+        evict_old_costly = true;
+        lru_cache.ForEachItemBelow(frame_tick - old_age, Cleanup);
+        evict_old_costly = false;
+    }
     lru_cache.ForEachItemBelow(frame_tick - ticks_to_destroy, Cleanup);
     if (total_used_memory >= critical_memory) {
         Configure(true);