nx2: read command lists the GPU wrote from the GPU's copy

Mario Kart World runs small command lists (20-byte entries) that its
culling dispatches fill in each frame. Accurate GPU mode reads every
command list through the caches; Fast mode reads guest memory directly
(on the split's front thread too), where those entries were still the
words of an older frame, so the draws behind them were left out: the
trees around the courses vanished in Fast mode only.

The buffer cache now also marks the device pages the GPU writes (shader
stores, copies of GPU-written data) in a bitmap the split's front can
read. A command list on such a page waits for the back to run what was
queued before it and is flushed when its words are still only in host
buffers. In a race that is ~5% of command lists (13,500 a second of
262,000) and ~400 flushes a second.

1 files changed, 27 insertions, 0 deletions

diff --git a/src/video_core/dma_pusher.cpp b/src/video_core/dma_pusher.cpp
index 88d571ae5f..8c142b36db 100644
--- a/src/video_core/dma_pusher.cpp
+++ b/src/video_core/dma_pusher.cpp
@@ -162,6 +162,7 @@ bool DmaPusher::Step() {
             command_resume_offset +=
                 ProcessCommands(commands.subspan(command_resume_offset), command_resume_offset);
         } else {
+            FlushGpuWrittenCommands(dma_state.dma_get, size_t{header.size} * sizeof(u32));
             Tegra::Memory::GpuGuestMemory<Tegra::CommandHeader, Tegra::Memory::GuestMemoryFlags::UnsafeRead>headers(memory_manager, dma_state.dma_get, header.size, &command_headers);
             const std::span<const CommandHeader> commands{headers.data(), headers.size()};
             RecordParsedEntry(header, commands);
@@ -445,6 +446,32 @@ void DmaPusher::CallMultiMethod(const u32* base_start, u32 num_methods) {
     }
 }
 
+void DmaPusher::FlushGpuWrittenCommands(GPUVAddr address, size_t size) {
+    // Title 1 runs small command lists its culling dispatches fill in each frame. Accurate GPU
+    // mode reads every command list through the caches (safe reads); here only the ones on pages
+    // the GPU has written are, or their draws run with an older frame's words (trees vanish).
+    if (!Tegra::g_nx2_ampere.load(std::memory_order_relaxed) || size == 0) {
+        return;
+    }
+    bool written = false;
+    for (GPUVAddr page = address & ~GPUVAddr{0xfff}; page < address + size; page += 0x1000) {
+        if (const auto device = memory_manager.GpuToCpuAddress(page);
+            device && Tegra::Nx2IsGpuWritten(*device)) {
+            written = true;
+            break;
+        }
+    }
+    if (!written) {
+        return;
+    }
+    // The caches belong to the split's back: it first runs what the front queued (the dispatch
+    // that writes these words among it).
+    VideoCommon::GpuSplit::Get().Drain();
+    if (memory_manager.IsMemoryDirty(address, size, VideoCommon::CacheType::BufferCache)) {
+        memory_manager.FlushRegion(address, size, VideoCommon::CacheType::BufferCache);
+    }
+}
+
 void DmaPusher::ReportUnboundSubchannel() {
     // Methods sent to a subchannel with no engine (no SetObject yet) are dropped.
     static std::atomic<u32> reported{};