[video_core, maxwell] fixes for homebrew games that use MESA compiler (#4012)

two main issues:

shader_environment.cpp

Support Mesa nv50_ir-compiled homebrew fragment shaders
Mesa nv50_ir (used by NZP-Switch via libdrm_nouveau) emits Maxwell SASS that differs from NVN in two ways our shader pipeline didn't account for, causing "infinite" loops during shader scanning.

How?  TryFindSize() previously detected end-of-shader only via NVN's "BRA $-1" self-branch trailer (SELF_BRANCH_A/B). Mesa-compiled shaders end with an unconditional @PT EXIT and emit no trailer, so scanning ran past the shader and looped to MAXIMUM_SIZE.

Fix: Added a secondary match against @PT EXIT T (opcode 0xE30, predicate PT, flow T) as a fallback terminator. The mask is estrictive enough to reject predicated EXITs, conditional-flow EXITs, and sched control words, so well-formed NVN shaders see no ehavior change (their single @PT EXIT immediately precedes BRA $-1, and both detections return the same size).

load_store_attribute.cpp + attribute.h

IPA's is_perspective check only applied the ×position_w correction for IR::IsGeneric() attributes, and used a per-component SPH lookup. Two failure modes followed:

1. Mesa fragment shaders read varyings via legacy attribute slots
(ColorFrontDiffuse, FixedFncTexture, FogCoordinate) which are
remapped to generic varyings later by ConvertLegacyToGeneric. At
IPA-translation time they're still legacy, so IsGeneric() was
false and the inject was skipped — but the resulting GLSL/SPIR-V
varying was Smooth (perspective-correct), and the SASS still
issued its manual perspective dance via MUFU.RCP(ATTR_W)+FMUL,
compounding an extra ×clip_w factor. NZP fog (eye_z distance
varying) ended up as eye_z⁴×density² instead of eye_z²×density²,
saturating fog to white everywhere.

2. Even for legacy attrs that should inject, Mesa stores them in two formats selected by the IPA's interpolation_mode field:
Pass/Multiply/Constant → attr/w (perspective); inject ×position_w
Sc (ScreenLinear)      → raw attr; do NOT inject
Unconditionally injecting for Sc dims vertex colors by 1/clip_w because there's no SASS multiplier to round-trip it back.

Fix: extend the IPA correction path to also fire on IsLegacyAttribute(), gated on interpolation_mode != Sc. While here, the generic-path check is widened from per-component to vector-level (first non-Unused PixelImap, with all-Unused fallthrough treated as Perspective) so it matches the GLSL interpolation qualifier picked by CollectInterpolationInfo — this fixes the same all-zeroed-imap case Mesa exposes for explicit generics. IsLegacyAttribute moved from translate_program.cpp's anonymous namespace into attribute.h next to IsGeneric so IPA can reach it.

Tested on NaziZombies:Portable (Switch homebrew): fog now matches real
hardware, vertex-colored geometry (hand, gun, decals) renders correctly.

Reviewed-on: https://git.eden-emu.dev/eden-emu/eden/pulls/4012
Reviewed-by: Lizzie <lizzie@eden-emu.dev>
Reviewed-by: MaranBr <maranbr@eden-emu.dev>

1 files changed, 6 insertions, 0 deletions

diff --git a/src/video_core/shader_environment.cpp b/src/video_core/shader_environment.cpp
index 29a79810cd..2f21a6f492 100644
--- a/src/video_core/shader_environment.cpp
+++ b/src/video_core/shader_environment.cpp
@@ -254,6 +254,9 @@ std::optional<u64> GenericEnvironment::TryFindSize() {
     static constexpr u64 SELF_BRANCH_A = 0xE2400FFFFF87000FULL;
     static constexpr u64 SELF_BRANCH_B = 0xE2400FFFFF07000FULL;
 
+    static constexpr u64 MESA_EXIT_MASK = 0xFFF00000000F001FULL;
+    static constexpr u64 MESA_EXIT_VALUE = (0xE30ULL << 52) | (0x7ULL << 16) | 0xFULL;
+
     code.resize(MAXIMUM_SIZE / INST_SIZE);
 
     GPUVAddr guest_addr{program_base + start_address};
@@ -267,6 +270,9 @@ std::optional<u64> GenericEnvironment::TryFindSize() {
             if (inst == SELF_BRANCH_A || inst == SELF_BRANCH_B) {
                 return offset + index;
             }
+            if ((inst & MESA_EXIT_MASK) == MESA_EXIT_VALUE) {
+                return offset + index + INST_SIZE;
+            }
         }
         guest_addr += BLOCK_SIZE;
         size += BLOCK_SIZE;