[Gpu] Backend-neutral shader compiler and guest-GPU renderer seam (#200)

* [ShaderCompiler] Extract the backend-neutral shader compiler project

Move the Gen5 (gfx10) microcode decoder, the scalar evaluator, the
shader IR, and the metadata reader out of SharpEmu.Libs/Agc into a new
SharpEmu.ShaderCompiler project — the half of shader compilation every
codegen backend (SPIR-V today; MSL and DXIL later) consumes. Types go
public: they are the contract now. Nothing in the project may depend on
a host graphics API; the SPIR-V-specific artifact types
(Gen5SpirvShader, Gen5SpirvStage) stay beside the emitter in Libs.

Three couplings surfaced by the move, each resolved at the right depth:
GuestDrawKind was defined inside VulkanVideoPresenter despite being a
guest-domain, decoder-produced concept — it moves to the shared project;
the evaluator's one HLE dependency (the tracked-libc-heap read
fallback) becomes an injectable hook that a Libs module initializer
installs before any caller can reach the evaluator; and the inline-
constant table is promoted to a shared Gen5InlineConstants so backends
cannot drift on constant semantics (the SPIR-V translator now delegates
to it).

The ShaderDump tool drops its reflection over the moved types in favor
of direct typed calls; only the SPIR-V emitter, still internal to Libs
until it moves to its own backend project, is reached via reflection.
Verified by a clean solution build, the existing test suite, and a full
ShaderDump conformance run.

* [ShaderCompiler] Move the SPIR-V emitter into SharpEmu.ShaderCompiler.Vulkan

Gen5SpirvTranslator (with its ALU partial), SpirvModuleBuilder,
SpirvFixedShaders, and the Gen5SpirvShader/Gen5SpirvStage artifact types
move whole from SharpEmu.Libs/Agc into the first per-backend codegen
project. Notably it needs no Vulkan bindings reference: emitters
produce bytes from the shared IR; renderers own graphics APIs. Types go
public as the backend's contract; AgcExports and the presenter consume
them exactly as before.

The ShaderDump tool drops its last reflection: with both halves of the
pipeline public it drives decode and all three emit entry points with
direct typed calls, retiring the PadWithDefaults invoke shim — and it
no longer references SharpEmu.Libs at all, making the conformance tool
emulator-independent by design. Verified by a clean solution build, the
test suite, a full ShaderDump conformance run, and a locked-mode
restore under the pinned SDK.

* [Gpu] Extract the guest-GPU backend seam (IGuestGpuBackend)

The AGC/VideoOut/SystemService export layers now reach the renderer
through IGuestGpuBackend via GuestGpu.Current (mirroring HostPlatform),
instead of calling VulkanVideoPresenter statics. The Vulkan backend is
a thin adapter over the existing presenter, so the extraction stays
mechanical; only the adapter and the presenter itself reference the
presenter now.

The types crossing the seam move to Gpu/GuestGpuTypes.cs and drop their
Vulkan prefixes, which an audit showed were misnomers: every field is a
neutral primitive or a raw guest value (guest addresses, format and
number-type codes, CB_BLEND register bitfields, verbatim sampler
descriptor dwords). The one genuine Vulkan value in the old surface —
the Silk.NET Format inside VulkanRenderTargetFormat, which callers
never read — stops crossing: TryDecodeRenderTargetFormat is replaced at
the seam by TryGetRenderTargetOutputKind, which surfaces only the
Gen5PixelOutputKind callers actually consume, keeping native formats a
backend-internal concern. ToVulkanSampler in AgcExports is renamed
ToGuestSampler to match what it always produced.

Seam rules are documented on the interface: no host-API value crosses,
and submission stays coarse-grained with synchronization internal to
backends. Interim exception, resolved next: shader parameters are still
SPIR-V blobs.

* [Gpu] Move shader compilation behind the guest-GPU backend

The seam's interim exception is gone: AgcExports no longer calls
Gen5SpirvTranslator or handles SPIR-V bytes. IGuestGpuBackend gains the
three TryCompile entry points, which take the backend-neutral
(Gen5ShaderState, Gen5ShaderEvaluation) contract plus the flat
per-role resource-slot bases a multi-stage draw needs, and return
opaque IGuestCompiledShader handles that only the producing backend can
submit — the Vulkan backend wraps its SPIR-V in
VulkanCompiledGuestShader and rejects foreign handles loudly. Draw and
dispatch submissions take handles instead of byte arrays; the shader
caches in AgcExports store handles.

IGuestCompiledShader.Payload exposes the backend-defined compiled bytes
for exactly two callers: the diagnostics dump and the size trace —
documented as never-interpret. The unused _pixelSpirvCache is deleted.
With this, a Metal or DX12 backend plugs in by implementing
IGuestGpuBackend with its own codegen; nothing in the export layers
knows which shader format exists.

Verified by a clean solution build, the test suite, and a full
ShaderDump conformance run under the pinned SDK.

* [Gpu] Fix rename collateral from the seam extraction

Address review findings: a doc comment picked up the mechanical
VulkanVideoPresenter -> GuestGpu.Current rewrite and ended up naming
members that do not exist on the interface, and CreateVulkanIndexBuffer
kept its Vulkan prefix while every sibling factory was de-Vulkanized —
it produces the neutral GuestIndexBuffer, so it is CreateGuestIndexBuffer.

* [Gpu] Label diagnostics dumps with the backend's payload extension

Address the review's altitude finding on DumpSpirv: the dump helper's
IR-disassembly half is backend-neutral and stays put, but writing the
opaque payload to a hardcoded .spv interpreted bytes the seam says
never to interpret. IGuestCompiledShader now declares its payload's
file extension, and the renamed DumpCompiledShader takes the handle and
writes honestly-labeled dumps whichever backend produced them.

* [Gpu] Make the shader-cache hit path allocation-free and lock-free

Every translated draw built its cache key with a LINQ Select feeding
string.Join plus one interpolated string per render target — steady
per-draw allocation whether or not the shaders were already cached. The
output layout is now packed exactly into a ulong (guest slot in 6 bits
+ output kind in 2 bits per target, host locations being the byte
positions, target count in the key beside it), and the
Gen5PixelOutputBinding array is only materialized on a cache miss,
where compilation dwarfs it.

The graphics/compute shader caches switch from Dictionary guarded by
_submitTraceGate to ConcurrentDictionary, making the per-draw and
per-dispatch hit paths lock-free and decoupling them from the tracing
gate they coincidentally shared. And the seam-shaped render-target list
is built once when a translated draw is created instead of a
Select/ToArray per submission of a cached draw.

* [Gpu] Replace LINQ with explicit loops in code this branch introduced

Project rule going forward: no LINQ — it allocates enumerators,
closures, and delegates, and this codebase is GC-pause-sensitive. The
pixel-output and guest-render-target array builds and the ShaderDump
store-PC collection become plain loops; pre-existing LINQ elsewhere is
left for changes that already touch those lines.

* [ShaderCompiler] Suppress CA2255 on the evaluator hook installer

The analyzer coverage that arrived with the rebase flags
ModuleInitializer in library code; this is the rule's intended advanced
scenario — the hook must be installed before any code path can reach
the evaluator, and every such path enters through this assembly — so
suppress with that justification rather than weaken the guarantee to a
static constructor's lazier timing.

* [Gpu] Resolve rebase artifacts onto main

Dedupe the System.Collections.Concurrent using in AgcExports that the
rebase merge duplicated (main and this branch each added it), and
regenerate the lock files for the new shader-compiler projects and
SharpEmu.Libs against main's current package graph so --locked-mode
restore matches at the branch tip.

* [CI] Comment per-platform build artifact links on PRs

Adds a workflow_run workflow that, after "Build and Release" finishes a
pull-request build, posts (and keeps updated in place) a single PR
comment linking the Windows, Linux, and macOS artifacts from that run.

It runs via workflow_run rather than in the build workflow because PRs
from forks build with a read-only token that cannot comment; the
follow-on run executes in the base-repo context with write access and
without checking out fork code. GitHub only triggers workflow_run from
the default branch, so this takes effect once merged to main.
This commit is contained in:
Gutemberg Ribeiro
2026-07-15 18:11:24 +01:00
committed by GitHub
parent c69ac6ddab
commit 30fdd8d6ed
35 changed files with 1308 additions and 646 deletions

View File

@@ -4,10 +4,9 @@
// Synthetic-shader conformance dumper.
//
// Feeds hand-assembled Gen5 (gfx10) instruction words through the real
// decode -> SPIR-V pipeline (Gen5ShaderTranslator / Gen5SpirvTranslator, via
// reflection so no emulator source changes are required) and writes the
// resulting vertex, pixel, and compute SPIR-V blobs to disk. The blobs can then be
// checked with spirv-val / spirv-dis.
// decode -> SPIR-V pipeline (SharpEmu.ShaderCompiler + SharpEmu.ShaderCompiler.Vulkan)
// and writes the resulting vertex, pixel, and compute SPIR-V blobs to disk. The blobs
// can then be checked with spirv-val / spirv-dis.
//
// Programs that contain buffer_store_dword automatically get a single
// global-memory binding covering every store, which the emitter exposes as
@@ -21,9 +20,9 @@
// Usage: SharpEmu.Tools.ShaderDump [output-directory]
using System.Buffers.Binary;
using System.Reflection;
using SharpEmu.HLE;
using SharpEmu.Libs.CxxAbi;
using SharpEmu.ShaderCompiler;
using SharpEmu.ShaderCompiler.Vulkan;
const ulong ProgramAddress = 0x100000;
@@ -138,44 +137,6 @@ const ulong ProgramAddress = 0x100000;
]),
];
var assembly = typeof(CxaGuardExports).Assembly;
var shaderTranslator = assembly.GetType("SharpEmu.Libs.Agc.Gen5ShaderTranslator")
?? throw new InvalidOperationException("Gen5ShaderTranslator not found");
var spirvTranslator = assembly.GetType("SharpEmu.Libs.Agc.Gen5SpirvTranslator")
?? throw new InvalidOperationException("Gen5SpirvTranslator not found");
var describe = shaderTranslator.GetMethod(
"Describe",
BindingFlags.Public | BindingFlags.Static)
?? throw new InvalidOperationException("Gen5ShaderTranslator.Describe not found");
var tryDecode = shaderTranslator.GetMethod(
"TryDecodeProgram",
BindingFlags.NonPublic | BindingFlags.Static)
?? throw new InvalidOperationException("Gen5ShaderTranslator.TryDecodeProgram not found");
var stateType = assembly.GetType("SharpEmu.Libs.Agc.Gen5ShaderState")
?? throw new InvalidOperationException("Gen5ShaderState not found");
var evaluationType = assembly.GetType("SharpEmu.Libs.Agc.Gen5ShaderEvaluation")
?? throw new InvalidOperationException("Gen5ShaderEvaluation not found");
var imageBindingType = assembly.GetType("SharpEmu.Libs.Agc.Gen5ImageBinding")
?? throw new InvalidOperationException("Gen5ImageBinding not found");
var globalBindingType = assembly.GetType("SharpEmu.Libs.Agc.Gen5GlobalMemoryBinding")
?? throw new InvalidOperationException("Gen5GlobalMemoryBinding not found");
var pixelOutputBindingType = assembly.GetType("SharpEmu.Libs.Agc.Gen5PixelOutputBinding")
?? throw new InvalidOperationException("Gen5PixelOutputBinding not found");
var pixelOutputKindType = assembly.GetType("SharpEmu.Libs.Agc.Gen5PixelOutputKind")
?? throw new InvalidOperationException("Gen5PixelOutputKind not found");
var tryCompile = spirvTranslator.GetMethod(
"TryCompileVertexShader",
BindingFlags.Public | BindingFlags.Static)
?? throw new InvalidOperationException("Gen5SpirvTranslator.TryCompileVertexShader not found");
var tryCompilePixel = spirvTranslator.GetMethods(BindingFlags.Public | BindingFlags.Static)
.Single(method =>
method.Name == "TryCompilePixelShader" &&
method.GetParameters()[2].ParameterType.IsGenericType);
var tryCompileCompute = spirvTranslator.GetMethod(
"TryCompileComputeShader",
BindingFlags.Public | BindingFlags.Static)
?? throw new InvalidOperationException("Gen5SpirvTranslator.TryCompileComputeShader not found");
var outputDirectory = args.Length > 0
? args[0]
: Path.Combine(AppContext.BaseDirectory, "spv");
@@ -190,19 +151,18 @@ foreach (var (name, expectTranslate, words) in testPrograms)
Console.WriteLine(
$"[{name}] decode: " +
(string)describe.Invoke(null, [ctx, ProgramAddress, ProgramAddress])!);
Gen5ShaderTranslator.Describe(ctx, ProgramAddress, ProgramAddress));
object?[] decodeArgs = [ctx, ProgramAddress, null, null];
if (!(bool)tryDecode.Invoke(null, decodeArgs)!)
if (!Gen5ShaderTranslator.TryDecodeProgram(ctx, ProgramAddress, out var program, out var decodeError))
{
if (expectTranslate)
{
failures++;
Console.WriteLine($"[{name}] FAILED: decode error ({decodeArgs[3]})");
Console.WriteLine($"[{name}] FAILED: decode error ({decodeError})");
}
else
{
Console.WriteLine($"[{name}] decode failed as expected ({decodeArgs[3]})");
Console.WriteLine($"[{name}] decode failed as expected ({decodeError})");
}
continue;
@@ -219,167 +179,110 @@ foreach (var (name, expectTranslate, words) in testPrograms)
// Buffer stores need a global-memory binding; the emitter resolves them by
// instruction PC, so collect store PCs from the decoded program itself.
var programObj = decodeArgs[2]!;
var instructions = (System.Collections.IEnumerable)programObj
.GetType().GetProperty("Instructions")!.GetValue(programObj)!;
var storePcs = new List<uint>();
foreach (var instruction in instructions)
foreach (var instruction in program!.Instructions)
{
var op = (string)instruction.GetType().GetProperty("Opcode")!.GetValue(instruction)!;
if (op.StartsWith("BufferStore", StringComparison.Ordinal))
if (instruction.Opcode.StartsWith("BufferStore", StringComparison.Ordinal))
{
storePcs.Add((uint)instruction.GetType().GetProperty("Pc")!.GetValue(instruction)!);
storePcs.Add(instruction.Pc);
}
}
// The binding's scalar base (8 -> s[8:11]) must match the srsrc field of
// the hand-assembled buffer_store words, and the 64-byte backing store
// must cover every hand-assembled store offset.
var globalBindings = Array.CreateInstance(globalBindingType, storePcs.Count > 0 ? 1 : 0);
if (storePcs.Count > 0)
{
globalBindings.SetValue(
Activator.CreateInstance(
globalBindingType,
8u,
0UL,
(IReadOnlyList<uint>)storePcs,
new byte[64]),
0);
}
var globalBindings = storePcs.Count > 0
? new[] { new Gen5GlobalMemoryBinding(8u, 0UL, storePcs, new byte[64]) }
: Array.Empty<Gen5GlobalMemoryBinding>();
var state = Activator.CreateInstance(
stateType,
programObj,
new uint[16],
null,
null,
0u)!;
var evaluation = Activator.CreateInstance(
evaluationType,
var state = new Gen5ShaderState(program, new uint[16], Metadata: null);
var evaluation = new Gen5ShaderEvaluation(
new uint[256],
new uint[256],
new Dictionary<uint, IReadOnlyList<uint>>(),
Array.CreateInstance(imageBindingType, 0),
globalBindings,
null,
null,
null)!;
Array.Empty<Gen5ImageBinding>(),
globalBindings);
var compileArgs = PadWithDefaults(tryCompile, [state, evaluation, null, null]);
if ((bool)tryCompile.Invoke(null, BindingFlags.OptionalParamBinding, null, compileArgs, null)!)
if (Gen5SpirvTranslator.TryCompileVertexShader(state, evaluation, out var vertexShader, out var vertexError))
{
var shader = compileArgs[2]!;
var spirv = (byte[])shader.GetType().GetProperty("Spirv")!.GetValue(shader)!;
var path = Path.Combine(outputDirectory, $"{name}.spv");
File.WriteAllBytes(path, spirv);
Console.WriteLine($"[{name}] emit: success, {spirv.Length} bytes -> {path}");
File.WriteAllBytes(path, vertexShader.Spirv);
Console.WriteLine($"[{name}] emit: success, {vertexShader.Spirv.Length} bytes -> {path}");
}
else
{
failures++;
Console.WriteLine($"[{name}] emit: FAILED ({compileArgs[3]})");
Console.WriteLine($"[{name}] emit: FAILED ({vertexError})");
}
var computeArgs = PadWithDefaults(tryCompileCompute, [state, evaluation, 1u, 1u, 1u, null, null]);
if ((bool)tryCompileCompute.Invoke(null, BindingFlags.OptionalParamBinding, null, computeArgs, null)!)
if (Gen5SpirvTranslator.TryCompileComputeShader(state, evaluation, 1, 1, 1, out var computeShader, out var computeError))
{
var shader = computeArgs[5]!;
var spirv = (byte[])shader.GetType().GetProperty("Spirv")!.GetValue(shader)!;
var path = Path.Combine(outputDirectory, $"{name}-cs.spv");
File.WriteAllBytes(path, spirv);
Console.WriteLine($"[{name}] compute emit: success, {spirv.Length} bytes -> {path}");
File.WriteAllBytes(path, computeShader.Spirv);
Console.WriteLine($"[{name}] compute emit: success, {computeShader.Spirv.Length} bytes -> {path}");
}
else
{
failures++;
Console.WriteLine($"[{name}] compute emit: FAILED ({computeArgs[6]})");
Console.WriteLine($"[{name}] compute emit: FAILED ({computeError})");
}
if (name.StartsWith("mrt", StringComparison.Ordinal))
{
(uint GuestSlot, uint HostLocation, string Kind)[] outputSpecs = name switch
Gen5PixelOutputBinding[] pixelOutputs = name switch
{
"mrt" => new (uint GuestSlot, uint HostLocation, string Kind)[]
{
(0, 0, "Float"),
(3, 1, "Uint"),
(6, 2, "Sint"),
},
"mrt-float2" => [(0, 0, "Float"), (1, 1, "Float")],
"mrt8" => Enumerable.Range(0, 8)
.Select(index => ((uint)index, (uint)index, "Float"))
.ToArray(),
_ => [(0, 0, "Float")],
"mrt" =>
[
new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float),
new Gen5PixelOutputBinding(3, 1, Gen5PixelOutputKind.Uint),
new Gen5PixelOutputBinding(6, 2, Gen5PixelOutputKind.Sint),
],
"mrt-float2" =>
[
new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float),
new Gen5PixelOutputBinding(1, 1, Gen5PixelOutputKind.Float),
],
"mrt8" =>
[
new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float),
new Gen5PixelOutputBinding(1, 1, Gen5PixelOutputKind.Float),
new Gen5PixelOutputBinding(2, 2, Gen5PixelOutputKind.Float),
new Gen5PixelOutputBinding(3, 3, Gen5PixelOutputKind.Float),
new Gen5PixelOutputBinding(4, 4, Gen5PixelOutputKind.Float),
new Gen5PixelOutputBinding(5, 5, Gen5PixelOutputKind.Float),
new Gen5PixelOutputBinding(6, 6, Gen5PixelOutputKind.Float),
new Gen5PixelOutputBinding(7, 7, Gen5PixelOutputKind.Float),
],
_ => [new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float)],
};
var pixelOutputs = Array.CreateInstance(pixelOutputBindingType, outputSpecs.Length);
for (var index = 0; index < outputSpecs.Length; index++)
{
var spec = outputSpecs[index];
pixelOutputs.SetValue(
Activator.CreateInstance(
pixelOutputBindingType,
spec.GuestSlot,
spec.HostLocation,
Enum.Parse(pixelOutputKindType, spec.Kind)),
index);
}
var pixelArgs = PadWithDefaults(
tryCompilePixel,
[state, evaluation, pixelOutputs, null, null]);
if ((bool)tryCompilePixel.Invoke(
null,
BindingFlags.OptionalParamBinding,
null,
pixelArgs,
null)!)
if (Gen5SpirvTranslator.TryCompilePixelShader(state, evaluation, pixelOutputs, out var pixelShader, out var pixelError))
{
var shader = pixelArgs[3]!;
var spirv = (byte[])shader.GetType().GetProperty("Spirv")!.GetValue(shader)!;
var path = Path.Combine(outputDirectory, $"{name}-ps.spv");
File.WriteAllBytes(path, spirv);
Console.WriteLine($"[{name}] pixel emit: success, {spirv.Length} bytes -> {path}");
File.WriteAllBytes(path, pixelShader.Spirv);
Console.WriteLine($"[{name}] pixel emit: success, {pixelShader.Spirv.Length} bytes -> {path}");
}
else
{
failures++;
Console.WriteLine($"[{name}] pixel emit: FAILED ({pixelArgs[4]})");
Console.WriteLine($"[{name}] pixel emit: FAILED ({pixelError})");
}
if (name == "mrt")
{
var invalidOutputs = Array.CreateInstance(pixelOutputBindingType, 2);
invalidOutputs.SetValue(
Activator.CreateInstance(
pixelOutputBindingType,
0u,
0u,
Enum.Parse(pixelOutputKindType, "Float")),
0);
invalidOutputs.SetValue(
Activator.CreateInstance(
pixelOutputBindingType,
3u,
7u,
Enum.Parse(pixelOutputKindType, "Float")),
1);
var invalidPixelArgs = PadWithDefaults(
tryCompilePixel,
[state, evaluation, invalidOutputs, null, null]);
if ((bool)tryCompilePixel.Invoke(
null,
BindingFlags.OptionalParamBinding,
null,
invalidPixelArgs,
null)!)
Gen5PixelOutputBinding[] invalidOutputs =
[
new Gen5PixelOutputBinding(0, 0, Gen5PixelOutputKind.Float),
new Gen5PixelOutputBinding(3, 7, Gen5PixelOutputKind.Float),
];
if (Gen5SpirvTranslator.TryCompilePixelShader(state, evaluation, invalidOutputs, out _, out var invalidError))
{
failures++;
Console.WriteLine("[mrt] FAILED: sparse host locations were accepted");
}
else
{
Console.WriteLine($"[mrt] sparse host locations rejected as expected ({invalidPixelArgs[4]})");
Console.WriteLine($"[mrt] sparse host locations rejected as expected ({invalidError})");
}
}
}
@@ -390,37 +293,6 @@ Console.WriteLine(failures == 0
: $"RESULT: {failures} unexpected outcome(s)");
Environment.ExitCode = failures == 0 ? 0 : 1;
// Reflection Invoke does not apply C# default parameter values, so a newly
// added optional parameter on a translator entry point would otherwise throw
// TargetParameterCountException. Type.Missing + OptionalParamBinding lets the
// runtime substitute the declared defaults; only a new *required* parameter
// should force a tool update.
static object?[] PadWithDefaults(MethodInfo method, object?[] arguments)
{
var parameters = method.GetParameters();
if (arguments.Length > parameters.Length)
{
throw new InvalidOperationException(
$"{method.DeclaringType?.Name}.{method.Name} takes fewer parameters than the tool supplies");
}
var padded = new object?[parameters.Length];
arguments.CopyTo(padded, 0);
for (var i = arguments.Length; i < padded.Length; i++)
{
if (!parameters[i].IsOptional)
{
throw new InvalidOperationException(
$"{method.DeclaringType?.Name}.{method.Name} gained a required parameter " +
$"'{parameters[i].Name}' — the tool needs updating");
}
padded[i] = Type.Missing;
}
return padded;
}
internal sealed class FakeMemory : ICpuMemory
{
private readonly List<(ulong Base, byte[] Data)> _regions = [];