emu-mirrors/Ryujinx
0
0
Fork 0
mirror of https://github.com/GreemDev/Ryujinx.git synced 2024-12-22 18:05:46 +00:00
Code Wiki Activity

Memory Read/Write Tracking using Region Handles (#1272)

Browse source 
* WIP Range Tracking

- Texture invalidation seems to have large problems
- Buffer/Pool invalidation may have problems
- Mirror memory tracking puts an additional `add` in compiled code, we likely just want to make HLE access slower if this is the final solution.
- Native project is in the messiest possible location.
- [HACK] JIT memory access always uses native "fast" path
- [HACK] Trying some things with texture invalidation and views.

It works :)

Still a few hacks, messy things, slow things

More work in progress stuff (also move to memory project)

Quite a bit faster now.
- Unmapping GPU VA and CPU VA will now correctly update write tracking regions, and invalidate textures for the former.
- The Virtual range list is now non-overlapping like the physical one.
- Fixed some bugs where regions could leak.
- Introduced a weird bug that I still need to track down (consistent invalid buffer in MK8 ribbon road)

Move some stuff.

I think we'll eventually just put the dll and so for this in a nuget package.

Fix rebase.

[WIP] MultiRegionHandle variable size ranges

- Avoid reprotecting regions that change often (needs some tweaking)
- There's still a bug in buffers, somehow.
- Might want different api for minimum granularity

Fix rebase issue

Commit everything needed for software only tracking.

Remove native components.

Remove more native stuff.

Cleanup

Use a separate window for the background context, update opentk. (fixes linux)

Some experimental changes

Should get things working up to scratch - still need to try some things with flush/modification and res scale.

Include address with the region action.

Initial work to make range tracking work

Still a ton of bugs

Fix some issues with the new stuff.

* Fix texture flush instability

There's still some weird behaviour, but it's much improved without this. (textures with cpu modified data were flushing over it)

* Find the destination texture for Buffer->Texture full copy

Greatly improves performance for nvdec videos (with range tracking)

* Further improve texture tracking

* Disable Memory Tracking for view parents

This is a temporary approach to better match behaviour on master (where invalidations would be soaked up by views, rather than trigger twice)

The assumption is that when views are created to a texture, they will cover all of its data anyways. Of course, this can easily be improved in future.

* Introduce some tracking tests.

WIP

* Complete base tests.

* Add more tests for multiregion, fix existing test.

* Cleanup Part 1

* Remove unnecessary code from memory tracking

* Fix some inconsistencies with 3D texture rule.

* Add dispose tests.

* Use a background thread for the background context.

Rather than setting and unsetting a context as current, doing the work on a dedicated thread with signals seems to be a bit faster.

Also nerf the multithreading test a bit.

* Copy to texture with matching alignment

This extends the copy to work for some videos with unusual size, such as tutorial videos in SMO. It will only occur if the destination texture already exists at XCount size.

* Track reads for buffer copies. Synchronize new buffers before copying overlaps.

* Remove old texture flushing mechanisms.

Range tracking all the way, baby.

* Wake the background thread when disposing.

Avoids a deadlock when games are closed.

* Address Feedback 1

* Separate TextureCopy instance for background thread

Also `BackgroundContextWorker.InBackground` for a more sensible idenfifier for if we're in a background thread.

* Add missing XML docs.

* Address Feedback

* Maybe I should start drinking coffee.

* Some more feedback.

* Remove flush warning, Refocus window after making background context
This commit is contained in:
riperiperi 2020-10-16 21:18:35 +01:00 committed by GitHub
parent f02791b20c
commit b4d8d893a4
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
50 changed files with 2973 additions and 441 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

23
ARMeilleure/Instructions/InstEmitMemoryHelper.cs
View file

@ -391,23 +391,22 @@ namespace ARMeilleure.Instructions
if (lblSlowPath != null)
{
context.BranchIf(lblSlowPath, pte, Const(0L), Comparison.LessOrEqual);
ulong protection = (write ? 3UL : 1UL) << 48;
context.BranchIfTrue(lblSlowPath, context.BitwiseAnd(pte, Const(protection)));
}
else
{
// When no label is provided to jump to a slow path if the address is invalid,
// we do the validation ourselves, and throw if needed.
if (write)
{
Operand lblNotWatched = Label();
// Is the page currently being monitored for modifications? If so we need to call MarkRegionAsModified.
context.BranchIf(lblNotWatched, pte, Const(0L), Comparison.GreaterOrEqual, BasicBlockFrequency.Cold);
Operand lblNotWatched = Label();
// Mark the region as modified. Size here doesn't matter as address is assumed to be size aligned here.
context.Call(typeof(NativeInterface).GetMethod(nameof(NativeInterface.MarkRegionAsModified)), address, Const(1UL));
context.MarkLabel(lblNotWatched);
}
// Is the page currently being tracked for read/write? If so we need to call MarkRegionAsModified.
context.BranchIf(lblNotWatched, pte, Const(0L), Comparison.GreaterOrEqual, BasicBlockFrequency.Cold);
// Mark the region as modified. Size here doesn't matter as address is assumed to be size aligned here.
context.Call(typeof(NativeInterface).GetMethod(nameof(NativeInterface.SignalMemoryTracking)), address, Const(1UL), Const(write ? 1 : 0));
context.MarkLabel(lblNotWatched);
Operand lblNonNull = Label();
@ -417,10 +416,10 @@ namespace ARMeilleure.Instructions
// The call is not expected to return (it should throw).
context.Call(typeof(NativeInterface).GetMethod(nameof(NativeInterface.ThrowInvalidMemoryAccess)), address);
context.MarkLabel(lblNonNull);
pte = context.BitwiseAnd(pte, Const(0xffffffffffffUL));
}
pte = context.BitwiseAnd(pte, Const(0xffffffffffffUL)); // Ignore any software protection bits. (they are still used by c# memory access)
Operand pageOffset = context.BitwiseAnd(address, Const(address.Type, PageMask));
if (pageOffset.Type == OperandType.I32)

14
ARMeilleure/Instructions/NativeInterface.cs
View file

@ -163,27 +163,27 @@ namespace ARMeilleure.Instructions
#region "Read"
public static byte ReadByte(ulong address)
{
return GetMemoryManager().Read<byte>(address);
return GetMemoryManager().ReadTracked<byte>(address);
}
public static ushort ReadUInt16(ulong address)
{
return GetMemoryManager().Read<ushort>(address);
return GetMemoryManager().ReadTracked<ushort>(address);
}
public static uint ReadUInt32(ulong address)
{
return GetMemoryManager().Read<uint>(address);
return GetMemoryManager().ReadTracked<uint>(address);
}
public static ulong ReadUInt64(ulong address)
{
return GetMemoryManager().Read<ulong>(address);
return GetMemoryManager().ReadTracked<ulong>(address);
}
public static V128 ReadVector128(ulong address)
{
return GetMemoryManager().Read<V128>(address);
return GetMemoryManager().ReadTracked<V128>(address);
}
#endregion
@ -214,9 +214,9 @@ namespace ARMeilleure.Instructions
}
#endregion
public static void MarkRegionAsModified(ulong address, ulong size)
public static void SignalMemoryTracking(ulong address, ulong size, bool write)
{
GetMemoryManager().MarkRegionAsModified(address, size);
GetMemoryManager().SignalMemoryTracking(address, size, write);
}
public static void ThrowInvalidMemoryAccess(ulong address)

3
ARMeilleure/Memory/IMemoryManager.cs
View file

@ -9,12 +9,13 @@ namespace ARMeilleure.Memory
IntPtr PageTablePointer { get; }
T Read<T>(ulong va) where T : unmanaged;
T ReadTracked<T>(ulong va) where T : unmanaged;
void Write<T>(ulong va, T value) where T : unmanaged;
ref T GetRef<T>(ulong va) where T : unmanaged;
bool IsMapped(ulong va);
void MarkRegionAsModified(ulong va, ulong size);
void SignalMemoryTracking(ulong va, ulong size, bool write);
}
}

2
ARMeilleure/Translation/Delegates.cs
View file

@ -117,7 +117,6 @@ namespace ARMeilleure.Translation
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.GetTpidr32))); // A32 only.
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.GetTpidrEl0)));
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.GetTpidrEl032))); // A32 only.
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.MarkRegionAsModified)));
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.ReadByte)));
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.ReadUInt16)));
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.ReadUInt32)));
@ -129,6 +128,7 @@ namespace ARMeilleure.Translation
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.SetFpsrQc))); // A32 only.
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.SetTpidrEl0)));
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.SetTpidrEl032))); // A32 only.
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.SignalMemoryTracking)));
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.SupervisorCall)));
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.ThrowInvalidMemoryAccess)));
SetDelegateInfo(typeof(NativeInterface).GetMethod(nameof(NativeInterface.Undefined)));

2
ARMeilleure/Translation/PTC/Ptc.cs
View file

@ -21,7 +21,7 @@ namespace ARMeilleure.Translation.PTC
{
private const string HeaderMagic = "PTChd";
private const int InternalVersion = 1577; //! To be incremented manually for each change to the ARMeilleure project.
private const int InternalVersion = 1273; //! To be incremented manually for each change to the ARMeilleure project.
private const string ActualDir = "0";
private const string BackupDir = "1";

2
Ryujinx.Audio/Ryujinx.Audio.csproj
View file

@ -16,7 +16,7 @@
</PropertyGroup>
<ItemGroup>
<PackageReference Include="OpenTK.NetStandard" Version="1.0.5.12" />
<PackageReference Include="OpenTK.NetStandard" Version="1.0.5.22" />
</ItemGroup>
<ItemGroup>

293
Ryujinx.Cpu/MemoryManager.cs
View file

@ -1,6 +1,9 @@
using ARMeilleure.Memory;
using ARMeilleure.Memory;
using Ryujinx.Cpu.Tracking;
using Ryujinx.Memory;
using Ryujinx.Memory.Tracking;
using System;
using System.Collections.Generic;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using System.Threading;
@ -10,7 +13,7 @@ namespace Ryujinx.Cpu
/// <summary>
/// Represents a CPU memory manager.
/// </summary>
public sealed class MemoryManager : IMemoryManager, IDisposable
public sealed class MemoryManager : IMemoryManager, IDisposable, IVirtualMemoryManager
{
public const int PageBits = 12;
public const int PageSize = 1 << PageBits;
@ -35,6 +38,8 @@ namespace Ryujinx.Cpu
/// </summary>
public IntPtr PageTablePointer => _pageTable.Pointer;
public MemoryTracking Tracking { get; }
/// <summary>
/// Creates a new instance of the memory manager.
/// </summary>
@ -58,6 +63,9 @@ namespace Ryujinx.Cpu
_addressSpaceSize = asSize;
_backingMemory = backingMemory;
_pageTable = new MemoryBlock((asSize / PageSize) * PteSize);
Tracking = new MemoryTracking(this, backingMemory, PageSize);
Tracking.EnablePhysicalProtection = false; // Disabled for now, as protection is done in software.
}
/// <summary>
@ -71,14 +79,18 @@ namespace Ryujinx.Cpu
/// <param name="size">Size to be mapped</param>
public void Map(ulong va, ulong pa, ulong size)
{
while (size != 0)
ulong remainingSize = size;
ulong oVa = va;
ulong oPa = pa;
while (remainingSize != 0)
{
_pageTable.Write((va / PageSize) * PteSize, PaToPte(pa));
va += PageSize;
pa += PageSize;
size -= PageSize;
remainingSize -= PageSize;
}
Tracking.Map(oVa, oPa, size);
}
/// <summary>
@ -88,13 +100,16 @@ namespace Ryujinx.Cpu
/// <param name="size">Size of the range to be unmapped</param>
public void Unmap(ulong va, ulong size)
{
while (size != 0)
ulong remainingSize = size;
ulong oVa = va;
while (remainingSize != 0)
{
_pageTable.Write((va / PageSize) * PteSize, 0UL);
va += PageSize;
size -= PageSize;
remainingSize -= PageSize;
}
Tracking.Unmap(oVa, size);
}
/// <summary>
@ -109,6 +124,18 @@ namespace Ryujinx.Cpu
return MemoryMarshal.Cast<byte, T>(GetSpan(va, Unsafe.SizeOf<T>()))[0];
}
/// <summary>
/// Reads data from CPU mapped memory, with read tracking
/// </summary>
/// <typeparam name="T">Type of the data being read</typeparam>
/// <param name="va">Virtual address of the data in memory</param>
/// <returns>The data</returns>
public T ReadTracked<T>(ulong va) where T : unmanaged
{
SignalMemoryTracking(va, (ulong)Unsafe.SizeOf<T>(), false);
return MemoryMarshal.Cast<byte, T>(GetSpan(va, Unsafe.SizeOf<T>()))[0];
}
/// <summary>
/// Reads data from CPU mapped memory.
/// </summary>
@ -133,7 +160,7 @@ namespace Ryujinx.Cpu
}
/// <summary>
/// Writes data to CPU mapped memory.
/// Writes data to CPU mapped memory, with write tracking.
/// </summary>
/// <param name="va">Virtual address to write the data into</param>
/// <param name="data">Data to be written</param>
@ -145,13 +172,13 @@ namespace Ryujinx.Cpu
return;
}
MarkRegionAsModified(va, (ulong)data.Length);
SignalMemoryTracking(va, (ulong)data.Length, true);
WriteImpl(va, data);
}
/// <summary>
/// Writes data to CPU mapped memory, without tracking.
/// Writes data to CPU mapped memory, without write tracking.
/// </summary>
/// <param name="va">Virtual address to write the data into</param>
/// <param name="data">Data to be written</param>
@ -222,15 +249,21 @@ namespace Ryujinx.Cpu
/// </remarks>
/// <param name="va">Virtual address of the data</param>
/// <param name="size">Size of the data</param>
/// <param name="tracked">True if read tracking is triggered on the span</param>
/// <returns>A read-only span of the data</returns>
/// <exception cref="InvalidMemoryRegionException">Throw for unhandled invalid or unmapped memory accesses</exception>
public ReadOnlySpan<byte> GetSpan(ulong va, int size)
public ReadOnlySpan<byte> GetSpan(ulong va, int size, bool tracked = false)
{
if (size == 0)
{
return ReadOnlySpan<byte>.Empty;
}
if (tracked)
{
SignalMemoryTracking(va, (ulong)size, false);
}
if (IsContiguousAndMapped(va, size))
{
return _backingMemory.GetSpan(GetPhysicalAddressInternal(va), size);
@ -295,7 +328,7 @@ namespace Ryujinx.Cpu
ThrowMemoryNotContiguous();
}
MarkRegionAsModified(va, (ulong)Unsafe.SizeOf<T>());
SignalMemoryTracking(va, (ulong)Unsafe.SizeOf<T>(), true);
return ref _backingMemory.GetRef<T>(GetPhysicalAddressInternal(va));
}
@ -337,6 +370,56 @@ namespace Ryujinx.Cpu
return true;
}
/// <summary>
/// Gets the physical regions that make up the given virtual address region.
/// If any part of the virtual region is unmapped, null is returned.
/// </summary>
/// <param name="va">Virtual address of the range</param>
/// <param name="size">Size of the range</param>
/// <returns>Array of physical regions</returns>
public (ulong address, ulong size)[] GetPhysicalRegions(ulong va, ulong size)
{
if (!ValidateAddress(va))
{
return null;
}
ulong endVa = (va + size + PageMask) & ~(ulong)PageMask;
va &= ~(ulong)PageMask;
int pages = (int)((endVa - va) / PageSize);
List<(ulong, ulong)> regions = new List<(ulong, ulong)>();
ulong regionStart = GetPhysicalAddressInternal(va);
ulong regionSize = PageSize;
for (int page = 0; page < pages - 1; page++)
{
if (!ValidateAddress(va + PageSize))
{
return null;
}
ulong newPa = GetPhysicalAddressInternal(va + PageSize);
if (GetPhysicalAddressInternal(va) + PageSize != newPa)
{
regions.Add((regionStart, regionSize));
regionStart = newPa;
regionSize = 0;
}
va += PageSize;
regionSize += PageSize;
}
regions.Add((regionStart, regionSize));
return regions.ToArray();
}
private void ReadImpl(ulong va, Span<byte> data)
{
if (data.Length == 0)
@ -377,99 +460,6 @@ namespace Ryujinx.Cpu
}
}
/// <summary>
/// Checks if a specified virtual memory region has been modified by the CPU since the last call.
/// </summary>
/// <param name="va">Virtual address of the region</param>
/// <param name="size">Size of the region</param>
/// <param name="id">Resource identifier number (maximum is 15)</param>
/// <param name="modifiedRanges">Optional array where the modified ranges should be written</param>
/// <returns>The number of modified ranges</returns>
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public int QueryModified(ulong va, ulong size, int id, (ulong, ulong)[] modifiedRanges = null)
{
if (!ValidateAddress(va))
{
return 0;
}
ulong maxSize = _addressSpaceSize - va;
if (size > maxSize)
{
size = maxSize;
}
// We need to ensure that the tagged pointer value is negative,
// JIT generated code checks that to take the slow paths and call the MemoryManager Read/Write methods.
long tag = (0x8000L | (1L << id)) << 48;
ulong endVa = (va + size + PageMask) & ~(ulong)PageMask;
va &= ~(ulong)PageMask;
ulong rgStart = va;
ulong rgSize = 0;
int rangeIndex = 0;
for (; va < endVa; va += PageSize)
{
while (true)
{
ref long pte = ref _pageTable.GetRef<long>((va >> PageBits) * PteSize);
long pteValue = pte;
// If the PTE value is 0, that means that the page is unmapped.
// We behave as if the page was not modified, since modifying a page
// that is not even mapped is impossible.
if ((pteValue & tag) == tag || pteValue == 0)
{
if (rgSize != 0)
{
if (modifiedRanges != null && rangeIndex < modifiedRanges.Length)
{
modifiedRanges[rangeIndex] = (rgStart, rgSize);
}
rangeIndex++;
rgSize = 0;
}
break;
}
else
{
if (Interlocked.CompareExchange(ref pte, pteValue | tag, pteValue) == pteValue)
{
if (rgSize == 0)
{
rgStart = va;
}
rgSize += PageSize;
break;
}
}
}
}
if (rgSize != 0)
{
if (modifiedRanges != null && rangeIndex < modifiedRanges.Length)
{
modifiedRanges[rangeIndex] = (rgStart, rgSize);
}
rangeIndex++;
}
return rangeIndex;
}
/// <summary>
/// Checks if the page at a given CPU virtual address.
/// </summary>
@ -516,13 +506,24 @@ namespace Ryujinx.Cpu
}
/// <summary>
/// Marks a region of memory as modified by the CPU.
/// Reprotect a region of virtual memory for tracking. Sets software protection bits.
/// </summary>
/// <param name="va">Virtual address of the region</param>
/// <param name="size">Size of the region</param>
public void MarkRegionAsModified(ulong va, ulong size)
/// <param name="va">Virtual address base</param>
/// <param name="size">Size of the region to protect</param>
/// <param name="protection">Memory protection to set</param>
public void TrackingReprotect(ulong va, ulong size, MemoryPermission protection)
{
// Protection is inverted on software pages, since the default value is 0.
protection = (~protection) & MemoryPermission.ReadAndWrite;
long tag = (long)protection << 48;
if (tag > 0)
{
tag |= long.MinValue; // If any protection is present, the whole pte is negative.
}
ulong endVa = (va + size + PageMask) & ~(ulong)PageMask;
long invTagMask = ~(0xffffL << 48);
while (va < endVa)
{
@ -533,13 +534,77 @@ namespace Ryujinx.Cpu
do
{
pte = Volatile.Read(ref pageRef);
if (pte >= 0)
{
break;
}
}
while (Interlocked.CompareExchange(ref pageRef, pte & ~(0xffffL << 48), pte) != pte);
while (Interlocked.CompareExchange(ref pageRef, (pte & invTagMask) | tag, pte) != pte);
va += PageSize;
}
}
/// <summary>
/// Obtains a memory tracking handle for the given virtual region. This should be disposed when finished with.
/// </summary>
/// <param name="address">CPU virtual address of the region</param>
/// <param name="size">Size of the region</param>
/// <returns>The memory tracking handle</returns>
public CpuRegionHandle BeginTracking(ulong address, ulong size)
{
return new CpuRegionHandle(Tracking.BeginTracking(address, size));
}
/// <summary>
/// Obtains a memory tracking handle for the given virtual region, with a specified granularity. This should be disposed when finished with.
/// </summary>
/// <param name="address">CPU virtual address of the region</param>
/// <param name="size">Size of the region</param>
/// <param name="granularity">Desired granularity of write tracking</param>
/// <returns>The memory tracking handle</returns>
public CpuMultiRegionHandle BeginGranularTracking(ulong address, ulong size, ulong granularity)
{
return new CpuMultiRegionHandle(Tracking.BeginGranularTracking(address, size, granularity));
}
/// <summary>
/// Obtains a smart memory tracking handle for the given virtual region, with a specified granularity. This should be disposed when finished with.
/// </summary>
/// <param name="address">CPU virtual address of the region</param>
/// <param name="size">Size of the region</param>
/// <param name="granularity">Desired granularity of write tracking</param>
/// <returns>The memory tracking handle</returns>
public CpuSmartMultiRegionHandle BeginSmartGranularTracking(ulong address, ulong size, ulong granularity)
{
return new CpuSmartMultiRegionHandle(Tracking.BeginSmartGranularTracking(address, size, granularity));
}
/// <summary>
/// Alerts the memory tracking that a given region has been read from or written to.
/// This should be called before read/write is performed.
/// </summary>
/// <param name="va">Virtual address of the region</param>
/// <param name="size">Size of the region</param>
public void SignalMemoryTracking(ulong va, ulong size, bool write)
{
// We emulate guard pages for software memory access. This makes for an easy transition to
// tracking using host guard pages in future, but also supporting platforms where this is not possible.
// Write tag includes read protection, since we don't have any read actions that aren't performed before write too.
long tag = (write ? 3L : 1L) << 48;
ulong endVa = (va + size + PageMask) & ~(ulong)PageMask;
while (va < endVa)
{
ref long pageRef = ref _pageTable.GetRef<long>((va >> PageBits) * PteSize);
long pte;
pte = Volatile.Read(ref pageRef);
if ((pte & tag) != 0)
{
Tracking.VirtualMemoryEvent(va, size, write);
break;
}
va += PageSize;
}

23
Ryujinx.Cpu/Tracking/CpuMultiRegionHandle.cs Normal file
View file

@ -0,0 +1,23 @@
using Ryujinx.Memory.Tracking;
using System;
namespace Ryujinx.Cpu.Tracking
{
public class CpuMultiRegionHandle : IMultiRegionHandle
{
private readonly MultiRegionHandle _impl;
public bool Dirty => _impl.Dirty;
internal CpuMultiRegionHandle(MultiRegionHandle impl)
{
_impl = impl;
}
public void Dispose() => _impl.Dispose();
public void QueryModified(Action<ulong, ulong> modifiedAction) => _impl.QueryModified(modifiedAction);
public void QueryModified(ulong address, ulong size, Action<ulong, ulong> modifiedAction) => _impl.QueryModified(address, size, modifiedAction);
public void QueryModified(ulong address, ulong size, Action<ulong, ulong> modifiedAction, int sequenceNumber) => _impl.QueryModified(address, size, modifiedAction, sequenceNumber);
public void SignalWrite() => _impl.SignalWrite();
}
}

23
Ryujinx.Cpu/Tracking/CpuRegionHandle.cs Normal file
View file

@ -0,0 +1,23 @@
using Ryujinx.Memory.Tracking;
namespace Ryujinx.Cpu.Tracking
{
public class CpuRegionHandle : IRegionHandle
{
private readonly RegionHandle _impl;
public bool Dirty => _impl.Dirty;
public ulong Address => _impl.Address;
public ulong Size => _impl.Size;
public ulong EndAddress => _impl.EndAddress;
internal CpuRegionHandle(RegionHandle impl)
{
_impl = impl;
}
public void Dispose() => _impl.Dispose();
public void RegisterAction(RegionSignal action) => _impl.RegisterAction(action);
public void Reprotect() => _impl.Reprotect();
}
}

23
Ryujinx.Cpu/Tracking/CpuSmartMultiRegionHandle.cs Normal file
View file

@ -0,0 +1,23 @@
using Ryujinx.Memory.Tracking;
using System;
namespace Ryujinx.Cpu.Tracking
{
public class CpuSmartMultiRegionHandle : IMultiRegionHandle
{
private readonly SmartMultiRegionHandle _impl;
public bool Dirty => _impl.Dirty;
internal CpuSmartMultiRegionHandle(SmartMultiRegionHandle impl)
{
_impl = impl;
}
public void Dispose() => _impl.Dispose();
public void QueryModified(Action<ulong, ulong> modifiedAction) => _impl.QueryModified(modifiedAction);
public void QueryModified(ulong address, ulong size, Action<ulong, ulong> modifiedAction) => _impl.QueryModified(address, size, modifiedAction);
public void QueryModified(ulong address, ulong size, Action<ulong, ulong> modifiedAction, int sequenceNumber) => _impl.QueryModified(address, size, modifiedAction, sequenceNumber);
public void SignalWrite() => _impl.SignalWrite();
}
}

2
Ryujinx.Graphics.GAL/IRenderer.cs
View file

@ -10,6 +10,8 @@ namespace Ryujinx.Graphics.GAL
IWindow Window { get; }
void BackgroundContextAction(Action action);
IShader CompileShader(ShaderProgram shader);
BufferHandle CreateBuffer(int size);

3
Ryujinx.Graphics.Gpu/Engine/Inline2Memory.cs
View file

@ -42,7 +42,8 @@ namespace Ryujinx.Graphics.Gpu.Engine
ulong dstBaseAddress = _context.MemoryManager.Translate(_params.DstAddress.Pack());
_context.Methods.TextureManager.Flush(dstBaseAddress, (ulong)_size);
// Trigger read tracking, to flush any managed resources in the destination region.
_context.PhysicalMemory.GetSpan(dstBaseAddress, _size, true);
_finished = false;
}

143
Ryujinx.Graphics.Gpu/Engine/MethodCopyBuffer.cs
View file

@ -1,3 +1,4 @@
using Ryujinx.Common;
using Ryujinx.Graphics.Gpu.State;
using Ryujinx.Graphics.Texture;
using System;
@ -7,6 +8,37 @@ namespace Ryujinx.Graphics.Gpu.Engine
{
partial class Methods
{
private const int StrideAlignment = 32;
private const int GobAlignment = 64;
/// <summary>
/// Determine if a buffer-to-texture region covers the entirety of a texture.
/// </summary>
/// <param name="cbp">Copy command parameters</param>
/// <param name="tex">Texture to compare</param>
/// <param name="linear">True if the texture is linear, false if block linear</param>
/// <param name="bpp">Texture bytes per pixel</param>
/// <param name="stride">Texture stride</param>
/// <returns></returns>
private bool IsTextureCopyComplete(CopyBufferParams cbp, CopyBufferTexture tex, bool linear, int bpp, int stride)
{
if (linear)
{
int alignWidth = StrideAlignment / bpp;
return tex.RegionX == 0 &&
tex.RegionY == 0 &&
stride / bpp == BitUtils.AlignUp(cbp.XCount, alignWidth);
}
else
{
int alignWidth = GobAlignment / bpp;
return tex.RegionX == 0 &&
tex.RegionY == 0 &&
tex.Width == BitUtils.AlignUp(cbp.XCount, alignWidth) &&
tex.Height == cbp.YCount;
}
}
/// <summary>
/// Performs a buffer to buffer, or buffer to texture copy.
/// </summary>
@ -62,54 +94,97 @@ namespace Ryujinx.Graphics.Gpu.Engine
(int srcBaseOffset, int srcSize) = srcCalculator.GetRectangleRange(src.RegionX, src.RegionY, cbp.XCount, cbp.YCount);
(int dstBaseOffset, int dstSize) = dstCalculator.GetRectangleRange(dst.RegionX, dst.RegionY, cbp.XCount, cbp.YCount);
ReadOnlySpan<byte> srcSpan = _context.PhysicalMemory.GetSpan(srcBaseAddress + (ulong)srcBaseOffset, srcSize);
ReadOnlySpan<byte> srcSpan = _context.PhysicalMemory.GetSpan(srcBaseAddress + (ulong)srcBaseOffset, srcSize, true);
Span<byte> dstSpan = _context.PhysicalMemory.GetSpan(dstBaseAddress + (ulong)dstBaseOffset, dstSize).ToArray();
bool completeSource = src.RegionX == 0 && src.RegionY == 0 && src.Width == cbp.XCount && src.Height == cbp.YCount;
bool completeDest = dst.RegionX == 0 && dst.RegionY == 0 && dst.Width == cbp.XCount && dst.Height == cbp.YCount;
bool completeSource = IsTextureCopyComplete(cbp, src, srcLinear, srcBpp, cbp.SrcStride);
bool completeDest = IsTextureCopyComplete(cbp, dst, dstLinear, dstBpp, cbp.DstStride);
if (completeSource && completeDest && srcCalculator.LayoutMatches(dstCalculator))
if (completeSource && completeDest)
{
srcSpan.CopyTo(dstSpan); // No layout conversion has to be performed, just copy the data entirely.
}
else
{
unsafe bool Convert<T>(Span<byte> dstSpan, ReadOnlySpan<byte> srcSpan) where T : unmanaged
Image.Texture target = TextureManager.FindTexture(dst, cbp, swizzle, dstLinear);
if (target != null)
{
fixed (byte* dstPtr = dstSpan, srcPtr = srcSpan)
ReadOnlySpan<byte> data;
if (srcLinear)
{
byte* dstBase = dstPtr - dstBaseOffset; // Layout offset is relative to the base, so we need to subtract the span's offset.
byte* srcBase = srcPtr - srcBaseOffset;
data = LayoutConverter.ConvertLinearStridedToLinear(
target.Info.Width,
target.Info.Height,
1,
1,
cbp.SrcStride,
target.Info.FormatInfo.BytesPerPixel,
srcSpan);
}
else
{
data = LayoutConverter.ConvertBlockLinearToLinear(
src.Width,
src.Height,
1,
target.Info.Levels,
1,
1,
1,
srcBpp,
src.MemoryLayout.UnpackGobBlocksInY(),
src.MemoryLayout.UnpackGobBlocksInZ(),
src.MemoryLayout.UnpackGobBlocksInX(),
new SizeInfo((int)target.Size),
srcSpan);
}
for (int y = 0; y < cbp.YCount; y++)
target.SetData(data);
target.SignalModified();
return;
}
else if (srcCalculator.LayoutMatches(dstCalculator))
{
srcSpan.CopyTo(dstSpan); // No layout conversion has to be performed, just copy the data entirely.
_context.PhysicalMemory.Write(dstBaseAddress + (ulong)dstBaseOffset, dstSpan);
return;
}
}
unsafe bool Convert<T>(Span<byte> dstSpan, ReadOnlySpan<byte> srcSpan) where T : unmanaged
{
fixed (byte* dstPtr = dstSpan, srcPtr = srcSpan)
{
byte* dstBase = dstPtr - dstBaseOffset; // Layout offset is relative to the base, so we need to subtract the span's offset.
byte* srcBase = srcPtr - srcBaseOffset;
for (int y = 0; y < cbp.YCount; y++)
{
srcCalculator.SetY(src.RegionY + y);
dstCalculator.SetY(dst.RegionY + y);
for (int x = 0; x < cbp.XCount; x++)
{
srcCalculator.SetY(src.RegionY + y);
dstCalculator.SetY(dst.RegionY + y);
int srcOffset = srcCalculator.GetOffset(src.RegionX + x);
int dstOffset = dstCalculator.GetOffset(dst.RegionX + x);
for (int x = 0; x < cbp.XCount; x++)
{
int srcOffset = srcCalculator.GetOffset(src.RegionX + x);
int dstOffset = dstCalculator.GetOffset(dst.RegionX + x);
*(T*)(dstBase + dstOffset) = *(T*)(srcBase + srcOffset);
}
*(T*)(dstBase + dstOffset) = *(T*)(srcBase + srcOffset);
}
}
return true;
}
bool _ = srcBpp switch
{
1 => Convert<byte>(dstSpan, srcSpan),
2 => Convert<ushort>(dstSpan, srcSpan),
4 => Convert<uint>(dstSpan, srcSpan),
8 => Convert<ulong>(dstSpan, srcSpan),
12 => Convert<Bpp12Pixel>(dstSpan, srcSpan),
16 => Convert<Vector128<byte>>(dstSpan, srcSpan),
_ => throw new NotSupportedException($"Unable to copy ${srcBpp} bpp pixel format.")
};
return true;
}
bool _ = srcBpp switch
{
1 => Convert<byte>(dstSpan, srcSpan),
2 => Convert<ushort>(dstSpan, srcSpan),
4 => Convert<uint>(dstSpan, srcSpan),
8 => Convert<ulong>(dstSpan, srcSpan),
12 => Convert<Bpp12Pixel>(dstSpan, srcSpan),
16 => Convert<Vector128<byte>>(dstSpan, srcSpan),
_ => throw new NotSupportedException($"Unable to copy ${srcBpp} bpp pixel format.")
};
_context.PhysicalMemory.Write(dstBaseAddress + (ulong)dstBaseOffset, dstSpan);
}
else

11
Ryujinx.Graphics.Gpu/Engine/Methods.cs
View file

@ -78,7 +78,6 @@ namespace Ryujinx.Graphics.Gpu.Engine
state.RegisterCallback(MethodOffset.CopyTexture, CopyTexture);
state.RegisterCallback(MethodOffset.TextureBarrier, TextureBarrier);
state.RegisterCallback(MethodOffset.InvalidateTextures, InvalidateTextures);
state.RegisterCallback(MethodOffset.TextureBarrierTiled, TextureBarrierTiled);
state.RegisterCallback(MethodOffset.VbElementU8, VbElementU8);
@ -1156,16 +1155,6 @@ namespace Ryujinx.Graphics.Gpu.Engine
_context.Renderer.Pipeline.TextureBarrier();
}
/// <summary>
/// Invalidates all modified textures on the cache.
/// </summary>
/// <param name="state">Current GPU state (unused)</param>
/// <param name="argument">Method call argument (unused)</param>
private void InvalidateTextures(GpuState state, int argument)
{
TextureManager.Flush();
}
/// <summary>
/// Issues a texture barrier.
/// This waits until previous texture writes from the GPU to finish, before

2
Ryujinx.Graphics.Gpu/Image/AutoDeleteCache.cs
View file

@ -43,7 +43,7 @@ namespace Ryujinx.Graphics.Gpu.Image
oldestTexture.SynchronizeMemory();
if (oldestTexture.IsModified)
if (oldestTexture.IsModified && !oldestTexture.ConsumeModified())
{
// The texture must be flushed if it falls out of the auto delete cache.
// Flushes out of the auto delete cache do not trigger write tracking,

18
Ryujinx.Graphics.Gpu/Image/Pool.cs
View file

@ -1,4 +1,5 @@
using Ryujinx.Common;
using Ryujinx.Cpu.Tracking;
using Ryujinx.Graphics.Gpu.Memory;
using System;
@ -34,7 +35,7 @@ namespace Ryujinx.Graphics.Gpu.Image
/// </summary>
public ulong Size { get; }
private readonly (ulong, ulong)[] _modifiedRanges;
private readonly CpuMultiRegionHandle _memoryTracking;
public Pool(GpuContext context, ulong address, int maximumId)
{
@ -50,11 +51,7 @@ namespace Ryujinx.Graphics.Gpu.Image
Address = address;
Size = size;
ulong endAddress = BitUtils.AlignUp(Address + Size, PhysicalMemory.PageSize);
ulong pagesCount = (endAddress - BitUtils.AlignDown(Address, PhysicalMemory.PageSize)) / PhysicalMemory.PageSize;
_modifiedRanges = new (ulong, ulong)[pagesCount];
_memoryTracking = context.PhysicalMemory.BeginGranularTracking(address, size);
}
/// <summary>
@ -71,12 +68,8 @@ namespace Ryujinx.Graphics.Gpu.Image
/// </summary>
public void SynchronizeMemory()
{
int count = Context.PhysicalMemory.QueryModified(Address, Size, ResourceName.TexturePool, _modifiedRanges);
for (int index = 0; index < count; index++)
_memoryTracking.QueryModified((ulong mAddress, ulong mSize) =>
{
(ulong mAddress, ulong mSize) = _modifiedRanges[index];
if (mAddress < Address)
{
mAddress = Address;
@ -90,7 +83,7 @@ namespace Ryujinx.Graphics.Gpu.Image
}
InvalidateRangeImpl(mAddress, mSize);
}
});
}
private void InvalidateRangeInternal(ulong offset, int size)
@ -152,6 +145,7 @@ namespace Ryujinx.Graphics.Gpu.Image
Items = null;
}
_memoryTracking.Dispose();
}
}
}

224
Ryujinx.Graphics.Gpu/Image/Texture.cs
View file

@ -1,9 +1,10 @@
using Ryujinx.Common;
using Ryujinx.Common.Logging;
using Ryujinx.Cpu.Tracking;
using Ryujinx.Graphics.GAL;
using Ryujinx.Graphics.Gpu.Memory;
using Ryujinx.Graphics.Texture;
using Ryujinx.Graphics.Texture.Astc;
using Ryujinx.Memory.Range;
using System;
using System.Collections.Generic;
using System.Diagnostics;
@ -40,12 +41,10 @@ namespace Ryujinx.Graphics.Gpu.Image
public TextureScaleMode ScaleMode { get; private set; }
/// <summary>
/// Set when a texture has been modified since it was last flushed.
/// Set when a texture has been modified by the Host GPU since it was last flushed.
/// </summary>
public bool IsModified { get; internal set; }
private bool _everModified;
private int _depth;
private int _layers;
private int _firstLayer;
@ -56,6 +55,8 @@ namespace Ryujinx.Graphics.Gpu.Image
private ITexture _arrayViewTexture;
private Target _arrayViewTarget;
private ITexture _flushHostTexture;
private Texture _viewStorage;
private List<Texture> _views;
@ -70,11 +71,6 @@ namespace Ryujinx.Graphics.Gpu.Image
/// </summary>
public LinkedListNode<Texture> CacheNode { get; set; }
/// <summary>
/// Event to fire when texture data is modified by the GPU.
/// </summary>
public event Action<Texture> Modified;
/// <summary>
/// Event to fire when texture data is disposed.
/// </summary>
@ -95,12 +91,10 @@ namespace Ryujinx.Graphics.Gpu.Image
/// </summary>
public ulong Size => (ulong)_sizeInfo.TotalSize;
private (ulong, ulong)[] _modifiedRanges;
private CpuRegionHandle _memoryTracking;
private int _referenceCount;
private int _sequenceNumber;
/// <summary>
/// Constructs a new instance of the cached GPU texture.
/// </summary>
@ -159,8 +153,6 @@ namespace Ryujinx.Graphics.Gpu.Image
_context = context;
_sizeInfo = sizeInfo;
_modifiedRanges = new (ulong, ulong)[(sizeInfo.TotalSize / PhysicalMemory.PageSize) + 1];
SetInfo(info);
_viewStorage = this;
@ -176,6 +168,8 @@ namespace Ryujinx.Graphics.Gpu.Image
/// <param name="withData">True if the texture is to be initialized with data</param>
public void InitializeData(bool isView, bool withData = false)
{
_memoryTracking = _context.PhysicalMemory.BeginTracking(Address, Size);
if (withData)
{
Debug.Assert(!isView);
@ -246,6 +240,8 @@ namespace Ryujinx.Graphics.Gpu.Image
/// <param name="texture">The child texture</param>
private void AddView(Texture texture)
{
DisableMemoryTracking();
_views.Add(texture);
texture._viewStorage = this;
@ -313,6 +309,16 @@ namespace Ryujinx.Graphics.Gpu.Image
}
}
/// <summary>
/// Disables memory tracking on this texture. Currently used for view containers, as we assume their views are covering all memory regions.
/// Textures with disabled memory tracking also cannot flush in most circumstances.
/// </summary>
public void DisableMemoryTracking()
{
_memoryTracking?.Dispose();
_memoryTracking = null;
}
/// <summary>
/// Recreates the texture storage (or view, in the case of child textures) of this texture.
/// This allows recreating the texture with a new size.
@ -452,6 +458,33 @@ namespace Ryujinx.Graphics.Gpu.Image
}
}
/// <summary>
/// Copy the host texture to a scaled one. If a texture is not provided, create it with the given scale.
/// </summary>
/// <param name="scale">Scale factor</param>
/// <param name="storage">Texture to use instead of creating one</param>
/// <returns>A host texture containing a scaled version of this texture</returns>
private ITexture GetScaledHostTexture(float scale, ITexture storage = null)
{
if (storage == null)
{
TextureCreateInfo createInfo = TextureManager.GetCreateInfo(Info, _context.Capabilities);
storage = _context.Renderer.CreateTexture(createInfo, scale);
}
if (Info.Target == Target.Texture2DArray)
{
CopyArrayScaled(storage);
}
else
{
HostTexture.CopyTo(storage, new Extents2D(0, 0, HostTexture.Width, HostTexture.Height), new Extents2D(0, 0, storage.Width, storage.Height), true);
}
return storage;
}
/// <summary>
/// Sets the Scale Factor on this texture, and immediately recreates it at the correct size.
/// When a texture is resized, a scaled copy is performed from the old texture to the new one, to ensure no data is lost.
@ -474,20 +507,10 @@ namespace Ryujinx.Graphics.Gpu.Image
if (ScaleFactor != scale)
{
Logger.Debug?.Print(LogClass.Gpu, $"Rescaling {Info.Width}x{Info.Height} {Info.FormatInfo.Format.ToString()} to ({ScaleFactor} to {scale}). ");
TextureCreateInfo createInfo = TextureManager.GetCreateInfo(Info, _context.Capabilities);
ScaleFactor = scale;
ITexture newStorage = _context.Renderer.CreateTexture(createInfo, ScaleFactor);
if (Info.Target == Target.Texture2DArray)
{
CopyArrayScaled(newStorage);
}
else
{
HostTexture.CopyTo(newStorage, new Extents2D(0, 0, HostTexture.Width, HostTexture.Height), new Extents2D(0, 0, newStorage.Width, newStorage.Height), true);
}
ITexture newStorage = GetScaledHostTexture(ScaleFactor);
Logger.Debug?.Print(LogClass.Gpu, $" Copy performed: {HostTexture.Width}x{HostTexture.Height} to {newStorage.Width}x{newStorage.Height}");
@ -525,10 +548,17 @@ namespace Ryujinx.Graphics.Gpu.Image
/// Checks if the memory for this texture was modified, and returns true if it was.
/// The modified flags are consumed as a result.
/// </summary>
/// <remarks>
/// If there is no memory tracking for this texture, it will always report as modified.
/// </remarks>
/// <returns>True if the texture was modified, false otherwise.</returns>
public bool ConsumeModified()
{
return _context.PhysicalMemory.QueryModified(Address, Size, ResourceName.Texture, _modifiedRanges) > 0;
bool wasDirty = _memoryTracking?.Dirty ?? true;
_memoryTracking?.Reprotect();
return wasDirty;
}
/// <summary>
@ -540,20 +570,14 @@ namespace Ryujinx.Graphics.Gpu.Image
/// </summary>
public void SynchronizeMemory()
{
// Texture buffers are not handled here, instead they are invalidated (if modified)
// when the texture is bound. This is handled by the buffer manager.
if ((_sequenceNumber == _context.SequenceNumber && _hasData) || Info.Target == Target.TextureBuffer)
if (Info.Target == Target.TextureBuffer)
{
return;
}
_sequenceNumber = _context.SequenceNumber;
int modifiedCount = _context.PhysicalMemory.QueryModified(Address, Size, ResourceName.Texture, _modifiedRanges);
if (_hasData)
{
if (modifiedCount == 0)
if (_memoryTracking?.Dirty != true)
{
return;
}
@ -561,47 +585,10 @@ namespace Ryujinx.Graphics.Gpu.Image
BlacklistScale();
}
_memoryTracking?.Reprotect();
ReadOnlySpan<byte> data = _context.PhysicalMemory.GetSpan(Address, (int)Size);
// If the texture was ever modified by the host GPU, we do partial invalidation
// of the texture by getting GPU data and merging in the pages of memory
// that were modified.
// Note that if ASTC is not supported by the GPU we can't read it back since
// it will use a different format. Since applications shouldn't be writing
// ASTC textures from the GPU anyway, ignoring it should be safe.
if (_everModified && !Info.FormatInfo.Format.IsAstc())
{
Span<byte> gpuData = GetTextureDataFromGpu(true);
ulong endAddress = Address + Size;
for (int i = 0; i < modifiedCount; i++)
{
(ulong modifiedAddress, ulong modifiedSize) = _modifiedRanges[i];
ulong endModifiedAddress = modifiedAddress + modifiedSize;
if (modifiedAddress < Address)
{
modifiedAddress = Address;
}
if (endModifiedAddress > endAddress)
{
endModifiedAddress = endAddress;
}
modifiedSize = endModifiedAddress - modifiedAddress;
int offset = (int)(modifiedAddress - Address);
int length = (int)modifiedSize;
data.Slice(offset, length).CopyTo(gpuData.Slice(offset, length));
}
data = gpuData;
}
IsModified = false;
data = ConvertToHostCompatibleFormat(data);
@ -611,6 +598,19 @@ namespace Ryujinx.Graphics.Gpu.Image
_hasData = true;
}
public void SetData(ReadOnlySpan<byte> data)
{
BlacklistScale();
_memoryTracking?.Reprotect();
IsModified = false;
HostTexture.SetData(data);
_hasData = true;
}
/// <summary>
/// Converts texture data to a format and layout that is supported by the host GPU.
/// </summary>
@ -699,6 +699,39 @@ namespace Ryujinx.Graphics.Gpu.Image
}
}
/// <summary>
/// Flushes the texture data, to be called from an external thread.
/// The host backend must ensure that we have shared access to the resource from this thread.
/// This is used when flushing from memory access handlers.
/// </summary>
public void ExternalFlush(ulong address, ulong size)
{
if (!IsModified || _memoryTracking == null)
{
return;
}
_context.Renderer.BackgroundContextAction(() =>
{
IsModified = false;
if (Info.FormatInfo.Format.IsAstc())
{
// ASTC textures are not in their original format, so cannot be flushed.
return;
}
ITexture texture = HostTexture;
if (ScaleFactor != 1f)
{
// If needed, create a texture to flush back to host at 1x scale.
texture = _flushHostTexture = GetScaledHostTexture(1f, _flushHostTexture);
}
_context.PhysicalMemory.WriteUntracked(Address, GetTextureDataFromGpu(false, texture));
});
}
/// <summary>
/// Gets data from the host GPU.
/// </summary>
@ -707,25 +740,32 @@ namespace Ryujinx.Graphics.Gpu.Image
/// This is not cheap, avoid doing that unless strictly needed.
/// </remarks>
/// <returns>Host texture data</returns>
private Span<byte> GetTextureDataFromGpu(bool blacklist)
private Span<byte> GetTextureDataFromGpu(bool blacklist, ITexture texture = null)
{
Span<byte> data;
if (blacklist)
if (texture != null)
{
BlacklistScale();
data = HostTexture.GetData();
}
else if (ScaleFactor != 1f)
{
float scale = ScaleFactor;
SetScale(1f);
data = HostTexture.GetData();
SetScale(scale);
data = texture.GetData();
}
else
{
data = HostTexture.GetData();
if (blacklist)
{
BlacklistScale();
data = HostTexture.GetData();
}
else if (ScaleFactor != 1f)
{
float scale = ScaleFactor;
SetScale(1f);
data = HostTexture.GetData();
SetScale(scale);
}
else
{
data = HostTexture.GetData();
}
}
if (Info.IsLinear)
@ -967,6 +1007,7 @@ namespace Ryujinx.Graphics.Gpu.Image
/// <param name="firstLevel">The first level of the view</param>
public void ReplaceView(Texture parent, TextureInfo info, ITexture hostTexture, int firstLayer, int firstLevel)
{
parent._viewStorage.SynchronizeMemory();
ReplaceStorage(hostTexture);
_firstLayer = parent._firstLayer + firstLayer;
@ -994,14 +1035,13 @@ namespace Ryujinx.Graphics.Gpu.Image
public void SignalModified()
{
IsModified = true;
_everModified = true;
Modified?.Invoke(this);
if (_viewStorage != this)
{
_viewStorage.SignalModified();
}
_memoryTracking?.RegisterAction(ExternalFlush);
}
/// <summary>
@ -1109,6 +1149,9 @@ namespace Ryujinx.Graphics.Gpu.Image
_arrayViewTexture?.Release();
_arrayViewTexture = null;
_flushHostTexture?.Release();
_flushHostTexture = null;
}
/// <summary>
@ -1118,6 +1161,10 @@ namespace Ryujinx.Graphics.Gpu.Image
public void Unmapped()
{
IsModified = false; // We shouldn't flush this texture, as its memory is no longer mapped.
CpuRegionHandle tracking = _memoryTracking;
tracking?.Reprotect();
tracking?.RegisterAction(null);
}
/// <summary>
@ -1128,6 +1175,7 @@ namespace Ryujinx.Graphics.Gpu.Image
DisposeTextures();
Disposed?.Invoke(this);
_memoryTracking?.Dispose();
}
}
}

141
Ryujinx.Graphics.Gpu/Image/TextureManager.cs
View file

@ -4,8 +4,8 @@ using Ryujinx.Graphics.Gpu.Image;
using Ryujinx.Graphics.Gpu.Memory;
using Ryujinx.Graphics.Gpu.State;
using Ryujinx.Graphics.Texture;
using Ryujinx.Memory.Range;
using System;
using System.Collections.Generic;
namespace Ryujinx.Graphics.Gpu.Image
{
@ -51,9 +51,6 @@ namespace Ryujinx.Graphics.Gpu.Image
private readonly AutoDeleteCache _cache;
private readonly HashSet<Texture> _modified;
private readonly HashSet<Texture> _modifiedLinear;
/// <summary>
/// The scaling factor applied to all currently bound render targets.
/// </summary>
@ -82,9 +79,6 @@ namespace Ryujinx.Graphics.Gpu.Image
_overlapInfo = new OverlapInfo[OverlapsBufferInitialCapacity];
_cache = new AutoDeleteCache();
_modified = new HashSet<Texture>(new ReferenceEqualityComparer<Texture>());
_modifiedLinear = new HashSet<Texture>(new ReferenceEqualityComparer<Texture>());
}
/// <summary>
@ -735,8 +729,9 @@ namespace Ryujinx.Graphics.Gpu.Image
for (int index = 0; index < overlapsCount; index++)
{
Texture overlap = _textureOverlaps[index];
TextureViewCompatibility overlapCompatibility = overlap.IsViewCompatible(info, size, out int firstLayer, out int firstLevel);
if (overlap.IsViewCompatible(info, size, out int firstLayer, out int firstLevel) == TextureViewCompatibility.Full)
if (overlapCompatibility == TextureViewCompatibility.Full)
{
if (!isSamplerTexture)
{
@ -745,7 +740,7 @@ namespace Ryujinx.Graphics.Gpu.Image
texture = overlap.CreateView(info, sizeInfo, firstLayer, firstLevel);
if (IsTextureModified(overlap))
if (overlap.IsModified)
{
texture.SignalModified();
}
@ -760,6 +755,11 @@ namespace Ryujinx.Graphics.Gpu.Image
break;
}
else if (overlapCompatibility == TextureViewCompatibility.CopyOnly)
{
// TODO: Copy rules for targets created after the container texture. See below.
overlap.DisableMemoryTracking();
}
}
// No match, create a new texture.
@ -849,7 +849,7 @@ namespace Ryujinx.Graphics.Gpu.Image
// Inherit modification from overlapping texture, do that before replacing
// the view since the replacement operation removes it from the list.
if (IsTextureModified(overlap))
if (overlap.IsModified)
{
texture.SignalModified();
}
@ -859,8 +859,13 @@ namespace Ryujinx.Graphics.Gpu.Image
// If the texture is a 3D texture, we need to additionally copy any slice
// of the 3D texture to the newly created 3D texture.
if (info.Target == Target.Texture3D)
if (info.Target == Target.Texture3D && viewCompatible > 0)
{
// TODO: This copy can currently only happen when the 3D texture is created.
// If a game clears and redraws the slices, we won't be able to copy the new data to the 3D texture.
// Disable tracking to try keep at least the original data in there for as long as possible.
texture.DisableMemoryTracking();
for (int index = 0; index < viewCompatible; index++)
{
Texture overlap = _textureOverlaps[index];
@ -872,7 +877,7 @@ namespace Ryujinx.Graphics.Gpu.Image
overlap.HostTexture.CopyTo(texture.HostTexture, oInfo.FirstLayer, oInfo.FirstLevel);
if (IsTextureModified(overlap))
if (overlap.IsModified)
{
texture.SignalModified();
}
@ -886,8 +891,6 @@ namespace Ryujinx.Graphics.Gpu.Image
if (!isSamplerTexture)
{
_cache.Add(texture);
texture.Modified += CacheTextureModified;
texture.Disposed += CacheTextureDisposed;
}
lock (_textures)
@ -901,42 +904,65 @@ namespace Ryujinx.Graphics.Gpu.Image
}
/// <summary>
/// Checks if a texture was modified by the host GPU.
/// Tries to find an existing texture matching the given buffer copy destination. If none is found, returns null.
/// </summary>
/// <param name="texture">Texture to be checked</param>
/// <returns>True if the texture was modified by the host GPU, false otherwise</returns>
public bool IsTextureModified(Texture texture)
/// <param name="tex">The texture information</param>
/// <param name="cbp">The copy buffer parameters</param>
/// <param name="swizzle">The copy buffer swizzle</param>
/// <param name="linear">True if the texture has a linear layout, false otherwise</param>
/// <returns>A matching texture, or null if there is no match</returns>
public Texture FindTexture(CopyBufferTexture tex, CopyBufferParams cbp, CopyBufferSwizzle swizzle, bool linear)
{
return _modified.Contains(texture);
}
ulong address = _context.MemoryManager.Translate(cbp.DstAddress.Pack());
/// <summary>
/// Signaled when a cache texture is modified, and adds it to a set to be enumerated when flushing textures.
/// </summary>
/// <param name="texture">The texture that was modified.</param>
private void CacheTextureModified(Texture texture)
{
texture.IsModified = true;
_modified.Add(texture);
if (texture.Info.IsLinear)
if (address == MemoryManager.BadAddress)
{
_modifiedLinear.Add(texture);
return null;
}
}
/// <summary>
/// Signaled when a cache texture is disposed, so it can be removed from the set of modified textures if present.
/// </summary>
/// <param name="texture">The texture that was diosposed.</param>
private void CacheTextureDisposed(Texture texture)
{
_modified.Remove(texture);
int bpp = swizzle.UnpackDstComponentsCount() * swizzle.UnpackComponentSize();
if (texture.Info.IsLinear)
int addressMatches = _textures.FindOverlaps(address, ref _textureOverlaps);
for (int i = 0; i < addressMatches; i++)
{
_modifiedLinear.Remove(texture);
Texture texture = _textureOverlaps[i];
FormatInfo format = texture.Info.FormatInfo;
if (texture.Info.DepthOrLayers > 1)
{
continue;
}
bool match;
if (linear)
{
// Size is not available for linear textures. Use the stride and end of the copy region instead.
match = texture.Info.IsLinear && texture.Info.Stride == cbp.DstStride && tex.RegionY + cbp.YCount <= texture.Info.Height;
}
else
{
// Bpp may be a mismatch between the target texture and the param.
// Due to the way linear strided and block layouts work, widths can be multiplied by Bpp for comparison.
// Note: tex.Width is the aligned texture size. Prefer param.XCount, as the destination should be a texture with that exact size.
bool sizeMatch = cbp.XCount * bpp == texture.Info.Width * format.BytesPerPixel && tex.Height == texture.Info.Height;
bool formatMatch = !texture.Info.IsLinear &&
texture.Info.GobBlocksInY == tex.MemoryLayout.UnpackGobBlocksInY() &&
texture.Info.GobBlocksInZ == tex.MemoryLayout.UnpackGobBlocksInZ();
match = sizeMatch && formatMatch;
}
if (match)
{
return texture;
}
}
return null;
}
/// <summary>
@ -1084,38 +1110,6 @@ namespace Ryujinx.Graphics.Gpu.Image
info.SwizzleA);
}
/// <summary>
/// Flushes all the textures in the cache that have been modified since the last call.
/// </summary>
public void Flush()
{
foreach (Texture texture in _modifiedLinear)
{
if (texture.IsModified)
{
texture.Flush();
}
}
_modifiedLinear.Clear();
}
/// <summary>
/// Flushes the textures in the cache inside a given range that have been modified since the last call.
/// </summary>
/// <param name="address">The range start address</param>
/// <param name="size">The range size</param>
public void Flush(ulong address, ulong size)
{
foreach (Texture texture in _modified)
{
if (texture.OverlapsWith(address, size) && texture.IsModified)
{
texture.Flush();
}
}
}
/// <summary>
/// Removes a texture from the cache.
/// </summary>
@ -1142,7 +1136,6 @@ namespace Ryujinx.Graphics.Gpu.Image
{
foreach (Texture texture in _textures)
{
_modified.Remove(texture);
texture.Dispose();
}
}

80
Ryujinx.Graphics.Gpu/Memory/Buffer.cs
View file

@ -1,4 +1,6 @@
using Ryujinx.Cpu.Tracking;
using Ryujinx.Graphics.GAL;
using Ryujinx.Memory.Range;
using System;
namespace Ryujinx.Graphics.Gpu.Memory
@ -8,6 +10,8 @@ namespace Ryujinx.Graphics.Gpu.Memory
/// </summary>
class Buffer : IRange, IDisposable
{
private static ulong GranularBufferThreshold = 4096;
private readonly GpuContext _context;
/// <summary>
@ -30,9 +34,11 @@ namespace Ryujinx.Graphics.Gpu.Memory
/// </summary>
public ulong EndAddress => Address + Size;
private readonly (ulong, ulong)[] _modifiedRanges;
private CpuSmartMultiRegionHandle _memoryTrackingGranular;
private CpuRegionHandle _memoryTracking;
private int _sequenceNumber;
private readonly int[] _sequenceNumbers;
private bool _useGranular;
/// <summary>
/// Creates a new instance of the buffer.
@ -48,9 +54,16 @@ namespace Ryujinx.Graphics.Gpu.Memory
Handle = context.Renderer.CreateBuffer((int)size);
_modifiedRanges = new (ulong, ulong)[size / PhysicalMemory.PageSize];
_useGranular = size > GranularBufferThreshold;
_sequenceNumbers = new int[size / MemoryManager.PageSize];
if (_useGranular)
{
_memoryTrackingGranular = context.PhysicalMemory.BeginSmartGranularTracking(address, size);
}
else
{
_memoryTracking = context.PhysicalMemory.BeginTracking(address, size);
}
}
/// <summary>
@ -91,42 +104,36 @@ namespace Ryujinx.Graphics.Gpu.Memory
/// <param name="size">Size in bytes of the range to synchronize</param>
public void SynchronizeMemory(ulong address, ulong size)
{
int currentSequenceNumber = _context.SequenceNumber;
bool needsSync = false;
ulong buffOffset = address - Address;
ulong buffEndOffset = (buffOffset + size + MemoryManager.PageMask) & ~MemoryManager.PageMask;
int startIndex = (int)(buffOffset / MemoryManager.PageSize);
int endIndex = (int)(buffEndOffset / MemoryManager.PageSize);
for (int index = startIndex; index < endIndex; index++)
if (_useGranular)
{
if (_sequenceNumbers[index] != currentSequenceNumber)
_memoryTrackingGranular.QueryModified(address, size, (ulong mAddress, ulong mSize) =>
{
_sequenceNumbers[index] = currentSequenceNumber;
if (mAddress < Address)
{
mAddress = Address;
}
needsSync = true;
ulong maxSize = Address + Size - mAddress;
if (mSize > maxSize)
{
mSize = maxSize;
}
int offset = (int)(mAddress - Address);
_context.Renderer.SetBufferData(Handle, offset, _context.PhysicalMemory.GetSpan(mAddress, (int)mSize));
}, _context.SequenceNumber);
}
else
{
if (_memoryTracking.Dirty && _context.SequenceNumber != _sequenceNumber)
{
_memoryTracking.Reprotect();
_context.Renderer.SetBufferData(Handle, 0, _context.PhysicalMemory.GetSpan(Address, (int)Size));
_sequenceNumber = _context.SequenceNumber;
}
}
if (!needsSync)
{
return;
}
int count = _context.PhysicalMemory.QueryModified(address, size, ResourceName.Buffer, _modifiedRanges);
for (int index = 0; index < count; index++)
{
(ulong mAddress, ulong mSize) = _modifiedRanges[index];
int offset = (int)(mAddress - Address);
_context.Renderer.SetBufferData(Handle, offset, _context.PhysicalMemory.GetSpan(mAddress, (int)mSize));
}
}
/// <summary>
@ -161,6 +168,9 @@ namespace Ryujinx.Graphics.Gpu.Memory
public void Dispose()
{
_context.Renderer.DeleteBuffer(Handle);
_memoryTrackingGranular?.Dispose();
_memoryTracking?.Dispose();
}
}
}

2
Ryujinx.Graphics.Gpu/Memory/BufferManager.cs
View file

@ -2,6 +2,7 @@ using Ryujinx.Common;
using Ryujinx.Graphics.GAL;
using Ryujinx.Graphics.Gpu.State;
using Ryujinx.Graphics.Shader;
using Ryujinx.Memory.Range;
using System;
namespace Ryujinx.Graphics.Gpu.Memory
@ -407,6 +408,7 @@ namespace Ryujinx.Graphics.Gpu.Memory
}
Buffer newBuffer = new Buffer(_context, address, endAddress - address);
newBuffer.SynchronizeMemory(address, endAddress - address);
_buffers.Add(newBuffer);

13
Ryujinx.Graphics.Gpu/Memory/IRange.cs
View file

@ -1,13 +0,0 @@
namespace Ryujinx.Graphics.Gpu.Memory
{
/// <summary>
/// Range of memory.
/// </summary>
interface IRange
{
ulong Address { get; }
ulong Size { get; }
bool OverlapsWith(ulong address, ulong size);
}
}

4
Ryujinx.Graphics.Gpu/Memory/MemoryManager.cs
View file

@ -125,6 +125,8 @@ namespace Ryujinx.Graphics.Gpu.Memory
{
lock (_pageTable)
{
MemoryUnmapped?.Invoke(this, new UnmapEventArgs(va, size));
for (ulong offset = 0; offset < size; offset += PageSize)
{
SetPte(va + offset, pa + offset);
@ -201,6 +203,8 @@ namespace Ryujinx.Graphics.Gpu.Memory
{
lock (_pageTable)
{
MemoryUnmapped?.Invoke(this, new UnmapEventArgs(va, size));
for (ulong offset = 0; offset < size; offset += PageSize)
{
if (IsPageInUse(va + offset))

41
Ryujinx.Graphics.Gpu/Memory/PhysicalMemory.cs
View file

@ -1,4 +1,5 @@
using Ryujinx.Cpu;
using Ryujinx.Cpu.Tracking;
using System;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
@ -29,10 +30,11 @@ namespace Ryujinx.Graphics.Gpu.Memory
/// </summary>
/// <param name="address">Start address of the range</param>
/// <param name="size">Size in bytes to be range</param>
/// <param name="tracked">True if read tracking is triggered on the span</param>
/// <returns>A read only span of the data at the specified memory location</returns>
public ReadOnlySpan<byte> GetSpan(ulong address, int size)
public ReadOnlySpan<byte> GetSpan(ulong address, int size, bool tracked = false)
{
return _cpuMemory.GetSpan(address, size);
return _cpuMemory.GetSpan(address, size, tracked);
}
/// <summary>
@ -78,17 +80,38 @@ namespace Ryujinx.Graphics.Gpu.Memory
}
/// <summary>
/// Checks if a specified virtual memory region has been modified by the CPU since the last call.
/// Obtains a memory tracking handle for the given virtual region. This should be disposed when finished with.
/// </summary>
/// <param name="address">CPU virtual address of the region</param>
/// <param name="size">Size of the region</param>
/// <param name="name">Resource name</param>
/// <param name="modifiedRanges">Optional array where the modified ranges should be written</param>
/// <returns>The number of modified ranges</returns>
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public int QueryModified(ulong address, ulong size, ResourceName name, (ulong, ulong)[] modifiedRanges = null)
/// <returns>The memory tracking handle</returns>
public CpuRegionHandle BeginTracking(ulong address, ulong size)
{
return _cpuMemory.QueryModified(address, size, (int)name, modifiedRanges);
return _cpuMemory.BeginTracking(address, size);
}
/// <summary>
/// Obtains a memory tracking handle for the given virtual region, with a specified granularity. This should be disposed when finished with.
/// </summary>
/// <param name="address">CPU virtual address of the region</param>
/// <param name="size">Size of the region</param>
/// <param name="granularity">Desired granularity of write tracking</param>
/// <returns>The memory tracking handle</returns>
public CpuMultiRegionHandle BeginGranularTracking(ulong address, ulong size, ulong granularity = 4096)
{
return _cpuMemory.BeginGranularTracking(address, size, granularity);
}
/// <summary>
/// Obtains a smart memory tracking handle for the given virtual region, with a specified granularity. This should be disposed when finished with.
/// </summary>
/// <param name="address">CPU virtual address of the region</param>
/// <param name="size">Size of the region</param>
/// <param name="granularity">Desired granularity of write tracking</param>
/// <returns>The memory tracking handle</returns>
public CpuSmartMultiRegionHandle BeginSmartGranularTracking(ulong address, ulong size, ulong granularity = 4096)
{
return _cpuMemory.BeginSmartGranularTracking(address, size, granularity);
}
}
}

102
Ryujinx.Graphics.OpenGL/BackgroundContextWorker.cs Normal file
View file

@ -0,0 +1,102 @@
using OpenTK;
using OpenTK.Graphics;
using Ryujinx.Common;
using System;
using System.Collections.Generic;
using System.Threading;
namespace Ryujinx.Graphics.OpenGL
{
class BackgroundContextWorker : IDisposable
{
[ThreadStatic]
public static bool InBackground;
private GameWindow _window;
private GraphicsContext _context;
private Thread _thread;
private bool _running;
private AutoResetEvent _signal;
private Queue<Action> _work;
private ObjectPool<ManualResetEventSlim> _invokePool;
public BackgroundContextWorker(IGraphicsContext baseContext)
{
_window = new GameWindow(
100, 100, GraphicsMode.Default,
"Background Window", OpenTK.GameWindowFlags.FixedWindow, OpenTK.DisplayDevice.Default,
3, 3, GraphicsContextFlags.ForwardCompatible, baseContext, false);
_window.Visible = false;
_context = (GraphicsContext)_window.Context;
_context.MakeCurrent(null);
_running = true;
_signal = new AutoResetEvent(false);
_work = new Queue<Action>();
_invokePool = new ObjectPool<ManualResetEventSlim>(() => new ManualResetEventSlim(), 10);
_thread = new Thread(Run);
_thread.Start();
}
private void Run()
{
InBackground = true;
_context.MakeCurrent(_window.WindowInfo);
while (_running)
{
Action action;
lock (_work)
{
_work.TryDequeue(out action);
}
if (action != null)
{
action();
}
else
{
_signal.WaitOne();
}
}
_window.Dispose();
}
public void Invoke(Action action)
{
ManualResetEventSlim actionComplete = _invokePool.Allocate();
lock (_work)
{
_work.Enqueue(() =>
{
action();
actionComplete.Set();
});
}
_signal.Set();
actionComplete.Wait();
actionComplete.Reset();
_invokePool.Release(actionComplete);
}
public void Dispose()
{
_running = false;
_signal.Set();
_thread.Join();
_signal.Dispose();
}
}
}

6
Ryujinx.Graphics.OpenGL/Pipeline.cs
View file

@ -5,6 +5,7 @@ using Ryujinx.Graphics.OpenGL.Image;
using Ryujinx.Graphics.OpenGL.Queries;
using Ryujinx.Graphics.Shader;
using System;
using System.Threading;
namespace Ryujinx.Graphics.OpenGL
{
@ -1163,6 +1164,11 @@ namespace Ryujinx.Graphics.OpenGL
internal (int drawHandle, int readHandle) GetBoundFramebuffers()
{
if (BackgroundContextWorker.InBackground)
{
return (0, 0);
}
return (_boundDrawFramebuffer, _boundReadFramebuffer);
}

30
Ryujinx.Graphics.OpenGL/Renderer.cs
View file

@ -1,4 +1,5 @@
using OpenTK.Graphics.OpenGL;
using OpenTK.Graphics;
using OpenTK.Graphics.OpenGL;
using Ryujinx.Common.Configuration;
using Ryujinx.Common.Logging;
using Ryujinx.Graphics.GAL;
@ -21,7 +22,9 @@ namespace Ryujinx.Graphics.OpenGL
public IWindow Window => _window;
internal TextureCopy TextureCopy { get; }
private TextureCopy _textureCopy;
private TextureCopy _backgroundTextureCopy;
internal TextureCopy TextureCopy => BackgroundContextWorker.InBackground ? _backgroundTextureCopy : _textureCopy;
internal ResourcePool ResourcePool { get; }
@ -34,7 +37,8 @@ namespace Ryujinx.Graphics.OpenGL
_pipeline = new Pipeline();
_counters = new Counters();
_window = new Window(this);
TextureCopy = new TextureCopy(this);
_textureCopy = new TextureCopy(this);
_backgroundTextureCopy = new TextureCopy(this);
ResourcePool = new ResourcePool();
}
@ -135,9 +139,27 @@ namespace Ryujinx.Graphics.OpenGL
_counters.QueueReset(type);
}
public void BackgroundContextAction(Action action)
{
if (GraphicsContext.CurrentContext != null)
{
action(); // We have a context already - use that (assuming it is the main one).
}
else
{
_window.BackgroundContext.Invoke(action);
}
}
public void InitializeBackgroundContext(IGraphicsContext baseContext)
{
_window.InitializeBackgroundContext(baseContext);
}
public void Dispose()
{
TextureCopy.Dispose();
_textureCopy.Dispose();
_backgroundTextureCopy.Dispose();
ResourcePool.Dispose();
_pipeline.Dispose();
_window.Dispose();

2
Ryujinx.Graphics.OpenGL/Ryujinx.Graphics.OpenGL.csproj
View file

@ -8,7 +8,7 @@
</PropertyGroup>
<ItemGroup>
<PackageReference Include="OpenTK.NetStandard" Version="1.0.5.12" />
<PackageReference Include="OpenTK.NetStandard" Version="1.0.5.22" />
</ItemGroup>
<ItemGroup>

12
Ryujinx.Graphics.OpenGL/Window.cs
View file

@ -1,4 +1,7 @@
using OpenTK;
using OpenTK.Graphics;
using OpenTK.Graphics.OpenGL;
using OpenTK.Platform;
using Ryujinx.Graphics.GAL;
using Ryujinx.Graphics.OpenGL.Image;
using System;
@ -17,6 +20,8 @@ namespace Ryujinx.Graphics.OpenGL
private int _copyFramebufferHandle;
internal BackgroundContextWorker BackgroundContext { get; private set; }
public Window(Renderer renderer)
{
_renderer = renderer;
@ -161,8 +166,15 @@ namespace Ryujinx.Graphics.OpenGL
return handle;
}
public void InitializeBackgroundContext(IGraphicsContext baseContext)
{
BackgroundContext = new BackgroundContextWorker(baseContext);
}
public void Dispose()
{
BackgroundContext.Dispose();
if (_copyFramebufferHandle != 0)
{
GL.DeleteFramebuffer(_copyFramebufferHandle);

23
Ryujinx.Memory.Tests/MockVirtualMemoryManager.cs Normal file
View file

@ -0,0 +1,23 @@
using Ryujinx.Memory.Tracking;
namespace Ryujinx.Memory.Tests
{
class MockVirtualMemoryManager : IVirtualMemoryManager
{
public bool NoMappings;
public MockVirtualMemoryManager(ulong size, int pageSize)
{
}
public (ulong address, ulong size)[] GetPhysicalRegions(ulong va, ulong size)
{
return NoMappings ? new (ulong address, ulong size)[0] : new (ulong address, ulong size)[] { (va, size) };
}
public void TrackingReprotect(ulong va, ulong size, MemoryPermission protection)
{
}
}
}

283
Ryujinx.Memory.Tests/MultiRegionTrackingTests.cs Normal file
View file

@ -0,0 +1,283 @@
using NUnit.Framework;
using Ryujinx.Memory.Tracking;
using System;
using System.Collections.Generic;
using System.Linq;
namespace Ryujinx.Memory.Tests
{
public class MultiRegionTrackingTests
{
private const int RndCnt = 3;
private const ulong MemorySize = 0x8000;
private const int PageSize = 4096;
private MemoryBlock _memoryBlock;
private MemoryTracking _tracking;
private MockVirtualMemoryManager _memoryManager;
[SetUp]
public void Setup()
{
_memoryBlock = new MemoryBlock(MemorySize);
_memoryManager = new MockVirtualMemoryManager(MemorySize, PageSize);
_tracking = new MemoryTracking(_memoryManager, _memoryBlock, PageSize);
}
[TearDown]
public void Teardown()
{
_memoryBlock.Dispose();
}
private IMultiRegionHandle GetGranular(bool smart, ulong address, ulong size, ulong granularity)
{
return smart ?
_tracking.BeginSmartGranularTracking(address, size, granularity) :
(IMultiRegionHandle)_tracking.BeginGranularTracking(address, size, granularity);
}
private void RandomOrder(Random random, List<int> indices, Action<int> action)
{
List<int> choices = indices.ToList();
while (choices.Count > 0)
{
int choice = random.Next(choices.Count);
action(choices[choice]);
choices.RemoveAt(choice);
}
}
private int ExpectQueryInOrder(IMultiRegionHandle handle, ulong startAddress, ulong size, Func<ulong, bool> addressPredicate)
{
int regionCount = 0;
ulong lastAddress = startAddress;
handle.QueryModified(startAddress, size, (address, range) =>
{
Assert.IsTrue(addressPredicate(address)); // Written pages must be even.
Assert.GreaterOrEqual(address, lastAddress); // Must be signalled in ascending order, regardless of write order.
lastAddress = address;
regionCount++;
});
return regionCount;
}
private int ExpectQueryInOrder(IMultiRegionHandle handle, ulong startAddress, ulong size, Func<ulong, bool> addressPredicate, int sequenceNumber)
{
int regionCount = 0;
ulong lastAddress = startAddress;
handle.QueryModified(startAddress, size, (address, range) =>
{
Assert.IsTrue(addressPredicate(address)); // Written pages must be even.
Assert.GreaterOrEqual(address, lastAddress); // Must be signalled in ascending order, regardless of write order.
lastAddress = address;
regionCount++;
}, sequenceNumber);
return regionCount;
}
private void PreparePages(IMultiRegionHandle handle, int pageCount, ulong address = 0)
{
Random random = new Random();
// Make sure the list has minimum granularity (smart region changes granularity based on requested ranges)
RandomOrder(random, Enumerable.Range(0, pageCount).ToList(), (i) =>
{
ulong resultAddress = ulong.MaxValue;
handle.QueryModified((ulong)i * PageSize + address, PageSize, (address, range) =>
{
resultAddress = address;
});
Assert.AreEqual(resultAddress, (ulong)i * PageSize + address);
});
}
[Test]
public void DirtyRegionOrdering([Values] bool smart)
{
const int pageCount = 32;
IMultiRegionHandle handle = GetGranular(smart, 0, PageSize * pageCount, PageSize);
Random random = new Random();
PreparePages(handle, pageCount);
IEnumerable<int> halfRange = Enumerable.Range(0, pageCount / 2);
List<int> odd = halfRange.Select(x => x * 2 + 1).ToList();
List<int> even = halfRange.Select(x => x * 2).ToList();
// Write to all the odd pages.
RandomOrder(random, odd, (i) =>
{
_tracking.VirtualMemoryEvent((ulong)i * PageSize, PageSize, true);
});
int oddRegionCount = ExpectQueryInOrder(handle, 0, PageSize * pageCount, (address) => (address / PageSize) % 2 == 1);
Assert.AreEqual(oddRegionCount, pageCount / 2); // Must have written to all odd pages.
// Write to all the even pages.
RandomOrder(random, even, (i) =>
{
_tracking.VirtualMemoryEvent((ulong)i * PageSize, PageSize, true);
});
int evenRegionCount = ExpectQueryInOrder(handle, 0, PageSize * pageCount, (address) => (address / PageSize) % 2 == 0);
Assert.AreEqual(evenRegionCount, pageCount / 2);
}
[Test]
public void SequenceNumber([Values] bool smart)
{
// The sequence number can be used to ignore dirty flags, and defer their consumption until later.
// If a user consumes a dirty flag with sequence number 1, then there is a write to the protected region,
// the dirty flag will not be acknowledged until the sequence number is 2.
// This is useful for situations where we know that the data was complete when the sequence number was set.
// ...essentially, when that data can only be updated on a future sequence number.
const int pageCount = 32;
IMultiRegionHandle handle = GetGranular(smart, 0, PageSize * pageCount, PageSize);
PreparePages(handle, pageCount);
Random random = new Random();
IEnumerable<int> halfRange = Enumerable.Range(0, pageCount / 2);
List<int> odd = halfRange.Select(x => x * 2 + 1).ToList();
List<int> even = halfRange.Select(x => x * 2).ToList();
// Write to all the odd pages.
RandomOrder(random, odd, (i) =>
{
_tracking.VirtualMemoryEvent((ulong)i * PageSize, PageSize, true);
});
int oddRegionCount = 0;
// Track with sequence number 1. Future dirty flags should only be consumed with sequence number != 1.
// Only track the odd pages, so the even ones don't have their sequence number set.
foreach (int index in odd)
{
handle.QueryModified((ulong)index * PageSize, PageSize, (address, range) =>
{
oddRegionCount++;
}, 1);
}
Assert.AreEqual(oddRegionCount, pageCount / 2); // Must have written to all odd pages.
// Write to all pages.
_tracking.VirtualMemoryEvent(0, PageSize * pageCount, true);
// Only the even regions should be reported for sequence number 1.
int evenRegionCount = ExpectQueryInOrder(handle, 0, PageSize * pageCount, (address) => (address / PageSize) % 2 == 0, 1);
Assert.AreEqual(evenRegionCount, pageCount / 2); // Must have written to all even pages.
oddRegionCount = 0;
handle.QueryModified(0, PageSize * pageCount, (address, range) => { oddRegionCount++; }, 1);
Assert.AreEqual(oddRegionCount, 0); // Sequence number has not changed, so found no dirty subregions.
// With sequence number 2, all all pages should be reported as modified.
oddRegionCount = ExpectQueryInOrder(handle, 0, PageSize * pageCount, (address) => (address / PageSize) % 2 == 1, 2);
Assert.AreEqual(oddRegionCount, pageCount / 2); // Must have written to all odd pages.
}
[Test]
public void SmartRegionTracking()
{
// Smart multi region handles dynamically change their tracking granularity based on QueryMemory calls.
// This can save on reprotects on larger resources.
const int pageCount = 32;
IMultiRegionHandle handle = GetGranular(true, 0, PageSize * pageCount, PageSize);
// Query some large regions to prep the subdivision of the tracking region.
int[] regionSizes = new int[] { 6, 4, 3, 2, 6, 1 };
ulong address = 0;
for (int i = 0; i < regionSizes.Length; i++)
{
int region = regionSizes[i];
handle.QueryModified(address, (ulong)(PageSize * region), (address, size) => { });
// There should be a gap between regions,
// So that they don't combine and we can see the full effects.
address += (ulong)(PageSize * (region + 1));
}
// Clear modified.
handle.QueryModified((address, size) => { });
// Trigger each region with a 1 byte write.
address = 0;
for (int i = 0; i < regionSizes.Length; i++)
{
int region = regionSizes[i];
_tracking.VirtualMemoryEvent(address, 1, true);
address += (ulong)(PageSize * (region + 1));
}
int regionInd = 0;
ulong expectedAddress = 0;
// Expect each region to trigger in its entirety, in address ascending order.
handle.QueryModified((address, size) => {
int region = regionSizes[regionInd++];
Assert.AreEqual(address, expectedAddress);
Assert.AreEqual(size, (ulong)(PageSize * region));
expectedAddress += (ulong)(PageSize * (region + 1));
});
}
[Test]
public void DisposeMultiHandles([Values] bool smart)
{
// Create and initialize two overlapping Multi Region Handles, with PageSize granularity.
const int pageCount = 32;
const int overlapStart = 16;
Assert.AreEqual((0, 0), _tracking.GetRegionCounts());
IMultiRegionHandle handleLow = GetGranular(smart, 0, PageSize * pageCount, PageSize);
PreparePages(handleLow, pageCount);
Assert.AreEqual((pageCount, pageCount), _tracking.GetRegionCounts());
IMultiRegionHandle handleHigh = GetGranular(smart, PageSize * overlapStart, PageSize * pageCount, PageSize);
PreparePages(handleHigh, pageCount, PageSize * overlapStart);
// Combined pages (and assuming overlapStart <= pageCount) should be pageCount after overlapStart.
int totalPages = overlapStart + pageCount;
Assert.AreEqual((totalPages, totalPages), _tracking.GetRegionCounts());
handleLow.Dispose(); // After disposing one, the pages for the other remain.
Assert.AreEqual((pageCount, pageCount), _tracking.GetRegionCounts());
handleHigh.Dispose(); // After disposing the other, there are no pages left.
Assert.AreEqual((0, 0), _tracking.GetRegionCounts());
}
}
}

425
Ryujinx.Memory.Tests/TrackingTests.cs Normal file
View file

@ -0,0 +1,425 @@
using NUnit.Framework;
using Ryujinx.Memory.Tracking;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading;
namespace Ryujinx.Memory.Tests
{
public class TrackingTests
{
private const int RndCnt = 3;
private const ulong MemorySize = 0x8000;
private const int PageSize = 4096;
private MemoryBlock _memoryBlock;
private MemoryTracking _tracking;
private MockVirtualMemoryManager _memoryManager;
[SetUp]
public void Setup()
{
_memoryBlock = new MemoryBlock(MemorySize);
_memoryManager = new MockVirtualMemoryManager(MemorySize, PageSize);
_tracking = new MemoryTracking(_memoryManager, _memoryBlock, PageSize);
}
[TearDown]
public void Teardown()
{
_memoryBlock.Dispose();
}
private bool TestSingleWrite(RegionHandle handle, ulong address, ulong size, bool physical = false)
{
handle.Reprotect();
if (physical)
{
_tracking.PhysicalMemoryEvent(address, true);
}
else
{
_tracking.VirtualMemoryEvent(address, size, true);
}
return handle.Dirty;
}
[Test]
public void SingleRegion()
{
RegionHandle handle = _tracking.BeginTracking(0, PageSize);
(ulong address, ulong size)? readTrackingTriggered = null;
handle.RegisterAction((address, size) =>
{
readTrackingTriggered = (address, size);
});
bool dirtyInitial = handle.Dirty;
Assert.True(dirtyInitial); // Handle starts dirty.
handle.Reprotect();
bool dirtyAfterReprotect = handle.Dirty;
Assert.False(dirtyAfterReprotect); // Handle is no longer dirty.
_tracking.VirtualMemoryEvent(PageSize * 2, 4, true);
_tracking.VirtualMemoryEvent(PageSize * 2, 4, false);
bool dirtyAfterUnrelatedReadWrite = handle.Dirty;
Assert.False(dirtyAfterUnrelatedReadWrite); // Not dirtied, as the write was to an unrelated address.
Assert.IsNull(readTrackingTriggered); // Hasn't been triggered yet
_tracking.VirtualMemoryEvent(0, 4, false);
bool dirtyAfterRelatedRead = handle.Dirty;
Assert.False(dirtyAfterRelatedRead); // Only triggers on write.
Assert.AreEqual(readTrackingTriggered, (0UL, 4UL)); // Read action was triggered.
readTrackingTriggered = null;
_tracking.VirtualMemoryEvent(0, 4, true);
bool dirtyAfterRelatedWrite = handle.Dirty;
Assert.True(dirtyAfterRelatedWrite); // Dirty flag should now be set.
_tracking.VirtualMemoryEvent(4, 4, true);
bool dirtyAfterRelatedWrite2 = handle.Dirty;
Assert.True(dirtyAfterRelatedWrite2); // Dirty flag should still be set.
handle.Reprotect();
bool dirtyAfterReprotect2 = handle.Dirty;
Assert.False(dirtyAfterReprotect2); // Handle is no longer dirty.
handle.Dispose();
bool dirtyAfterDispose = TestSingleWrite(handle, 0, 4);
Assert.False(dirtyAfterDispose); // Handle cannot be triggered when disposed
bool dirtyAfterDispose2 = TestSingleWrite(handle, 0, 4, true);
Assert.False(dirtyAfterDispose2);
}
[Test]
public void OverlappingRegions()
{
RegionHandle allHandle = _tracking.BeginTracking(0, PageSize * 16);
allHandle.Reprotect();
(ulong address, ulong size)? readTrackingTriggeredAll = null;
Action registerReadAction = () =>
{
readTrackingTriggeredAll = null;
allHandle.RegisterAction((address, size) =>
{
readTrackingTriggeredAll = (address, size);
});
};
registerReadAction();
// Create 16 page sized handles contained within the allHandle.
RegionHandle[] containedHandles = new RegionHandle[16];
for (int i = 0; i < 16; i++)
{
containedHandles[i] = _tracking.BeginTracking((ulong)i * PageSize, PageSize);
containedHandles[i].Reprotect();
}
for (int i = 0; i < 16; i++)
{
// No handles are dirty.
Assert.False(allHandle.Dirty);
Assert.IsNull(readTrackingTriggeredAll);
for (int j = 0; j < 16; j++)
{
Assert.False(containedHandles[j].Dirty);
}
_tracking.VirtualMemoryEvent((ulong)i * PageSize, 1, true);
// Only the handle covering the entire range and the relevant contained handle are dirty.
Assert.True(allHandle.Dirty);
Assert.AreEqual(readTrackingTriggeredAll, ((ulong)i * PageSize, 1UL)); // Triggered read tracking
for (int j = 0; j < 16; j++)
{
if (j == i)
{
Assert.True(containedHandles[j].Dirty);
}
else
{
Assert.False(containedHandles[j].Dirty);
}
}
// Clear flags and reset read action.
registerReadAction();
allHandle.Reprotect();
containedHandles[i].Reprotect();
}
}
[Test]
public void PageAlignment(
[Values(1ul, 512ul, 2048ul, 4096ul, 65536ul)] [Random(1ul, 65536ul, RndCnt)] ulong address,
[Values(1ul, 4ul, 1024ul, 4096ul, 65536ul)] [Random(1ul, 65536ul, RndCnt)] ulong size)
{
ulong alignedStart = (address / PageSize) * PageSize;
ulong alignedEnd = ((address + size + PageSize - 1) / PageSize) * PageSize;
ulong alignedSize = alignedEnd - alignedStart;
RegionHandle handle = _tracking.BeginTracking(address, size);
// Anywhere inside the pages the region is contained on should trigger.
bool originalRangeTriggers = TestSingleWrite(handle, address, size);
Assert.True(originalRangeTriggers);
bool alignedRangeTriggers = TestSingleWrite(handle, alignedStart, alignedSize);
Assert.True(alignedRangeTriggers);
bool alignedStartTriggers = TestSingleWrite(handle, alignedStart, 1);
Assert.True(alignedStartTriggers);
bool alignedEndTriggers = TestSingleWrite(handle, alignedEnd - 1, 1);
Assert.True(alignedEndTriggers);
// Outside the tracked range should not trigger.
bool alignedBeforeTriggers = TestSingleWrite(handle, alignedStart - 1, 1);
Assert.False(alignedBeforeTriggers);
bool alignedAfterTriggers = TestSingleWrite(handle, alignedEnd, 1);
Assert.False(alignedAfterTriggers);
}
[Test, Timeout(1000)]
public void Multithreading()
{
// Multithreading sanity test
// Multiple threads can easily read/write memory regions from any existing handle.
// Handles can also be owned by different threads, though they should have one owner thread.
// Handles can be created and disposed at any time, by any thread.
// This test should not throw or deadlock due to invalid state.
const int threadCount = 1;
const int handlesPerThread = 16;
long finishedTime = 0;
RegionHandle[] handles = new RegionHandle[threadCount * handlesPerThread];
Random globalRand = new Random();
for (int i = 0; i < handles.Length; i++)
{
handles[i] = _tracking.BeginTracking((ulong)i * PageSize, PageSize);
handles[i].Reprotect();
}
List<Thread> testThreads = new List<Thread>();
// Dirty flag consumer threads
int dirtyFlagReprotects = 0;
for (int i = 0; i < threadCount; i++)
{
int randSeed = i;
testThreads.Add(new Thread(() =>
{
int handleBase = randSeed * handlesPerThread;
while (Stopwatch.GetTimestamp() < finishedTime)
{
Random random = new Random(randSeed);
RegionHandle handle = handles[handleBase + random.Next(handlesPerThread)];
if (handle.Dirty)
{
handle.Reprotect();
Interlocked.Increment(ref dirtyFlagReprotects);
}
}
}));
}
// Write trigger threads
int writeTriggers = 0;
for (int i = 0; i < threadCount; i++)
{
int randSeed = i;
testThreads.Add(new Thread(() =>
{
Random random = new Random(randSeed);
ulong handleBase = (ulong)(randSeed * handlesPerThread * PageSize);
while (Stopwatch.GetTimestamp() < finishedTime)
{
_tracking.VirtualMemoryEvent(handleBase + (ulong)random.Next(PageSize * handlesPerThread), PageSize / 2, true);
Interlocked.Increment(ref writeTriggers);
}
}));
}
// Handle create/delete threads
int handleLifecycles = 0;
for (int i = 0; i < threadCount; i++)
{
int randSeed = i;
testThreads.Add(new Thread(() =>
{
int maxAddress = threadCount * handlesPerThread * PageSize;
Random random = new Random(randSeed + 512);
while (Stopwatch.GetTimestamp() < finishedTime)
{
RegionHandle handle = _tracking.BeginTracking((ulong)random.Next(maxAddress), (ulong)random.Next(65536));
handle.Dispose();
Interlocked.Increment(ref handleLifecycles);
}
}));
}
finishedTime = Stopwatch.GetTimestamp() + Stopwatch.Frequency / 2; // Run for 500ms;
foreach (Thread thread in testThreads)
{
thread.Start();
}
foreach (Thread thread in testThreads)
{
thread.Join();
}
Assert.Greater(dirtyFlagReprotects, 10);
Assert.Greater(writeTriggers, 10);
Assert.Greater(handleLifecycles, 10);
}
[Test]
public void ReadActionThreadConsumption()
{
// Read actions should only be triggered once for each registration.
// The implementation should use an interlocked exchange to make sure other threads can't get the action.
RegionHandle handle = _tracking.BeginTracking(0, PageSize);
int triggeredCount = 0;
int registeredCount = 0;
int signalThreadsDone = 0;
bool isRegistered = false;
Action registerReadAction = () =>
{
registeredCount++;
handle.RegisterAction((address, size) =>
{
isRegistered = false;
Interlocked.Increment(ref triggeredCount);
});
};
const int threadCount = 16;
const int iterationCount = 10000;
Thread[] signalThreads = new Thread[threadCount];
for (int i = 0; i < threadCount; i++)
{
int randSeed = i;
signalThreads[i] = new Thread(() =>
{
Random random = new Random(randSeed);
for (int j = 0; j < iterationCount; j++)
{
_tracking.VirtualMemoryEvent((ulong)random.Next(PageSize), 4, false);
}
Interlocked.Increment(ref signalThreadsDone);
});
}
for (int i = 0; i < threadCount; i++)
{
signalThreads[i].Start();
}
while (signalThreadsDone != -1)
{
if (signalThreadsDone == threadCount)
{
signalThreadsDone = -1;
}
if (!isRegistered)
{
isRegistered = true;
registerReadAction();
}
}
// The action should trigger exactly once for every registration,
// then we register once after all the threads signalling it cease.
Assert.AreEqual(registeredCount, triggeredCount + 1);
}
[Test]
public void PhysicalMemoryMapping()
{
// Tracking is done in the virtual space usually, but we also support tracking on physical regions.
// The physical regions that make up a virtual region are determined when the region is created,
// or when a mapping changes.
// These tests verify that the region cannot be signalled after unmapping, and can after remapping.
RegionHandle handle = _tracking.BeginTracking(PageSize, PageSize);
Assert.True(handle.Dirty);
bool trackedWriteTriggers = TestSingleWrite(handle, PageSize, 1, true);
Assert.True(trackedWriteTriggers);
_memoryManager.NoMappings = true;
_tracking.Unmap(PageSize, PageSize);
bool unmappedWriteTriggers = TestSingleWrite(handle, PageSize, 1, true);
Assert.False(unmappedWriteTriggers);
_memoryManager.NoMappings = false;
_tracking.Map(PageSize, PageSize, PageSize);
bool remappedWriteTriggers = TestSingleWrite(handle, PageSize, 1, true);
Assert.True(remappedWriteTriggers);
}
[Test]
public void DisposeHandles()
{
// Ensure that disposed handles correctly remove their virtual and physical regions.
RegionHandle handle = _tracking.BeginTracking(0, PageSize);
handle.Reprotect();
Assert.AreEqual((1, 1), _tracking.GetRegionCounts());
handle.Dispose();
Assert.AreEqual((0, 0), _tracking.GetRegionCounts());
// Two handles, small entirely contains big.
// We expect there to be three regions after creating both, one for the small region and two covering the big one around it.
// Regions are always split to avoid overlapping, which is why there are three instead of two.
RegionHandle handleSmall = _tracking.BeginTracking(PageSize, PageSize);
RegionHandle handleBig = _tracking.BeginTracking(0, PageSize * 4);
Assert.AreEqual((3, 3), _tracking.GetRegionCounts());
// After disposing the big region, only the small one will remain.
handleBig.Dispose();
Assert.AreEqual((1, 1), _tracking.GetRegionCounts());
handleSmall.Dispose();
Assert.AreEqual((0, 0), _tracking.GetRegionCounts());
}
}
}

16
Ryujinx.Memory/Range/INonOverlappingRange.cs Normal file
View file

@ -0,0 +1,16 @@
namespace Ryujinx.Memory.Range
{
/// <summary>
/// Range of memory that can be split in two.
/// </summary>
interface INonOverlappingRange : IRange
{
/// <summary>
/// Split this region into two, around the specified address.
/// This region is updated to end at the split address, and a new region is created to represent past that point.
/// </summary>
/// <param name="splitAddress">Address to split the region around</param>
/// <returns>The second part of the split region, with start address at the given split.</returns>
public INonOverlappingRange Split(ulong splitAddress);
}
}

31
Ryujinx.Memory/Range/IRange.cs Normal file
View file

@ -0,0 +1,31 @@
namespace Ryujinx.Memory.Range
{
/// <summary>
/// Range of memory.
/// </summary>
public interface IRange
{
/// <summary>
/// Base address.
/// </summary>
ulong Address { get; }
/// <summary>
/// Size of the range.
/// </summary>
ulong Size { get; }
/// <summary>
/// End address.
/// </summary>
ulong EndAddress { get; }
/// <summary>
/// Check if this range overlaps with another.
/// </summary>
/// <param name="address">Base address</param>
/// <param name="size">Size of the range</param>
/// <returns>True if overlapping, false otherwise</returns>
bool OverlapsWith(ulong address, ulong size);
}
}

108
Ryujinx.Memory/Range/NonOverlappingRangeList.cs Normal file
View file

@ -0,0 +1,108 @@
using System;
using System.Collections.Generic;
namespace Ryujinx.Memory.Range
{
/// <summary>
/// A range list that assumes ranges are non-overlapping, with list items that can be split in two to avoid overlaps.
/// </summary>
/// <typeparam name="T">Type of the range.</typeparam>
class NonOverlappingRangeList<T> : RangeList<T> where T : INonOverlappingRange
{
/// <summary>
/// Finds a list of regions that cover the desired (address, size) range.
/// If this range starts or ends in the middle of an existing region, it is split and only the relevant part is added.
/// If there is no matching region, or there is a gap, then new regions are created with the factory.
/// Regions are added to the list in address ascending order.
/// </summary>
/// <param name="list">List to add found regions to</param>
/// <param name="address">Start address of the search region</param>
/// <param name="size">Size of the search region</param>
/// <param name="factory">Factory for creating new ranges</param>
public void GetOrAddRegions(List<T> list, ulong address, ulong size, Func<ulong, ulong, T> factory)
{
// (regarding the specific case this generalized function is used for)
// A new region may be split into multiple parts if multiple virtual regions have mapped to it.
// For instance, while a virtual mapping could cover 0-2 in physical space, the space 0-1 may have already been reserved...
// So we need to return both the split 0-1 and 1-2 ranges.
var results = new T[1];
int count = FindOverlapsNonOverlapping(address, size, ref results);
if (count == 0)
{
// The region is fully unmapped. Create and add it to the range list.
T region = factory(address, size);
list.Add(region);
Add(region);
}
else
{
ulong lastAddress = address;
ulong endAddress = address + size;
for (int i = 0; i < count; i++)
{
T region = results[i];
if (count == 1 && region.Address == address && region.Size == size)
{
// Exact match, no splitting required.
list.Add(region);
return;
}
if (lastAddress < region.Address)
{
// There is a gap between this region and the last. We need to fill it.
T fillRegion = factory(lastAddress, region.Address - lastAddress);
list.Add(fillRegion);
Add(fillRegion);
}
if (region.Address < address)
{
// Split the region around our base address and take the high half.
region = Split(region, address);
}
if (region.EndAddress > address + size)
{
// Split the region around our end address and take the low half.
Split(region, address + size);
}
list.Add(region);
lastAddress = region.EndAddress;
}
if (lastAddress < endAddress)
{
// There is a gap between this region and the end. We need to fill it.
T fillRegion = factory(lastAddress, endAddress - lastAddress);
list.Add(fillRegion);
Add(fillRegion);
}
}
}
/// <summary>
/// Splits a region around a target point and updates the region list.
/// The original region's size is modified, but its address stays the same.
/// A new region starting from the split address is added to the region list and returned.
/// </summary>
/// <param name="region">The region to split</param>
/// <param name="splitAddress">The address to split with</param>
/// <returns>The new region (high part)</returns>
private T Split(T region, ulong splitAddress)
{
Remove(region);
T newRegion = (T)region.Split(splitAddress);
Add(region);
Add(newRegion);
return newRegion;
}
}
}

35
Ryujinx.Graphics.Gpu/Memory/RangeList.cs → Ryujinx.Memory/Range/RangeList.cs
View file

@ -2,20 +2,22 @@ using System;
using System.Collections;
using System.Collections.Generic;
namespace Ryujinx.Graphics.Gpu.Memory
namespace Ryujinx.Memory.Range
{
/// <summary>
/// List of GPU resources with data on guest memory.
/// Sorted list of ranges that supports binary search.
/// </summary>
/// <typeparam name="T">Type of the GPU resource</typeparam>
class RangeList<T> : IEnumerable<T> where T : IRange
/// <typeparam name="T">Type of the range.</typeparam>
public class RangeList<T> : IEnumerable<T> where T : IRange
{
private const int ArrayGrowthSize = 32;
private readonly List<T> _items;
public int Count => _items.Count;
/// <summary>
/// Creates a new GPU resources list.
/// Creates a new range list.
/// </summary>
public RangeList()
{
@ -135,24 +137,21 @@ namespace Ryujinx.Graphics.Gpu.Memory
ulong endAddress = address + size;
lock (_items)
foreach (T item in _items)
{
foreach (T item in _items)
if (item.Address >= endAddress)
{
if (item.Address >= endAddress)
break;
}
if (item.OverlapsWith(address, size))
{
if (outputIndex == output.Length)
{
break;
Array.Resize(ref output, outputIndex + ArrayGrowthSize);
}
if (item.OverlapsWith(address, size))
{
if (outputIndex == output.Length)
{
Array.Resize(ref output, outputIndex + ArrayGrowthSize);
}
output[outputIndex++] = item;
}
output[outputIndex++] = item;
}
}

2
Ryujinx.Memory/Ryujinx.Memory.csproj
View file

@ -1,4 +1,4 @@
<Project Sdk="Microsoft.NET.Sdk">
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>

63
Ryujinx.Memory/Tracking/AbstractRegion.cs Normal file
View file

@ -0,0 +1,63 @@
using Ryujinx.Memory.Range;
namespace Ryujinx.Memory.Tracking
{
/// <summary>
/// A region of memory.
/// </summary>
abstract class AbstractRegion : INonOverlappingRange
{
/// <summary>
/// Base address.
/// </summary>
public ulong Address { get; }
/// <summary>
/// Size of the range in bytes.
/// </summary>
public ulong Size { get; protected set; }
/// <summary>
/// End address.
/// </summary>
public ulong EndAddress => Address + Size;
/// <summary>
/// Create a new region.
/// </summary>
/// <param name="address">Base address</param>
/// <param name="size">Size of the range</param>
protected AbstractRegion(ulong address, ulong size)
{
Address = address;
Size = size;
}
/// <summary>
/// Check if this range overlaps with another.
/// </summary>
/// <param name="address">Base address</param>
/// <param name="size">Size of the range</param>
/// <returns>True if overlapping, false otherwise</returns>
public bool OverlapsWith(ulong address, ulong size)
{
return Address < address + size && address < EndAddress;
}
/// <summary>
/// Signals to the handles that a memory event has occurred, and unprotects the region. Assumes that the tracking lock has been obtained.
/// </summary>
/// <param name="address">Address accessed</param>
/// <param name="size">Size of the region affected in bytes</param>
/// <param name="write">Whether the region was written to or read</param>
public abstract void Signal(ulong address, ulong size, bool write);
/// <summary>
/// Split this region into two, around the specified address.
/// This region is updated to end at the split address, and a new region is created to represent past that point.
/// </summary>
/// <param name="splitAddress">Address to split the region around</param>
/// <returns>The second part of the split region, with start address at the given split.</returns>
public abstract INonOverlappingRange Split(ulong splitAddress);
}
}

48
Ryujinx.Memory/Tracking/IMultiRegionHandle.cs Normal file
View file

@ -0,0 +1,48 @@
using System;
namespace Ryujinx.Memory.Tracking
{
public interface IMultiRegionHandle : IDisposable
{
/// <summary>
/// True if any write has occurred to the whole region since the last use of QueryModified (with no subregion specified).
/// </summary>
bool Dirty { get; }
/// <summary>
/// Check if any part of the region has been modified, and perform an action for each.
/// Contiguous modified regions are combined.
/// </summary>
/// <param name="modifiedAction">Action to perform for modified regions</param>
void QueryModified(Action<ulong, ulong> modifiedAction);
/// <summary>
/// Check if part of the region has been modified within a given range, and perform an action for each.
/// The range is aligned to the level of granularity of the contained handles.
/// Contiguous modified regions are combined.
/// </summary>
/// <param name="address">Start address of the range</param>
/// <param name="size">Size of the range</param>
/// <param name="modifiedAction">Action to perform for modified regions</param>
void QueryModified(ulong address, ulong size, Action<ulong, ulong> modifiedAction);
/// <summary>
/// Check if part of the region has been modified within a given range, and perform an action for each.
/// The sequence number provided is compared with each handle's saved sequence number.
/// If it is equal, then the handle's dirty flag is ignored. Otherwise, the sequence number is saved.
/// The range is aligned to the level of granularity of the contained handles.
/// Contiguous modified regions are combined.
/// </summary>
/// <param name="address">Start address of the range</param>
/// <param name="size">Size of the range</param>
/// <param name="modifiedAction">Action to perform for modified regions</param>
/// <param name="sequenceNumber">Current sequence number</param>
void QueryModified(ulong address, ulong size, Action<ulong, ulong> modifiedAction, int sequenceNumber);
/// <summary>
/// Signal that one of the subregions of this multi-region has been modified. This sets the overall dirty flag.
/// </summary>
void SignalWrite();
}
}

16
Ryujinx.Memory/Tracking/IRegionHandle.cs Normal file
View file

@ -0,0 +1,16 @@
using System;
namespace Ryujinx.Memory.Tracking
{
public interface IRegionHandle : IDisposable
{
bool Dirty { get; }
ulong Address { get; }
ulong Size { get; }
ulong EndAddress { get; }
void Reprotect();
void RegisterAction(RegionSignal action);
}
}

9
Ryujinx.Memory/Tracking/IVirtualMemoryManager.cs Normal file
View file

@ -0,0 +1,9 @@
namespace Ryujinx.Memory.Tracking
{
public interface IVirtualMemoryManager
{
(ulong address, ulong size)[] GetPhysicalRegions(ulong va, ulong size);
void TrackingReprotect(ulong va, ulong size, MemoryPermission protection);
}
}

321
Ryujinx.Memory/Tracking/MemoryTracking.cs Normal file
View file

@ -0,0 +1,321 @@
using Ryujinx.Memory.Range;
using System.Collections.Generic;
namespace Ryujinx.Memory.Tracking
{
/// <summary>
/// Manages memory tracking for a given virutal/physical memory block.
/// </summary>
public class MemoryTracking
{
private readonly IVirtualMemoryManager _memoryManager;
private readonly MemoryBlock _block;
// Only use these from within the lock.
private readonly NonOverlappingRangeList<VirtualRegion> _virtualRegions;
private readonly NonOverlappingRangeList<PhysicalRegion> _physicalRegions;
// Only use these from within the lock.
private readonly VirtualRegion[] _virtualResults = new VirtualRegion[10];
private readonly PhysicalRegion[] _physicalResults = new PhysicalRegion[10];
private readonly int _pageSize;
/// <summary>
/// This lock must be obtained when traversing or updating the region-handle hierarchy.
/// It is not required when reading dirty flags.
/// </summary>
internal object TrackingLock = new object();
public bool EnablePhysicalProtection { get; set; }
/// <summary>
/// Create a new tracking structure for the given "physical" memory block,
/// with a given "virtual" memory manager that will provide mappings and virtual memory protection.
/// </summary>
/// <param name="memoryManager">Virtual memory manager</param>
/// <param name="block">Physical memory block</param>
/// <param name="pageSize">Page size of the virtual memory space</param>
public MemoryTracking(IVirtualMemoryManager memoryManager, MemoryBlock block, int pageSize)
{
_memoryManager = memoryManager;
_block = block;
_pageSize = pageSize;
_virtualRegions = new NonOverlappingRangeList<VirtualRegion>();
_physicalRegions = new NonOverlappingRangeList<PhysicalRegion>();
}
private (ulong address, ulong size) PageAlign(ulong address, ulong size)
{
ulong pageMask = (ulong)_pageSize - 1;
ulong rA = address & ~pageMask;
ulong rS = ((address + size + pageMask) & ~pageMask) - rA;
return (rA, rS);
}
/// <summary>
/// Indicate that a virtual region has been mapped, and which physical region it has been mapped to.
/// Should be called after the mapping is complete.
/// </summary>
/// <param name="va">Virtual memory address</param>
/// <param name="pa">Physical memory address</param>
/// <param name="size">Size to be mapped</param>
public void Map(ulong va, ulong pa, ulong size)
{
// A mapping may mean we need to re-evaluate each VirtualRegion's affected area.
// Find all handles that overlap with the range, we need to recalculate their physical regions
lock (TrackingLock)
{
var results = _virtualResults;
int count = _virtualRegions.FindOverlapsNonOverlapping(va, size, ref results);
for (int i = 0; i < count; i++)
{
VirtualRegion region = results[i];
region.RecalculatePhysicalChildren();
}
}
}
/// <summary>
/// Indicate that a virtual region has been unmapped.
/// Should be called after the unmapping is complete.
/// </summary>
/// <param name="va">Virtual memory address</param>
/// <param name="size">Size to be unmapped</param>
public void Unmap(ulong va, ulong size)
{
// An unmapping may mean we need to re-evaluate each VirtualRegion's affected area.
// Find all handles that overlap with the range, we need to recalculate their physical regions
lock (TrackingLock)
{
var results = _virtualResults;
int count = _virtualRegions.FindOverlapsNonOverlapping(va, size, ref results);
for (int i = 0; i < count; i++)
{
VirtualRegion region = results[i];
region.RecalculatePhysicalChildren();
}
}
}
/// <summary>
/// Get a list of virtual regions that a handle covers.
/// </summary>
/// <param name="va">Starting virtual memory address of the handle</param>
/// <param name="size">Size of the handle's memory region</param>
/// <returns>A list of virtual regions within the given range</returns>
internal List<VirtualRegion> GetVirtualRegionsForHandle(ulong va, ulong size)
{
List<VirtualRegion> result = new List<VirtualRegion>();
_virtualRegions.GetOrAddRegions(result, va, size, (va, size) => new VirtualRegion(this, va, size));
return result;
}
/// <summary>
/// Get a list of physical regions that a virtual region covers.
/// Note that this becomes outdated if the virtual or physical regions are unmapped or remapped.
/// </summary>
/// <param name="va">Virtual memory address</param>
/// <param name="size">Size of the virtual region</param>
/// <returns>A list of physical regions the virtual region covers</returns>
internal List<PhysicalRegion> GetPhysicalRegionsForVirtual(ulong va, ulong size)
{
List<PhysicalRegion> result = new List<PhysicalRegion>();
// Get a list of physical regions for this virtual region, from our injected virtual mapping function.
(ulong Address, ulong Size)[] physicalRegions = _memoryManager.GetPhysicalRegions(va, size);
if (physicalRegions != null)
{
foreach (var region in physicalRegions)
{
_physicalRegions.GetOrAddRegions(result, region.Address, region.Size, (pa, size) => new PhysicalRegion(this, pa, size));
}
}
return result;
}
/// <summary>
/// Remove a virtual region from the range list. This assumes that the lock has been acquired.
/// </summary>
/// <param name="region">Region to remove</param>
internal void RemoveVirtual(VirtualRegion region)
{
_virtualRegions.Remove(region);
}
/// <summary>
/// Remove a physical region from the range list. This assumes that the lock has been acquired.
/// </summary>
/// <param name="region">Region to remove</param>
internal void RemovePhysical(PhysicalRegion region)
{
_physicalRegions.Remove(region);
}
/// <summary>
/// Obtains a memory tracking handle for the given virtual region, with a specified granularity. This should be disposed when finished with.
/// </summary>
/// <param name="address">CPU virtual address of the region</param>
/// <param name="size">Size of the region</param>
/// <param name="granularity">Desired granularity of write tracking</param>
/// <returns>The memory tracking handle</returns>
public MultiRegionHandle BeginGranularTracking(ulong address, ulong size, ulong granularity)
{
(address, size) = PageAlign(address, size);
return new MultiRegionHandle(this, address, size, granularity);
}
/// <summary>
/// Obtains a smart memory tracking handle for the given virtual region, with a specified granularity. This should be disposed when finished with.
/// </summary>
/// <param name="address">CPU virtual address of the region</param>
/// <param name="size">Size of the region</param>
/// <param name="granularity">Desired granularity of write tracking</param>
/// <returns>The memory tracking handle</returns>
public SmartMultiRegionHandle BeginSmartGranularTracking(ulong address, ulong size, ulong granularity)
{
(address, size) = PageAlign(address, size);
return new SmartMultiRegionHandle(this, address, size, granularity);
}
/// <summary>
/// Obtains a memory tracking handle for the given virtual region. This should be disposed when finished with.
/// </summary>
/// <param name="address">CPU virtual address of the region</param>
/// <param name="size">Size of the region</param>
/// <returns>The memory tracking handle</returns>
public RegionHandle BeginTracking(ulong address, ulong size)
{
(address, size) = PageAlign(address, size);
lock (TrackingLock)
{
RegionHandle handle = new RegionHandle(this, address, size);
return handle;
}
}
/// <summary>
/// Signal that a physical memory event happened at the given location.
/// </summary>
/// <param name="address">Physical address accessed</param>
/// <param name="write">Whether the region was written to or read</param>
/// <returns>True if the event triggered any tracking regions, false otherwise</returns>
public bool PhysicalMemoryEvent(ulong address, bool write)
{
// Look up the physical region using the region list.
// Signal up the chain to relevant handles.
lock (TrackingLock)
{
var results = _physicalResults;
int count = _physicalRegions.FindOverlapsNonOverlapping(address, 1, ref results); // TODO: get/use the actual access size?
if (count == 0)
{
_block.Reprotect(address & ~(ulong)(_pageSize - 1), (ulong)_pageSize, MemoryPermission.ReadAndWrite);
return false; // We can't handle this - unprotect and return.
}
for (int i = 0; i < count; i++)
{
PhysicalRegion region = results[i];
region.Signal(address, 1, write);
}
}
return true;
}
/// <summary>
/// Signal that a virtual memory event happened at the given location (one byte).
/// </summary>
/// <param name="address">Virtual address accessed</param>
/// <param name="write">Whether the address was written to or read</param>
/// <returns>True if the event triggered any tracking regions, false otherwise</returns>
public bool VirtualMemoryEventTracking(ulong address, bool write)
{
return VirtualMemoryEvent(address, 1, write);
}
/// <summary>
/// Signal that a virtual memory event happened at the given location.
/// </summary>
/// <param name="address">Virtual address accessed</param>
/// <param name="size">Size of the region affected in bytes</param>
/// <param name="write">Whether the region was written to or read</param>
/// <returns>True if the event triggered any tracking regions, false otherwise</returns>
public bool VirtualMemoryEvent(ulong address, ulong size, bool write)
{
// Look up the virtual region using the region list.
// Signal up the chain to relevant handles.
lock (TrackingLock)
{
var results = _virtualResults;
int count = _virtualRegions.FindOverlapsNonOverlapping(address, size, ref results);
if (count == 0)
{
_memoryManager.TrackingReprotect(address & ~(ulong)(_pageSize - 1), (ulong)_pageSize, MemoryPermission.ReadAndWrite);
return false; // We can't handle this - it's probably a real invalid access.
}
for (int i = 0; i < count; i++)
{
VirtualRegion region = results[i];
region.Signal(address, size, write);
}
}
return true;
}
/// <summary>
/// Reprotect a given physical region, if enabled. This is protected on the memory block provided during initialization.
/// </summary>
/// <param name="region">Region to reprotect</param>
/// <param name="permission">Memory permission to protect with</param>
internal void ProtectPhysicalRegion(PhysicalRegion region, MemoryPermission permission)
{
if (EnablePhysicalProtection)
{
_block.Reprotect(region.Address, region.Size, permission);
}
}
/// <summary>
/// Reprotect a given virtual region. The virtual memory manager will handle this.
/// </summary>
/// <param name="region">Region to reprotect</param>
/// <param name="permission">Memory permission to protect with</param>
internal void ProtectVirtualRegion(VirtualRegion region, MemoryPermission permission)
{
_memoryManager.TrackingReprotect(region.Address, region.Size, permission);
}
/// <summary>
/// Returns the number of virtual and physical regions currently being tracked.
/// Useful for tests and metrics.
/// </summary>
/// <returns>The number of virtual regions, and the number of physical regions</returns>
public (int VirtualCount, int PhysicalCount) GetRegionCounts()
{
lock (TrackingLock)
{
return (_virtualRegions.Count, _physicalRegions.Count);
}
}
}
}

134
Ryujinx.Memory/Tracking/MultiRegionHandle.cs Normal file
View file

@ -0,0 +1,134 @@
using System;
namespace Ryujinx.Memory.Tracking
{
/// <summary>
/// A region handle that tracks a large region using many smaller handles, to provide
/// granular tracking that can be used to track partial updates.
/// </summary>
public class MultiRegionHandle : IMultiRegionHandle
{
/// <summary>
/// A list of region handles for each granularity sized chunk of the whole region.
/// </summary>
private readonly RegionHandle[] _handles;
private readonly ulong Address;
private readonly ulong Granularity;
private readonly ulong Size;
public bool Dirty { get; private set; } = true;
internal MultiRegionHandle(MemoryTracking tracking, ulong address, ulong size, ulong granularity)
{
_handles = new RegionHandle[size / granularity];
Granularity = granularity;
for (int i = 0; i < _handles.Length; i++)
{
RegionHandle handle = tracking.BeginTracking(address + (ulong)i * granularity, granularity);
handle.Parent = this;
_handles[i] = handle;
}
Address = address;
Size = size;
}
public void SignalWrite()
{
Dirty = true;
}
public void QueryModified(Action<ulong, ulong> modifiedAction)
{
if (!Dirty)
{
return;
}
Dirty = false;
QueryModified(Address, Size, modifiedAction);
}
public void QueryModified(ulong address, ulong size, Action<ulong, ulong> modifiedAction)
{
int startHandle = (int)((address - Address) / Granularity);
int lastHandle = (int)((address + (size - 1) - Address) / Granularity);
ulong rgStart = _handles[startHandle].Address;
ulong rgSize = 0;
for (int i = startHandle; i <= lastHandle; i++)
{
RegionHandle handle = _handles[i];
if (handle.Dirty)
{
rgSize += handle.Size;
handle.Reprotect();
}
else
{
// Submit the region scanned so far as dirty
if (rgSize != 0)
{
modifiedAction(rgStart, rgSize);
rgSize = 0;
}
rgStart = handle.EndAddress;
}
}
if (rgSize != 0)
{
modifiedAction(rgStart, rgSize);
}
}
public void QueryModified(ulong address, ulong size, Action<ulong, ulong> modifiedAction, int sequenceNumber)
{
int startHandle = (int)((address - Address) / Granularity);
int lastHandle = (int)((address + (size - 1) - Address) / Granularity);
ulong rgStart = _handles[startHandle].Address;
ulong rgSize = 0;
for (int i = startHandle; i <= lastHandle; i++)
{
RegionHandle handle = _handles[i];
if (handle.Dirty && sequenceNumber != handle.SequenceNumber)
{
rgSize += handle.Size;
handle.Reprotect();
}
else
{
// Submit the region scanned so far as dirty
if (rgSize != 0)
{
modifiedAction(rgStart, rgSize);
rgSize = 0;
}
rgStart = handle.EndAddress;
}
handle.SequenceNumber = sequenceNumber;
}
if (rgSize != 0)
{
modifiedAction(rgStart, rgSize);
}
}
public void Dispose()
{
foreach (var handle in _handles)
{
handle.Dispose();
}
}
}
}

97
Ryujinx.Memory/Tracking/PhysicalRegion.cs Normal file
View file

@ -0,0 +1,97 @@
using Ryujinx.Memory.Range;
using System.Collections.Generic;
namespace Ryujinx.Memory.Tracking
{
/// <summary>
/// A region of physical memory.
/// </summary>
class PhysicalRegion : AbstractRegion
{
public List<VirtualRegion> VirtualParents = new List<VirtualRegion>();
public MemoryPermission Protection { get; private set; }
public MemoryTracking Tracking;
public PhysicalRegion(MemoryTracking tracking, ulong address, ulong size) : base(address, size)
{
Tracking = tracking;
Protection = MemoryPermission.ReadAndWrite;
}
public override void Signal(ulong address, ulong size, bool write)
{
Protection = MemoryPermission.ReadAndWrite;
Tracking.ProtectPhysicalRegion(this, MemoryPermission.ReadAndWrite); // Remove our protection immedately.
foreach (var parent in VirtualParents)
{
parent.Signal(address, size, write);
}
}
/// <summary>
/// Update the protection of this region, based on our parent's requested protection.
/// </summary>
public void UpdateProtection()
{
// Re-evaluate protection, and commit to the block.
lock (Tracking.TrackingLock)
{
MemoryPermission result = MemoryPermission.ReadAndWrite;
foreach (var parent in VirtualParents)
{
result &= parent.GetRequiredPermission();
if (result == 0) break;
}
if (Protection != result)
{
Protection = result;
Tracking.ProtectPhysicalRegion(this, result);
}
}
}
public override INonOverlappingRange Split(ulong splitAddress)
{
PhysicalRegion newRegion = new PhysicalRegion(Tracking, splitAddress, EndAddress - splitAddress);
Size = splitAddress - Address;
// The new region inherits all of our parents.
newRegion.VirtualParents = new List<VirtualRegion>(VirtualParents);
foreach (var parent in VirtualParents)
{
parent.AddChild(newRegion);
}
return newRegion;
}
/// <summary>
/// Remove a parent virtual region from this physical region. Assumes that the tracking lock has been obtained.
/// </summary>
/// <param name="region">Region to remove</param>
/// <returns>True if there are no more parents and we should be removed, false otherwise.</returns>
public bool RemoveParent(VirtualRegion region)
{
VirtualParents.Remove(region);
UpdateProtection();
if (VirtualParents.Count == 0)
{
return true;
}
return false;
}
/// <summary>
/// Deletes this physical region if there are no more virtual parents.
/// </summary>
public void TryDelete()
{
if (VirtualParents.Count == 0)
{
Tracking.RemovePhysical(this);
}
}
}
}

134
Ryujinx.Memory/Tracking/RegionHandle.cs Normal file
View file

@ -0,0 +1,134 @@
using Ryujinx.Memory.Range;
using System.Collections.Generic;
using System.Threading;
namespace Ryujinx.Memory.Tracking
{
/// <summary>
/// A tracking handle for a given region of virtual memory. The Dirty flag is updated whenever any changes are made,
/// and an action can be performed when the region is read to or written from.
/// </summary>
public class RegionHandle : IRegionHandle, IRange
{
public bool Dirty { get; private set; } = true;
public ulong Address { get; }
public ulong Size { get; }
public ulong EndAddress { get; }
internal IMultiRegionHandle Parent { get; set; }
internal int SequenceNumber { get; set; }
private RegionSignal _preAction; // Action to perform before a read or write. This will block the memory access.
private readonly List<VirtualRegion> _regions;
private readonly MemoryTracking _tracking;
internal MemoryPermission RequiredPermission => _preAction != null ? MemoryPermission.None : (Dirty ? MemoryPermission.ReadAndWrite : MemoryPermission.Read);
/// <summary>
/// Create a new region handle. The handle is registered with the given tracking object,
/// and will be notified of any changes to the specified region.
/// </summary>
/// <param name="tracking">Tracking object for the target memory block</param>
/// <param name="address">Virtual address of the region to track</param>
/// <param name="size">Size of the region to track</param>
internal RegionHandle(MemoryTracking tracking, ulong address, ulong size)
{
Address = address;
Size = size;
EndAddress = address + size;
_tracking = tracking;
_regions = tracking.GetVirtualRegionsForHandle(address, size);
foreach (var region in _regions)
{
region.Handles.Add(this);
}
}
/// <summary>
/// Signal that a memory action occurred within this handle's virtual regions.
/// </summary>
/// <param name="write">Whether the region was written to or read</param>
internal void Signal(ulong address, ulong size, bool write)
{
RegionSignal action = Interlocked.Exchange(ref _preAction, null);
action?.Invoke(address, size);
if (write)
{
Dirty = true;
Parent?.SignalWrite();
}
}
/// <summary>
/// Consume the dirty flag for this handle, and reprotect so it can be set on the next write.
/// </summary>
public void Reprotect()
{
Dirty = false;
lock (_tracking.TrackingLock)
{
foreach (VirtualRegion region in _regions)
{
region.UpdateProtection();
}
}
}
/// <summary>
/// Register an action to perform when the tracked region is read or written.
/// The action is automatically removed after it runs.
/// </summary>
/// <param name="action">Action to call on read or write</param>
public void RegisterAction(RegionSignal action)
{
RegionSignal lastAction = Interlocked.Exchange(ref _preAction, action);
if (lastAction == null && action != lastAction)
{
lock (_tracking.TrackingLock)
{
foreach (VirtualRegion region in _regions)
{
region.UpdateProtection();
}
}
}
}
/// <summary>
/// Add a child virtual region to this handle.
/// </summary>
/// <param name="region">Virtual region to add as a child</param>
internal void AddChild(VirtualRegion region)
{
_regions.Add(region);
}
/// <summary>
/// Check if this region overlaps with another.
/// </summary>
/// <param name="address">Base address</param>
/// <param name="size">Size of the region</param>
/// <returns>True if overlapping, false otherwise</returns>
public bool OverlapsWith(ulong address, ulong size)
{
return Address < address + size && address < EndAddress;
}
/// <summary>
/// Dispose the handle. Within the tracking lock, this removes references from virtual and physical regions.
/// </summary>
public void Dispose()
{
lock (_tracking.TrackingLock)
{
foreach (VirtualRegion region in _regions)
{
region.RemoveHandle(this);
}
}
}
}
}

4
Ryujinx.Memory/Tracking/RegionSignal.cs Normal file
View file

@ -0,0 +1,4 @@
namespace Ryujinx.Memory.Tracking
{
public delegate void RegionSignal(ulong address, ulong size);
}

236
Ryujinx.Memory/Tracking/SmartMultiRegionHandle.cs Normal file
View file

@ -0,0 +1,236 @@
using System;
using System.Runtime.CompilerServices;
namespace Ryujinx.Memory.Tracking
{
/// <summary>
/// A MultiRegionHandle that attempts to segment a region's handles into the regions requested
/// to avoid iterating over granular chunks for canonically large regions.
/// If minimum granularity is to be expected, use MultiRegionHandle.
/// </summary>
public class SmartMultiRegionHandle : IMultiRegionHandle
{
/// <summary>
/// A list of region handles starting at each granularity size increment.
/// </summary>
private readonly RegionHandle[] _handles;
private readonly ulong _address;
private readonly ulong _granularity;
private readonly ulong _size;
private MemoryTracking _tracking;
public bool Dirty { get; private set; } = true;
internal SmartMultiRegionHandle(MemoryTracking tracking, ulong address, ulong size, ulong granularity)
{
// For this multi-region handle, the handle list starts empty.
// As regions are queried, they are added to the _handles array at their start index.
// When a region being added overlaps another, the existing region is split.
// A query can therefore scan multiple regions, though with no overlaps they can cover a large area.
_tracking = tracking;
_handles = new RegionHandle[size / granularity];
_granularity = granularity;
_address = address;
_size = size;
}
public void SignalWrite()
{
Dirty = true;
}
public void QueryModified(Action<ulong, ulong> modifiedAction)
{
if (!Dirty)
{
return;
}
Dirty = false;
QueryModified(_address, _size, modifiedAction);
}
[MethodImpl(MethodImplOptions.AggressiveInlining)]
private ulong HandlesToBytes(int handles)
{
return (ulong)handles * _granularity;
}
private void SplitHandle(int handleIndex, int splitIndex)
{
RegionHandle handle = _handles[handleIndex];
ulong address = _address + HandlesToBytes(handleIndex);
ulong size = HandlesToBytes(splitIndex - handleIndex);
// First, the target handle must be removed. Its data can still be used to determine the new handles.
handle.Dispose();
RegionHandle splitLow = _tracking.BeginTracking(address, size);
splitLow.Parent = this;
_handles[handleIndex] = splitLow;
RegionHandle splitHigh = _tracking.BeginTracking(address + size, handle.Size - size);
splitHigh.Parent = this;
_handles[splitIndex] = splitHigh;
}
private void CreateHandle(int startHandle, int lastHandle)
{
ulong startAddress = _address + HandlesToBytes(startHandle);
// Scan for the first handle before us. If it's overlapping us, it must be split.
for (int i = startHandle - 1; i >= 0; i--)
{
RegionHandle handle = _handles[i];
if (handle != null)
{
if (handle.EndAddress > startAddress)
{
SplitHandle(i, startHandle);
return; // The remainer of this handle should be filled in later on.
}
break;
}
}
// Scan for handles after us. We should create a handle that goes up to this handle's start point, if present.
for (int i = startHandle + 1; i <= lastHandle; i++)
{
RegionHandle handle = _handles[i];
if (handle != null)
{
// Fill up to the found handle.
handle = _tracking.BeginTracking(startAddress, HandlesToBytes(i - startHandle));
handle.Parent = this;
_handles[startHandle] = handle;
return;
}
}
// Can fill the whole range.
_handles[startHandle] = _tracking.BeginTracking(startAddress, HandlesToBytes(1 + lastHandle - startHandle));
_handles[startHandle].Parent = this;
}
public void QueryModified(ulong address, ulong size, Action<ulong, ulong> modifiedAction)
{
int startHandle = (int)((address - _address) / _granularity);
int lastHandle = (int)((address + (size - 1) - _address) / _granularity);
ulong rgStart = _address + (ulong)startHandle * _granularity;
ulong rgSize = 0;
ulong endAddress = _address + ((ulong)lastHandle + 1) * _granularity;
int i = startHandle;
while (i <= lastHandle)
{
RegionHandle handle = _handles[i];
if (handle == null)
{
// Missing handle. A new handle must be created.
CreateHandle(i, lastHandle);
handle = _handles[i];
}
if (handle.EndAddress > endAddress)
{
// End address of handle is beyond the end of the search. Force a split.
SplitHandle(i, lastHandle + 1);
handle = _handles[i];
}
if (handle.Dirty)
{
rgSize += handle.Size;
handle.Reprotect();
}
else
{
// Submit the region scanned so far as dirty
if (rgSize != 0)
{
modifiedAction(rgStart, rgSize);
rgSize = 0;
}
rgStart = handle.EndAddress;
}
i += (int)(handle.Size / _granularity);
}
if (rgSize != 0)
{
modifiedAction(rgStart, rgSize);
}
}
public void QueryModified(ulong address, ulong size, Action<ulong, ulong> modifiedAction, int sequenceNumber)
{
int startHandle = (int)((address - _address) / _granularity);
int lastHandle = (int)((address + (size - 1) - _address) / _granularity);
ulong rgStart = _address + (ulong)startHandle * _granularity;
ulong rgSize = 0;
ulong endAddress = _address + ((ulong)lastHandle + 1) * _granularity;
int i = startHandle;
while (i <= lastHandle)
{
RegionHandle handle = _handles[i];
if (handle == null)
{
// Missing handle. A new handle must be created.
CreateHandle(i, lastHandle);
handle = _handles[i];
}
if (handle.EndAddress > endAddress)
{
// End address of handle is beyond the end of the search. Force a split.
SplitHandle(i, lastHandle + 1);
handle = _handles[i];
}
if (handle.Dirty && sequenceNumber != handle.SequenceNumber)
{
rgSize += handle.Size;
handle.Reprotect();
}
else
{
// Submit the region scanned so far as dirty
if (rgSize != 0)
{
modifiedAction(rgStart, rgSize);
rgSize = 0;
}
rgStart = handle.EndAddress;
}
handle.SequenceNumber = sequenceNumber;
i += (int)(handle.Size / _granularity);
}
if (rgSize != 0)
{
modifiedAction(rgStart, rgSize);
}
}
public void Dispose()
{
foreach (var handle in _handles)
{
handle?.Dispose();
}
}
}
}

165
Ryujinx.Memory/Tracking/VirtualRegion.cs Normal file
View file

@ -0,0 +1,165 @@
using Ryujinx.Memory.Range;
using System.Collections.Generic;
namespace Ryujinx.Memory.Tracking
{
/// <summary>
/// A region of virtual memory.
/// </summary>
class VirtualRegion : AbstractRegion
{
public List<RegionHandle> Handles = new List<RegionHandle>();
private List<PhysicalRegion> _physicalChildren;
private readonly MemoryTracking _tracking;
public VirtualRegion(MemoryTracking tracking, ulong address, ulong size) : base(address, size)
{
_tracking = tracking;
UpdatePhysicalChildren();
}
public override void Signal(ulong address, ulong size, bool write)
{
_tracking.ProtectVirtualRegion(this, MemoryPermission.ReadAndWrite); // Remove our protection immedately.
foreach (var handle in Handles)
{
handle.Signal(address, size, write);
}
}
/// <summary>
/// Clears all physical children of this region. Assumes that the tracking lock has been obtained.
/// </summary>
private void ClearPhysicalChildren()
{
if (_physicalChildren != null)
{
foreach (PhysicalRegion child in _physicalChildren)
{
child.RemoveParent(this);
}
}
}
/// <summary>
/// Updates the physical children of this region, assuming that they are clear and that the tracking lock has been obtained.
/// </summary>
private void UpdatePhysicalChildren()
{
_physicalChildren = _tracking.GetPhysicalRegionsForVirtual(Address, Size);
foreach (PhysicalRegion child in _physicalChildren)
{
child.VirtualParents.Add(this);
}
}
/// <summary>
/// Recalculates the physical children for this virtual region. Assumes that the tracking lock has been obtained.
/// </summary>
public void RecalculatePhysicalChildren()
{
ClearPhysicalChildren();
UpdatePhysicalChildren();
}
/// <summary>
/// Gets the strictest permission that the child handles demand. Assumes that the tracking lock has been obtained.
/// </summary>
/// <returns>Protection level that this region demands</returns>
public MemoryPermission GetRequiredPermission()
{
// Start with Read/Write, each handle can strip off permissions as necessary.
// Assumes the tracking lock has already been obtained.
MemoryPermission result = MemoryPermission.ReadAndWrite;
foreach (var handle in Handles)
{
result &= handle.RequiredPermission;
if (result == 0) return result;
}
return result;
}
/// <summary>
/// Updates the protection for this virtual region, and all child physical regions.
/// </summary>
public void UpdateProtection()
{
// Re-evaluate protection for all physical children.
_tracking.ProtectVirtualRegion(this, GetRequiredPermission());
lock (_tracking.TrackingLock)
{
foreach (var child in _physicalChildren)
{
child.UpdateProtection();
}
}
}
/// <summary>
/// Removes a handle from this virtual region. If there are no handles left, this virtual region is removed.
/// </summary>
/// <param name="handle">Handle to remove</param>
public void RemoveHandle(RegionHandle handle)
{
bool removedRegions = false;
lock (_tracking.TrackingLock)
{
Handles.Remove(handle);
UpdateProtection();
if (Handles.Count == 0)
{
_tracking.RemoveVirtual(this);
foreach (var child in _physicalChildren)
{
removedRegions |= child.RemoveParent(this);
}
}
}
if (removedRegions)
{
// The first lock will unprotect any regions that have been removed. This second lock will remove them.
lock (_tracking.TrackingLock)
{
foreach (var child in _physicalChildren)
{
child.TryDelete();
}
}
}
}
/// <summary>
/// Add a child physical region to this virtual region. Assumes that the tracking lock has been obtained.
/// </summary>
/// <param name="region">Physical region to add as a child</param>
public void AddChild(PhysicalRegion region)
{
_physicalChildren.Add(region);
}
public override INonOverlappingRange Split(ulong splitAddress)
{
ClearPhysicalChildren();
VirtualRegion newRegion = new VirtualRegion(_tracking, splitAddress, EndAddress - splitAddress);
Size = splitAddress - Address;
UpdatePhysicalChildren();
// The new region inherits all of our parents.
newRegion.Handles = new List<RegionHandle>(Handles);
foreach (var parent in Handles)
{
parent.AddChild(newRegion);
}
return newRegion;
}
}
}

8
Ryujinx.sln
View file

@ -87,6 +87,14 @@ Global
{3AB294D0-2230-468F-9EB3-BDFCAEAE99A5}.Debug|Any CPU.Build.0 = Debug|Any CPU
{3AB294D0-2230-468F-9EB3-BDFCAEAE99A5}.Release|Any CPU.ActiveCfg = Release|Any CPU
{3AB294D0-2230-468F-9EB3-BDFCAEAE99A5}.Release|Any CPU.Build.0 = Release|Any CPU
{8E7D36DD-9626-47E2-8EF5-8F2F66751C9C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{8E7D36DD-9626-47E2-8EF5-8F2F66751C9C}.Debug|Any CPU.Build.0 = Debug|Any CPU
{8E7D36DD-9626-47E2-8EF5-8F2F66751C9C}.Profile Debug|Any CPU.ActiveCfg = Profile Debug|Any CPU
{8E7D36DD-9626-47E2-8EF5-8F2F66751C9C}.Profile Debug|Any CPU.Build.0 = Profile Debug|Any CPU
{8E7D36DD-9626-47E2-8EF5-8F2F66751C9C}.Profile Release|Any CPU.ActiveCfg = Profile Release|Any CPU
{8E7D36DD-9626-47E2-8EF5-8F2F66751C9C}.Profile Release|Any CPU.Build.0 = Profile Release|Any CPU
{8E7D36DD-9626-47E2-8EF5-8F2F66751C9C}.Release|Any CPU.ActiveCfg = Release|Any CPU
{8E7D36DD-9626-47E2-8EF5-8F2F66751C9C}.Release|Any CPU.Build.0 = Release|Any CPU
{5FD4E4F6-8928-4B3C-BE07-28A675C17226}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{5FD4E4F6-8928-4B3C-BE07-28A675C17226}.Debug|Any CPU.Build.0 = Debug|Any CPU
{5FD4E4F6-8928-4B3C-BE07-28A675C17226}.Release|Any CPU.ActiveCfg = Release|Any CPU

2
Ryujinx/Ryujinx.csproj
View file

@ -77,7 +77,7 @@
<PackageReference Include="GtkSharp" Version="3.22.25.56" />
<PackageReference Include="GtkSharp.Dependencies" Version="1.1.0" Condition="'$(RuntimeIdentifier)' != 'linux-x64' AND '$(RuntimeIdentifier)' != 'osx-x64'" />
<PackageReference Include="Ryujinx.Graphics.Nvdec.Dependencies" Version="4.3.0" Condition="'$(RuntimeIdentifier)' != 'linux-x64' AND '$(RuntimeIdentifier)' != 'osx-x64'" />
<PackageReference Include="OpenTK.NetStandard" Version="1.0.5.12" />
<PackageReference Include="OpenTK.NetStandard" Version="1.0.5.22" />
<PackageReference Include="SharpZipLib" Version="1.2.0" />
</ItemGroup>

8
Ryujinx/Ui/GLRenderer.cs
View file

@ -19,6 +19,11 @@ namespace Ryujinx.Ui
{
public class GlRenderer : GLWidget
{
static GlRenderer()
{
OpenTK.Graphics.GraphicsContext.ShareContexts = true;
}
private const int SwitchPanelWidth = 1280;
private const int SwitchPanelHeight = 720;
private const int TargetFps = 60;
@ -317,6 +322,9 @@ namespace Ryujinx.Ui
public void Render()
{
// First take exclusivity on the OpenGL context.
_renderer.InitializeBackgroundContext(GraphicsContext);
Gtk.Window parent = Toplevel as Gtk.Window;
parent.Present();
GraphicsContext.MakeCurrent(WindowInfo);
_renderer.Initialize(_glLogLevel);