Investigating the missing sync vars in s&box's DeltaSnapshot System

Introduction

For anyone interested in s&box's networking, I have another blog that goes into the details of how the DeltaSnapshot system works, and how it is designed to be efficient.

The Bug

So someone had pointed this issue to me that it was possible for an object to get two sync vars out of sync.

The code itself looked very simple: there is a counter that counts up and a fake boolean that flips between true and false depending on that counter. The code is below:

public sealed class SyncTestComp : Component
{
	[Property][Sync] public int Counter { get; set; }
	[Property][Sync] public int Flipper { get; set; }

	protected override void OnFixedUpdate()
	{
		if ( !IsProxy )
		{
			Counter++;

			if ( Counter % 2 == 0 )
			{
				Flipper = 0;
			}
			else
			{
				Flipper = 1;
			}
		}
	}

	protected override void OnUpdate()
	{
		if ( Counter % 2 == 0 && Flipper != 0 )
		{
			Log.Info( "Counter is even, but Flipper != 0" );
		}
		else if ( Counter % 2 != 0 && Flipper != 1 )
		{
			Log.Info( "Counter is odd, but Flipper is != 1" );
		}
	}
}

Somehow though the code in OnUpdate was logging, which was strange because you would assume if one was changed then the other would be changed as well, but that was not the case.

The Investigation

My first port of call was to repo this, I have this cool tool called Clumsy which allows you to simulate network conditions, so I set it up to lag for more than 250ms and it happened almost instantly, which was strange. What does the 250ms have to do with anything? (This would be a hint to the issue later on)

Once I was able to repo it I double checked that sync vars were being sent in the same packet, which they were. Now if you read my previous blog you would know the 250ms timer is the timer that the client has to acknowledge a snapshot, if it doesn't then the server will resend the snapshot. So I thought maybe the client was not acknowledging the snapshot in time, so I added some logging to see if that was the case, and it was! The client was not acknowledging the snapshot in time which means that older ack's were coming in late.

The code below is what checks what data to actual send via a delta snapshot cycle. You can see the timer right there!

/// <summary>
/// Try to get the hash of the value from the specified slot.
/// </summary>
public bool TryGetHash( int slot, out ulong hash, float timeNow )
{
	if ( _predictedData.TryGetValue( slot, out var predicted ) && timeNow <= predicted.ExpireTime )
	{
		hash = predicted.Hash;
		return true;
	}

	if ( Data.TryGetValue( slot, out var e ) )
	{
		hash = e.Hash;
		return true;
	}

	hash = 0;
	return false;
}

TryGetHash treats an expired prediction as if it didn't get there at all, rather than recognising that it simply didn't get back in time. The client could have received the data and processed it correctly, but the server assumes it didn't arrive and falls back to the last known value. That value could be the same as the one we are trying to send. In the true -> false -> true case, it is therefore likely that the value will not be sent again, which is why sync vars go missing.

Examples:

• predicted=2, incoming=3 -> IsNewer(2,3) = false -> remove (OK) • predicted=5, incoming=3 -> IsNewer(5,3) = true -> don't remove (keep prediction)

The Fix

Changing the code to this means that an expiry is treated as if it was never sent, so the server will send the value again, which is what we want. So we always get the right value!

/// <summary>
/// Try to get the hash of the value from the specified slot.
/// </summary>
public bool TryGetHash( int slot, out ulong hash, float timeNow )
{
	if ( _predictedData.TryGetValue( slot, out var predicted ) )
	{
		if ( timeNow <= predicted.ExpireTime )
		{
			value = predicted.Value;
			return true;
		}

		value = null;
		return false;
	}

	if ( Data.TryGetValue( slot, out var e ) )
	{
		hash = e.Hash;
		return true;
	}

	hash = 0;
	return false;
}

After the changes video:

The PR for this fix is here

Currently I am working on introducing dormant objects. This significantly reduces the amount of data sent to clients and the amount of work the server has to do.