.NET Core 3.1: the converter … read too much or not enough

The other day, I created a custom JSON converter for the new JSON API in .NET Core 3.1. At runtime, I received an exception stating “the converter … read too much or not enough”. It took me hours to figure out what was going on.

So here’s my converter. For better readability, I placed part of the code into a method that uses a Utf8JsonReader to extract a list of numbers from the underlying JSON stream. The mentioned exception occurred after the converter had been called.

public class GeoPolygonConverter : JsonConverter<GeoPolygon>
{ 
	public override GeoPolygon Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) 
	{
		var numbers = ReadValues(reader); 
		return new GeoPolygon(numbers); 
	}

	private static IEnumerable<double> ReadValues(Utf8JsonReader reader) 
	{ 
		var numbers = new List<double>();
		// use reader to read double values.
		return numbers; 
	}
}

The exception message lead me to believe that I used the reader incorrectly or that the underlying JSON was corrupt. I spent quite a while digging in those directions until I noticed the Read method’s signature: Utf8JsonReader is a struct, and also there’s a ref keyword.

The converter … read too much or not enough: The problem

Because Utf8JsonReader is a struct, each time an instance of Utf8JsonReader is passed into a method, a copy of it is created and passed to that method instead. When the reader advances, an internal field represents the new position. The internal field reflecting the position of the reader inside the method is different than the one outside of the method, so the outer reader is unaware of the advance of the inner reader. The state of the outer reader corrupts. When the converter is finished, the outer reader is used again. However, this process fails due to the outer reader’s corrupt state. This leads to the rather confusing error message: the converter… read too much or not enough.

On the other hand, when a ref keyword is present on a method parameter of Utf8JsonReader, a copy of the reader is not created. Instead, the method passes the address to the reader. This allows code inside of the method to access the exact same reader as code outside of the method. In this case, the internal position field inside the method would be the same as outside, with no corruption occurring.

Long story short…

Add a ref keyword to method parameters of type Utf8JsonReader.

private static IEnumerable<double> ReadValues(ref Utf8JsonReader reader) 
{
	var numbers = new List<double>();
	// use reader to read double values.
	return numbers; 
}