Let Utf8JsonReader process input with one complete JSON document per line

EDIT see #33030 (comment) for an API proposal

Apologies if this ticket has been created but I missed it or there's a flag that I missed. I looked around but didn't find a good match.

I work in BigData and a common scenario is that we have large files that contains JSON documents separated by newline like so:

{}
[]
{}

This is not a valid JSON document obviously but it would help me a lot if the Utf8JsonReader could be configured to process a sequence of JSON documents like this.

My alternatives are to read line by line (forcing me to parse the data and thus losing performance) or adding extra complexity when iterating over the objects so I can keep track when an object is done and I should continue to the next.

Neither is very attractive to me and IMHO since this is a quite common scenario it would make some sense to add it, especially since the parser support relaxed parsing wrt to trailing commas and comments

Code example of workaround:

open System
open System.IO
open System.Text.Json

[<EntryPoint>]
let main argv =
  let options = JsonReaderOptions ()
  let input = File.ReadAllBytes "bigdata.json"

  let bytes = ReadOnlyMemory input

  let mutable objectsSeen = 0
  let mutable tokensSeen = 0

  for i = 1 to 1000 do
    let rec processAllLines offset =
      let bytes = bytes.Span

      // Advance the span to next available object
      let bytes = bytes.Slice offset
      let reader = Utf8JsonReader (bytes, options)

      objectsSeen <- objectsSeen + 1
      let mutable level = 0
      let mutable cont = true

      // Consume a single line that assumed to contain a single object

      // reader is a ref struct which prevents usage of tail-rec
      //  :-(
      //  Not really a bug in Utf8JsonReader but rather in the F#
      //  story around ref structs
      while cont &&  reader.Read () do
        tokensSeen <- tokensSeen + 1
        match reader.TokenType with
        | JsonTokenType.StartArray
        | JsonTokenType.StartObject -> level <- level + 1
        | JsonTokenType.EndArray
        | JsonTokenType.EndObject ->
          level <- level - 1
          if level = 0 then
            cont <- false
        | _ -> ()

      let offset = offset + int reader.BytesConsumed

      // After consuming the line there's still trailing non
      //  printable lines, consume them in order to be able to
      //  tell if we reached the end of input
      let rec consumeEndOfLine o =
        if o < input.Length && input.[o] < 32uy then
          consumeEndOfLine (o + 1)
        else
          o

      let offset = consumeEndOfLine offset

      // Done?
      if offset < input.Length then
        processAllLines offset
    processAllLines 0

  printfn "Objects seen : %d" objectsSeen
  printfn "Tokens seen : %d" tokensSeen

  0

@Int32Overflow

From @Int32Overflow in #36750

The following JSON file (with multi content) is not supported:

{ "name": "Admin" }{ "name": "Publisher" }

The following exception occurs:
System.Text.Json.JsonReaderException: "'{' is invalid after a single JSON value. Expected end of data. LineNumber: 0 |

In the library Newtsoft Json.NET there was the following property "SupportMultipleContent". What is the solution here?

https://www.newtonsoft.com/json/help/html/ReadMultipleContentWithJsonReader.htm

... at the very least, using File.ReadLines(...) could help alleviate a large portion of your memory issues. What might be better for the workaround would be if there was some sort of File.ReadLinesAsBytes (since then you wouldn't be deserializing into string first) - essentially, something returning IEnumerable<ReadOnlySpan> that looks for newlines (or possibly something stranger, if you wanted to buffer the input).

I have a similar issue to this, and my issue may fit into this issue:

I am reading UTF-8 JSON data encapsulated in JavaScript from a web server. I've created a stream that skips to the JSON data (according to a specific preamble the JSON data is prefixed by). After the JsonDocument was read, I get an exception that there is data after the object's last token.
My current workaround is to save the entire web page in memory (about 2 MiB but I'm only interested in a few KiB) and put it into a JSON reader which works fine, e.g.:

var jsonData = Encoding.UTF8.GetBytes("{\"d\":\"test\"}}}}}"); //  A JSON object with additional data

// Wanted way
using var memoryStream = new MemoryStream(jsonData); // In my case this is a file stream or HTTP content stream
var document = await JsonDocument.ParseAsync(memoryStream); // FAILS!!
var jsonElement = document.RootElement;

// The "normal" way
var document = JsonDocument.Parse(jsonData.AsMemory()); // FAILS!!
var jsonElement = document.RootElement;

// Workaround:
var jsonReader = new Utf8JsonReader(jsonData.AsSpan(), isFinalBlock: true, default);
var jsonElement = (JsonElement)JsonSerializer.Deserialize<object>(ref jsonReader);

In the first and second sample, I get the following exception: System.Text.Json.JsonReaderException : '}' is invalid after a single JSON value. Expected end of data. LineNumber: 0 | BytePositionInLine: 12..

If I try to deserialize the object, it works fine, and additional data is ignored. The aforementioned is a considerable performance loss, as I have to read the web page and data I'm not interested in.

For example, a property AllowAdditionalContent in JsonDocumentOptions would be fine and would allow users to use System.Text.Json more flexibly.

I appreciate any help you can provide.
Angelo Breuer

@angelobreuer

@angelobreuer -

using var memoryStream = new MemoryStream(jsonData); // In my case this is a file stream or HTTP content stream`

If your stream is already skipping to the start, why not have it close at the end? Change the stream to look for the end of the JSON section, and returns EOF (Read returns 0) when it detects the end of the json section. If you don't have one, write a wrapping stream that does this.

// The "normal" way
var document = JsonDocument.Parse(jsonData.AsMemory()); // FAILS!!

Something similar is possible here, since memory and sequence objects can take a range, allowing you to extract just that portion.

@Clockwork-Muse

@Clockwork-Muse

If your stream is already skipping to the start, why not have it close at the end? Change the stream to look for the end of the JSON section, and returns EOF (Read returns 0) when it detects the end of the JSON section. If you don't have one, write a wrapping stream that does this.

As a workaround, I've already done this but scanning each "block" of bytes for the specific end sequence where the JSON ends have significant overhead.

I have migrated from Json.NET where the aforementioned was possible without any additional overhead:

using var streamReader = new StreamReader(someHttpResponseStream); // already skipped to start of JSON
using var jsonTextReader = new JsonTextReader(streamReader);
var data = await JObject.LoadAsync(jsonTextReader);

I've also checked other JSON serializers which allow the same (Utf8Json, Json.NET, ServiceStack.Text).

I think SupportMultipleContent-like option would be a useful addition to Utf8JsonReader, but we'd need to write up an API proposal.

Here is an API proposal based on a prototype I've been working on

Thanks, that looks pretty good.

For the JsonDeserializeAsyncEnumerableMode, would we imagine a future where you could opt-in to both RootLevelValue and Array, i.e. it'd yield top-level objects but if they were arrays it would instead enumerate their contents? I don't know how valuable that would be, but it would impact the shape of the enum.

would we imagine a future where you could opt-in to both RootLevelValue and Array, i.e. it'd yield top-level objects but if they were arrays it would instead enumerate their contents?

Probably not, and that's because of the ambiguity arising from the case where the element types themselves serialize as arrays. E.g. it would be unclear if DeserializeAsyncEnumerable<int[]>("""[]""") should be returning an empty or singleton IAE.

Video

AllowTrailingContent became AllowMultipleValues
The property was removed from JsonSerializarOptions (and the sourcegen equivalent) because it creates slight confusion and isn't strictly necessary
The JsonDeserializeAsyncEnumerableMode seemed like it needs to be more complicated than just an enum. In lieu of designing an options type during the meeting, we accepted an overload with a boolean that opted into this new behavior.
- We settled on naming this boolean topLevelValues

namespace System.Text.Json
{
    public partial struct JsonReaderOptions
    {
        public bool AllowMultipleValues { get; set; }
    }

    public partial static class JsonSerializer
    {
        public static IAsyncEnumerable<T> DeserializeAsyncEnumerable<T>(Stream utf8Json, bool topLevelValues, JsonSerializerOptions options = null, CancellationToken cancellationToken = default);
        public static IAsyncEnumerable<T> DeserializeAsyncEnumerable<T>(Stream utf8Json, JsonTypeInfo<T> jsonTypeInfo, bool topLevelValues, CancellationToken cancellationToken = default);
    }
}

Navigation Menu

Repository navigation

Let Utf8JsonReader process input with one complete JSON document per line #33030

20 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Let Utf8JsonReader process input with one complete JSON document per line #33030

Description

Activity

mrange commented on Mar 1, 2020

layomia commented on May 22, 2020

Clockwork-Muse commented on May 22, 2020

angelobreuer commented on Oct 14, 2020

Clockwork-Muse commented on Oct 14, 2020

angelobreuer commented on Oct 14, 2020

eiriktsarpalis commented on Oct 21, 2021

20 remaining items

stephentoub commented on Jul 1, 2024

eiriktsarpalis commented on Jul 1, 2024

bartonjs commented on Jul 2, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions