Avoid serializing inline Azure Storage queue messages twice

What is the issue

MessageManager.SerializeMessageDataAsync serializes every MessageData instance into rawContent, uses that string to calculate its UTF-8 size and select inline versus blob storage, and then serializes the same object a second time in the common InlineJson branch:

durabletask/src/DurableTask.AzureStorage/MessageManager.cs

Lines 103 to 123 in 5217032

    
           public async Task<string> SerializeMessageDataAsync(MessageData messageData, CancellationToken cancellationToken = default) 
        
           { 
        
               string rawContent = Utils.SerializeToJson(serializer, messageData); 
        
               messageData.TotalMessageSizeBytes = Encoding.UTF8.GetByteCount(rawContent); 
        
               MessageFormatFlags messageFormat = this.GetMessageFormatFlags(messageData); 
        
               if (messageFormat != MessageFormatFlags.InlineJson) 
        
               { 
        
                   // Get Compressed bytes and upload the full message to blob storage. 
        
                   byte[] messageBytes = Encoding.UTF8.GetBytes(rawContent); 
        
                   string blobName = this.GetNewLargeMessageBlobName(messageData); 
        
                   messageData.CompressedBlobName = blobName; 
        
                   await this.CompressAndUploadAsBytesAsync(messageBytes, blobName, cancellationToken); 
        
                   // Create a "wrapper" message which has the blob name but not a task message. 
        
                   var wrapperMessageData = new MessageData { CompressedBlobName = blobName }; 
        
                   return Utils.SerializeToJson(serializer, wrapperMessageData); 
        
               } 
        
               return Utils.SerializeToJson(serializer, messageData); 
        
           }

The only mutation between those two serializations is TotalMessageSizeBytes. That property is internal and is not a [DataMember], while MessageData is a [DataContract] whose wire properties are explicitly marked:

durabletask/src/DurableTask.AzureStorage/MessageData.cs

Lines 24 to 27 in 5217032

    
           /// Protocol class for all Azure Queue messages. 
        
           /// </summary> 
        
           [DataContract] 
        
           public class MessageData

durabletask/src/DurableTask.AzureStorage/MessageData.cs

Lines 94 to 108 in 5217032

    
           /// <summary> 
        
           /// TraceContext for correlation. 
        
           /// </summary> 
        
           [DataMember] 
        
           public string SerializableTraceContext { get; set; } 
        
           internal string Id => this.OriginalQueueMessage?.MessageId; 
        
           internal string QueueName { get; set; } 
        
           internal QueueMessage OriginalQueueMessage { get; set; } 
        
           internal long TotalMessageSizeBytes { get; set; } 
        
           internal MessageFormatFlags MessageFormat { get; set; }

Each serialization also creates a fresh StringBuilder, StringWriter, and result string:

durabletask/src/DurableTask.AzureStorage/Utils.cs

Lines 168 to 184 in 5217032

    
           /// <summary> 
        
           /// Serialize some object payload to a JSON-string representation. 
        
           /// This utility is resilient to end-user changes in the DefaultSettings of Newtonsoft. 
        
           /// </summary> 
        
           /// <param name="serializer">The serializer to use.</param> 
        
           /// <param name="payload">The object to serialize.</param> 
        
           /// <returns>The JSON-string representation of the payload</returns> 
        
           public static string SerializeToJson(JsonSerializer serializer, object payload) 
        
           { 
        
               StringBuilder stringBuilder = new StringBuilder(); 
        
               using (var stringWriter = new StringWriter(stringBuilder)) 
        
               { 
        
                   serializer.Serialize(stringWriter, payload); 
        
               } 
        
               var jsonStr = stringBuilder.ToString(); 
        
               return jsonStr; 
        
           }

This method is called for every outbound Azure Storage task-hub queue message:

durabletask/src/DurableTask.AzureStorage/Messaging/TaskHubQueue.cs

Lines 91 to 110 in 5217032

    
           async Task<MessageData> AddMessageAsync(TaskMessage taskMessage, OrchestrationInstance sourceInstance, SessionBase? session) 
        
           { 
        
               MessageData data; 
        
               try 
        
               { 
        
                   // We transfer to a new trace activity ID every time a new outbound queue message is created. 
        
                   Guid outboundTraceActivityId = Guid.NewGuid(); 
        
                   data = new MessageData( 
        
                       taskMessage, 
        
                       outboundTraceActivityId, 
        
                       this.storageQueue.Name, 
        
                       session?.GetCurrentEpisode(), 
        
                       sourceInstance); 
        
                   data.SequenceNumber = Interlocked.Increment(ref messageSequenceNumber); 
        
                   // Inject Correlation TraceContext on a queue. 
        
                   CorrelationTraceClient.Propagate( 
        
                       () => { data.SerializableTraceContext = GetSerializableTraceContext(taskMessage); }); 
        
                   string rawContent = await this.messageManager.SerializeMessageDataAsync(data);

The blob-offload branch legitimately serializes a small wrapper after uploading rawContent; only the inline branch repeats the original serialization.

Performance impact

Inline messages are the normal path for payloads below the 45 KiB threshold. Every such orchestration, activity, timer, sub-orchestration, and external-event message currently incurs two complete Newtonsoft JSON traversals and two sets of temporary buffers/strings instead of one.

The duplicate CPU and allocation cost scales linearly with message throughput and payload size. At high task-hub throughput this increases serialization CPU, memory bandwidth, Gen-0 pressure, and queue-send latency without changing the resulting wire payload.

Proposed backward-compatible solution

Return the already-produced rawContent in the InlineJson branch:

if (messageFormat != MessageFormatFlags.InlineJson)
{
    // Existing blob upload and wrapper serialization remain unchanged.
}

return rawContent;

This preserves the exact JSON that was already used for the byte-count/format decision. It changes no public API, queue schema, serializer settings, or blob behavior.

Validation

Add regression coverage for both UseDataContractSerialization modes and representative custom type-binder settings, asserting that the optimized inline result exactly matches the current second serialization.
Cover payloads immediately below and above the inline threshold to ensure the blob-wrapper path is unchanged.
Benchmark representative small, medium, and near-threshold messages with allocation diagnostics; the inline path should perform one MessageData serialization instead of two.

Navigation Menu

Repository navigation

Avoid serializing inline Azure Storage queue messages twice #1378

What is the issue

Performance impact

Proposed backward-compatible solution

Validation

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	public async Task<string> SerializeMessageDataAsync(MessageData messageData, CancellationToken cancellationToken = default)
	{
	string rawContent = Utils.SerializeToJson(serializer, messageData);
	messageData.TotalMessageSizeBytes = Encoding.UTF8.GetByteCount(rawContent);
	MessageFormatFlags messageFormat = this.GetMessageFormatFlags(messageData);

	if (messageFormat != MessageFormatFlags.InlineJson)
	{
	// Get Compressed bytes and upload the full message to blob storage.
	byte[] messageBytes = Encoding.UTF8.GetBytes(rawContent);
	string blobName = this.GetNewLargeMessageBlobName(messageData);
	messageData.CompressedBlobName = blobName;
	await this.CompressAndUploadAsBytesAsync(messageBytes, blobName, cancellationToken);

	// Create a "wrapper" message which has the blob name but not a task message.
	var wrapperMessageData = new MessageData { CompressedBlobName = blobName };
	return Utils.SerializeToJson(serializer, wrapperMessageData);
	}

	return Utils.SerializeToJson(serializer, messageData);
	}

	/// Protocol class for all Azure Queue messages.
	/// </summary>
	[DataContract]
	public class MessageData

	/// <summary>
	/// TraceContext for correlation.
	/// </summary>
	[DataMember]
	public string SerializableTraceContext { get; set; }

	internal string Id => this.OriginalQueueMessage?.MessageId;

	internal string QueueName { get; set; }

	internal QueueMessage OriginalQueueMessage { get; set; }

	internal long TotalMessageSizeBytes { get; set; }

	internal MessageFormatFlags MessageFormat { get; set; }

	/// <summary>
	/// Serialize some object payload to a JSON-string representation.
	/// This utility is resilient to end-user changes in the DefaultSettings of Newtonsoft.
	/// </summary>
	/// <param name="serializer">The serializer to use.</param>
	/// <param name="payload">The object to serialize.</param>
	/// <returns>The JSON-string representation of the payload</returns>
	public static string SerializeToJson(JsonSerializer serializer, object payload)
	{
	StringBuilder stringBuilder = new StringBuilder();
	using (var stringWriter = new StringWriter(stringBuilder))
	{
	serializer.Serialize(stringWriter, payload);
	}
	var jsonStr = stringBuilder.ToString();
	return jsonStr;
	}

	async Task<MessageData> AddMessageAsync(TaskMessage taskMessage, OrchestrationInstance sourceInstance, SessionBase? session)
	{
	MessageData data;
	try
	{
	// We transfer to a new trace activity ID every time a new outbound queue message is created.
	Guid outboundTraceActivityId = Guid.NewGuid();
	data = new MessageData(
	taskMessage,
	outboundTraceActivityId,
	this.storageQueue.Name,
	session?.GetCurrentEpisode(),
	sourceInstance);
	data.SequenceNumber = Interlocked.Increment(ref messageSequenceNumber);

	// Inject Correlation TraceContext on a queue.
	CorrelationTraceClient.Propagate(
	() => { data.SerializableTraceContext = GetSerializableTraceContext(taskMessage); });

	string rawContent = await this.messageManager.SerializeMessageDataAsync(data);

Avoid serializing inline Azure Storage queue messages twice #1378

Description

What is the issue

Performance impact

Proposed backward-compatible solution

Validation

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions