Repository navigation
Add durable activity retry metadata and telemetry - #1435
Draft
Varshitha Bachu (bachuv) wants to merge 1 commit into
Draft
Varshitha Bachu (bachuv) wants to merge 1 commit into
Varshitha Bachu (bachuv) wants to merge 1 commit into
Conversation
Carry validated replay-stable retry metadata through scheduled actions, history, and activity execution. Project retry events at orchestration completion with original timestamps and correlated allowlisted Application Insights telemetry. Add serialization, replay, dispatcher, and export regressions and document the existing generation boundary.
Comment on lines
+300
to
+311
| foreach (ActivityEvent existing in activity.Events) | ||
| { | ||
| if (existing.Name == Schema.Task.RetryScheduled) | ||
| { | ||
| var tags = existing.Tags.ToDictionary(tag => tag.Key, tag => tag.Value); | ||
| if (tags.TryGetValue(Schema.Task.TaskId, out object? taskId) && taskId is int id | ||
| && tags.TryGetValue(Schema.Task.RetryGroupId, out object? groupId) && groupId is string group) | ||
| { | ||
| emitted.Add((id, group)); | ||
| } | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Active issues can replace activity failures with validation errors when retry limits change and export a synthetic deserialization error instead of the documented failure-type fallback during replay.
2 open issues
What changed in this PR
Adds durable activity retry metadata across scheduling, history, activity execution, and telemetry while keeping retry ownership in RetryInterceptor.
Changes:
- Carries validated attempt numbers, retry limits, and replay-stable group identifiers independently of transport redelivery.
- Adds activity span attributes and completion-time retry events exported to Application Insights.
- Adds serialization, replay, dispatcher, and telemetry tests, plus documentation and optional prerelease packaging support.
| File | Description |
|---|---|
| test/DurableTask.Core.Tests/RetryMetadataTests.cs | Tests validation, serialization, replay, and retry isolation. |
| test/DurableTask.Core.Tests/DurableTelemetryModuleTests.cs | Tests event export, correlation, timestamps, and allowed fields. |
| test/DurableTask.Core.Tests/DurableTask.Core.Tests.csproj | References Application Insights for export tests. |
| test/DurableTask.Core.Tests/DispatcherMiddlewareTests.cs | Tests metadata propagation and completion/termination telemetry. |
| test/DurableTask.AzureStorage.Tests/TableEntityConverterTests.cs | Tests history-table metadata serialization and validation. |
| test/DurableTask.AzureStorage.Tests/DataContractJsonConverterTests.cs | Tests queue metadata preservation and validation. |
| src/DurableTask.Core/Tracing/TraceHelper.cs | Adds retry tags and history-based event projection. |
| src/DurableTask.Core/Tracing/Schema.cs | Defines retry telemetry attribute and event names. |
| src/DurableTask.Core/TaskOrchestrationDispatcher.cs | Propagates metadata and projects events before span completion. |
| src/DurableTask.Core/TaskOrchestrationContext.cs | Attaches scheduling metadata and captures replay failure types. |
| src/DurableTask.Core/TaskContext.cs | Exposes durable retry metadata to activities. |
| src/DurableTask.Core/TaskActivityDispatcher.cs | Copies scheduled metadata into activity context. |
| src/DurableTask.Core/RetryMetadata.cs | Defines the validated retry metadata contract. |
| src/DurableTask.Core/RetryInterceptor.cs | Tracks retry attempt context around each invocation. |
| src/DurableTask.Core/OrchestrationContext.cs | Holds asynchronous-flow-local retry context. |
| src/DurableTask.Core/History/TaskScheduledEvent.cs | Stores optional durable retry metadata in history. |
| src/DurableTask.Core/Command/ScheduleTaskOrchestratorAction.cs | Carries retry metadata in scheduling actions. |
| src/DurableTask.ApplicationInsights/DurableTelemetryModule.cs | Exports correlated retry events with allowlisted attributes. |
| src/DurableTask.ApplicationInsights/DurableTask.ApplicationInsights.csproj | Supports optional package version suffixes. |
| docs/telemetry/traces/semantic-conventions.md | Documents retry attributes, projection, and retention limits. |
| docs/telemetry/application-insights.md | Documents retry custom-event export behavior. |
| docs/features/retries.md | Documents activity retry metadata and compatibility. |
Review effort: Balanced
Comment on lines
+62
to
+63
| retryContext.AttemptNumber = retryCount + 1; | ||
| retryContext.PreviousFailure = lastException; |
|
|
||
| internal static string? GetTaskFailureType(Exception failure) => | ||
| (failure as OrchestrationException)?.FailureDetails?.ErrorType | ||
| ?? failure.InnerException?.GetType().FullName |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Activity handlers and telemetry need durable retry-attempt information that is distinct from transport redelivery. This adds one-based attempt metadata, including the configured maximum and a replay-stable logical activity identifier, without moving retry ownership out of
RetryInterceptor.Approach
RetryMetadataand carry it through scheduling actions,TaskScheduledEvent, and activityTaskContext. Preserve optional metadata through Azure Storage queue/table serialization; no-policy calls and older histories remain compatible.Compatibility and tradeoffs
Retry events become visible only when the orchestration span completes; replay does not emit duplicates or create additional orchestration spans. Existing
ContinueAsNewhistory resets and span lifecycles are unchanged, so inherited-trace continuation does not preserve prior-generation orchestration retry events for later reconstruction. Activity metadata and activity spans are independent of this retention boundary.Failure types use structured failure details or sanitized types captured during normal replay. When the original legacy type is unavailable, telemetry reports
TaskFailedExceptionrather than guessing or parsing messages/stacks. Retry events do not include inputs, error messages, or stack traces. No new persisted telemetry state is introduced.Validation
net8.0.net48.net48fresh-trace tests also fail on untouched HEAD; verified separately and left unchanged.Generated local NuGet artifacts are not included in this PR.