镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Reconciler with enabled "generationAwareEventProcessing" reconciles unchanged custom resource upon startup #3274

Open

Description

@ivanchuchulski

Bug Report

What did you do?

Hello,

We've encountered the behavior where we're using the default Reconciler settings, which has generationAwareEventProcessing turned on as well as a default maxReconciliationInterval set to 10 hours. Our CR implements the ObservedGenerationAwareStatus interface, so it has the observedGeneration property present in the status and it's handled by the framework.

We're seeing that upon Operator startup a reconciliation is triggered, when there is an already created custom resource in the cluster which has been reconciled already and has metadata.generation equal to status.observedGenertion.

The problem in this scenario is that the Operator is being updated and has updated logic, triggered by a pipeline automation script update. An update to the CR is going to follow in the next pipeline step. However, the triggered reconciliation captures the current CR in the cluster before the updated one can be applied. That in some cases leads to breaking dependent resources spec and subsequent failure (e.g. old incompatible images read from the CR set to an updated Deployment dependent resources spec encoded in the Operator code).

What did you expect to see?

No reconciliation triggered on the CR, which has it's generation and observed generation matching properties.

Environment

Kubernetes cluster type:

Gardener K8s 1.33.5

java-operator-sdk version is 4.4.4

Java version 17

Possible Solution

Not sure if the behavior could be caused by the maxReconciliationInterval being not set after the update or just informer caches being empty.

Activity

  1. csviri commented on Apr 7, 2026

    @csviri
    Collaborator

    Hi @ivanchuchulski , this is intentional. It is the default behavior / best practice to reconcile resources on controller startup.

    Note that you can implement this on your own check at the start of the reconiliation.

  2. ivanchuchulski commented on Apr 7, 2026

    @ivanchuchulski
    Author

    Hello @csviri

    Thanks for the quick response. Could you elaborate, what check do you mean we could implement?
    Could you point out what is the reason for the proactive reconciliation technically - is it the empty caches or you've just added explicitly this logic?

    Best Regards,
    Ivan Chuchulski

  3. csviri commented on Apr 7, 2026

    @csviri
    Collaborator

    Could you point out what is the reason for the proactive reconciliation technically - is it the empty caches or you've just added explicitly this logic?

    In general all the controller frameworks do this, the reason comes from the theory regarding those controllers, usually during the reconciliation controllers tend to react on events, and check all the resources what they manage. So if a controller was down for some time on start it checks if everything still in place.

    Could you elaborate, what check do you mean we could implement?

    you could do (oversimplified) :

     @Override
      public UpdateControl<MyCustomResource> reconcile(
          MyCustomResource resource,
          Context<MyCustomResource> context) {
        if (resource.getMetadata().getGeneration() == resource.getStatus().getObservedGeneration()) {
             return;
        }    
        // omitted code
        return UpdateControl.noUpdate();
      }

    if that fits your use case.

  4. ivanchuchulski commented on Apr 7, 2026

    @ivanchuchulski
    Author

    Hello,
    Thanks for the explanation. The check for the generations is exactly what I thought. Probably you meant to write the UpdateControl.noUpdate() when the if condition is true.

    However, we think we can't simply rely on this check in the following cases:

    • there is a periodic reconciliation triggered as you know is in place, with default time is 10 hours - if we just compare the generations, they'd match and we won't check reconcile any other dependent resource
    • if some dependent resource gets changed/deleted, this would trigger a reconciliation, but this won't affect the generation of the CR - with the plain check, we'd again omit reconciling the dependents
    • one more thing we have implemented on our end - as we have a Deployments as dependent resource, we have a check after we call reconcile on the dependent resource, to check if the Deployment is ready (reflecting it's Pods become ready) and if not, we put a specific state in the state and we reschedule a reconciliation with return UpdateControl.patchStatus(myCR).rescheduleAfter(CHECK_INTERVAL, TimeUnit.SECONDS);. That in turn updates the observedGeneration via the framework logic and in the next reconciliation, the check for equality would simply not be enough.

    I was thinking including some additional metadata in the status, like nextReconcileTime or lastReconcileTime, as I've seen the latter for sure in some other projects, but I'm not sure how to implement that.

    Best Regards,
    Ivan Chuchulski

  5. csviri commented on Apr 7, 2026

    @csviri
    Collaborator

    Yes, those also makes sense, of course that is the simplified view on generations.

    Was thinking about this:

    An update to the CR is going to follow in the next pipeline step. However, the triggered reconciliation captures the current CR in the cluster before the updated one can be applied.

    This might be quite specific on your usecase, but you could add some annotation based conversion logic inside the controller implemnetation, or simply update the custom resources before running the operator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions