You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Reconciler with enabled "generationAwareEventProcessing" reconciles unchanged custom resource upon startup #3274
We've encountered the behavior where we're using the default Reconciler settings, which has generationAwareEventProcessing turned on as well as a default maxReconciliationInterval set to 10 hours. Our CR implements the ObservedGenerationAwareStatus interface, so it has the observedGeneration property present in the status and it's handled by the framework.
We're seeing that upon Operator startup a reconciliation is triggered, when there is an already created custom resource in the cluster which has been reconciled already and has metadata.generation equal to status.observedGenertion.
The problem in this scenario is that the Operator is being updated and has updated logic, triggered by a pipeline automation script update. An update to the CR is going to follow in the next pipeline step. However, the triggered reconciliation captures the current CR in the cluster before the updated one can be applied. That in some cases leads to breaking dependent resources spec and subsequent failure (e.g. old incompatible images read from the CR set to an updated Deployment dependent resources spec encoded in the Operator code).
What did you expect to see?
No reconciliation triggered on the CR, which has it's generation and observed generation matching properties.
Environment
Kubernetes cluster type:
Gardener K8s 1.33.5
java-operator-sdk version is 4.4.4
Java version 17
Possible Solution
Not sure if the behavior could be caused by the maxReconciliationInterval being not set after the update or just informer caches being empty.
Thanks for the quick response. Could you elaborate, what check do you mean we could implement?
Could you point out what is the reason for the proactive reconciliation technically - is it the empty caches or you've just added explicitly this logic?
Could you point out what is the reason for the proactive reconciliation technically - is it the empty caches or you've just added explicitly this logic?
In general all the controller frameworks do this, the reason comes from the theory regarding those controllers, usually during the reconciliation controllers tend to react on events, and check all the resources what they manage. So if a controller was down for some time on start it checks if everything still in place.
Could you elaborate, what check do you mean we could implement?
Hello,
Thanks for the explanation. The check for the generations is exactly what I thought. Probably you meant to write the UpdateControl.noUpdate() when the if condition is true.
However, we think we can't simply rely on this check in the following cases:
there is a periodic reconciliation triggered as you know is in place, with default time is 10 hours - if we just compare the generations, they'd match and we won't check reconcile any other dependent resource
if some dependent resource gets changed/deleted, this would trigger a reconciliation, but this won't affect the generation of the CR - with the plain check, we'd again omit reconciling the dependents
one more thing we have implemented on our end - as we have a Deployments as dependent resource, we have a check after we call reconcile on the dependent resource, to check if the Deployment is ready (reflecting it's Pods become ready) and if not, we put a specific state in the state and we reschedule a reconciliation with return UpdateControl.patchStatus(myCR).rescheduleAfter(CHECK_INTERVAL, TimeUnit.SECONDS);. That in turn updates the observedGeneration via the framework logic and in the next reconciliation, the check for equality would simply not be enough.
I was thinking including some additional metadata in the status, like nextReconcileTime or lastReconcileTime, as I've seen the latter for sure in some other projects, but I'm not sure how to implement that.
Yes, those also makes sense, of course that is the simplified view on generations.
Was thinking about this:
An update to the CR is going to follow in the next pipeline step. However, the triggered reconciliation captures the current CR in the cluster before the updated one can be applied.
This might be quite specific on your usecase, but you could add some annotation based conversion logic inside the controller implemnetation, or simply update the custom resources before running the operator.
Activity
csviri commentedon Apr 7, 2026
Hi @ivanchuchulski , this is intentional. It is the default behavior / best practice to reconcile resources on controller startup.
Note that you can implement this on your own check at the start of the reconiliation.
ivanchuchulski commentedon Apr 7, 2026
Hello @csviri
Thanks for the quick response. Could you elaborate, what check do you mean we could implement?
Could you point out what is the reason for the proactive reconciliation technically - is it the empty caches or you've just added explicitly this logic?
Best Regards,
Ivan Chuchulski
csviri commentedon Apr 7, 2026
In general all the controller frameworks do this, the reason comes from the theory regarding those controllers, usually during the reconciliation controllers tend to react on events, and check all the resources what they manage. So if a controller was down for some time on start it checks if everything still in place.
you could do (oversimplified) :
if that fits your use case.
ivanchuchulski commentedon Apr 7, 2026
Hello,
Thanks for the explanation. The check for the generations is exactly what I thought. Probably you meant to write the
UpdateControl.noUpdate()when the if condition is true.However, we think we can't simply rely on this check in the following cases:
return UpdateControl.patchStatus(myCR).rescheduleAfter(CHECK_INTERVAL, TimeUnit.SECONDS);. That in turn updates theobservedGenerationvia the framework logic and in the next reconciliation, the check for equality would simply not be enough.I was thinking including some additional metadata in the status, like
nextReconcileTimeorlastReconcileTime, as I've seen the latter for sure in some other projects, but I'm not sure how to implement that.Best Regards,
Ivan Chuchulski
csviri commentedon Apr 7, 2026
Yes, those also makes sense, of course that is the simplified view on generations.
Was thinking about this:
This might be quite specific on your usecase, but you could add some annotation based conversion logic inside the controller implemnetation, or simply update the custom resources before running the operator.