You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Gradle build cache miss due to volatility of SpotlessTask.steps input #2168
Thanks for this report! We have gone to great lengths to make our step's equality correct, but 7.0.0.BETA1 is a huge change so I'm not shocked if we got one wrong.
Our current implementation puts all of each task's implementation details into account. Settings files, flags, etc. If you change a formatter's setting file, you want the formatter to rerun, you don't want to clear cache before you can find out what the new flag does.
I see this problem is present in spotlessJava, is it also present in spotlessKotlin? If not, then probably the issue is only in the importOrderFile or eclipse step, since they are unique to spotlessJava. If we comment out both steps, does the problem go away? Then add them one at a time until it comes back? If we can narrow down which step has the problem, then we can fix that step.
This is specific to the EclipseJdtFormatterStep (backed by EquoBasedStepBuilder).
Serializing EquoBasedStepBuilder has some references on a settings file with an absolute path.
This can be fixed by configuring the collection transient and adding a collection that would represent this input without volatility (Collection of file names?).
Is that there is a RoundtripState which can have absolute paths in it. And then there is an EqualityState which is extracted from the RoundtripState, and that should not have absolute paths in it.
The state which is used to calculate .equals and .hashCode comes from here
This can be fixed by configuring the collection transient and adding a collection that would represent this input without volatility (Collection of file names?).
This is what FileSignature and FileSignature.Promised do. FileSignature.Promised has the absolute path, and thus supports roundtrip serialization, and thus can be in the RoundtripState. For the EqualityState, it gets resolved into a regular FileSignature which loses the absolute paths.
I thought that the convoluted design above would let us satisfy the convoluted requirements the Gradle's configuration cache forces on us:
relocatable build cache -> we have to ignore absolute paths (EqualityState)
configuration cache -> we have to support roundtrip serialization -> we have to store absolute paths (RoundtripState)
One of two things is going on:
there is an absolute path in the Equo-based steps that needs to become FileSignature/Promise ified that I missed
Gradle is doing fingerprinting based on the serialized representation rather than based on hashcode (the hack that we used before configuration cache forced us to change)
Serializing EquoBasedStepBuilder has some references on a settings file with an absolute path
Any chance you can provide the filename in question?
I can confirm that the steps input property is fingerprinted with the Java serialization result.
You can see in this screenshot the absolute path to the settings file (in addition to my proposal of adding a collection containing only the filenames)
It is configured here in the consumer project.
Do you think it could work with a relative path being passed?
Unrelated to this issue, while investigating the problem I discovered that EndWithNewLineStep was not implementing Serializable although it should according to the method signature.
Re: EndWithNewlineStep, the code is correct. EndWithNewlineStep is not serializable, but EndWithNewlineStep.classis serializable. Because the step has no settings, it just needs a "key" for equality comparison.
Do you think it could work with a relative path being passed?
A “value property” is a task input property that is not a file or set of files. ... Build scans capture a hash of the value for comparison, not the value itself
So now the question is: is Gradle doing the exact same hack we did before this gigantic 7.0 epic? As in, they are using the serialized representation to do .equals and .hashcode, and ignoring the actual .equals and .hashcode? That's reasonable thing to do, but doing that and requiring roundtrip serialization of all configuration values is insane... I hope that's not what's going on...
Yeah, looks like there's a fundamental conflict here. Gradle uses the serialized representation of your task to do build cache, it does not respect .equals. But configuration cache requires your task to support full roundtrip serialization. As argued in #987, the roundtrip serialization has a lot of downsides, and the only upside (cross-machine configuration cache) is imo never going to happen, and if it does it's going to smear secret keys even further across the internet than it already has.
As it stands in spotless 7.x, local build cache and local configuration cache both work, but shared build cache is broken. In 6.x, everything worked except that the local configuration cache barfed if you started a new daemon.
The issue is reproducible with release version of the 7.0.0 plugin and gradle 8.9:
Execution failed for task ':spotlessJava'.
> If the initializer was null, then one of roundtripStateInternal or equalityStateInternal should be non-null, and neither was
Activity
nedtwigg commentedon Jun 12, 2024
Thanks for this report! We have gone to great lengths to make our step's equality correct, but
7.0.0.BETA1is a huge change so I'm not shocked if we got one wrong.Our current implementation puts all of each task's implementation details into account. Settings files, flags, etc. If you change a formatter's setting file, you want the formatter to rerun, you don't want to clear cache before you can find out what the new flag does.
I see this problem is present in
spotlessJava, is it also present inspotlessKotlin? If not, then probably the issue is only in theimportOrderFileoreclipsestep, since they are unique tospotlessJava. If we comment out both steps, does the problem go away? Then add them one at a time until it comes back? If we can narrow down which step has the problem, then we can fix that step.jprinet commentedon Jun 25, 2024
Sorry for the late reply @nedtwigg
I could eventually identify the root cause after investigating further the issue.
The cache key differs when running the build from different locations due to a change in the underlying fingerprint of FormatterStepSerializationRoundtrip.roundtripStateInternal
This is specific to the
EclipseJdtFormatterStep(backed by EquoBasedStepBuilder).Serializing EquoBasedStepBuilder has some references on a settings file with an absolute path.
This can be fixed by configuring the collection transient and adding a collection that would represent this input without volatility (Collection of file names?).
nedtwigg commentedon Jun 25, 2024
Huh. So the point of this:
spotless/lib/src/main/java/com/diffplug/spotless/FormatterStepSerializationRoundtrip.java
Lines 27 to 29 in 059523c
Is that there is a
RoundtripStatewhich can have absolute paths in it. And then there is anEqualityStatewhich is extracted from theRoundtripState, and that should not have absolute paths in it.The state which is used to calculate
.equalsand.hashCodecomes from herespotless/lib/src/main/java/com/diffplug/spotless/FormatterStepSerializationRoundtrip.java
Lines 44 to 50 in 059523c
This is what
FileSignatureandFileSignature.Promiseddo.FileSignature.Promisedhas the absolute path, and thus supports roundtrip serialization, and thus can be in theRoundtripState. For theEqualityState, it gets resolved into a regularFileSignaturewhich loses the absolute paths.I thought that the convoluted design above would let us satisfy the convoluted requirements the Gradle's configuration cache forces on us:
One of two things is going on:
FileSignature/Promiseified that I missedAny chance you can provide the filename in question?
jprinet commentedon Jun 26, 2024
I can confirm that the
stepsinput property is fingerprinted with the Java serialization result.You can see in this screenshot the absolute path to the settings file (in addition to my proposal of adding a collection containing only the filenames)

It is configured here in the consumer project.
Do you think it could work with a relative path being passed?
Unrelated to this issue, while investigating the problem I discovered that
EndWithNewLineStepwas not implementing Serializable although it should according to the method signature.jprinet commentedon Jul 2, 2024
any chance you could consider this option @nedtwigg ?
If not, will you implement a fix?
nedtwigg commentedon Jul 3, 2024
Re:
EndWithNewlineStep, the code is correct.EndWithNewlineStepis not serializable, butEndWithNewlineStep.classis serializable. Because the step has no settings, it just needs a "key" for equality comparison.I just added a test to show a few things:
.equalsor.hashCodespotless/lib-extra/src/test/java/com/diffplug/spotless/extra/java/EclipseJdtEqualityTest.java
Lines 35 to 47 in 53a2c48
From the docs for Gradle build cache:
So now the question is: is Gradle doing the exact same hack we did before this gigantic 7.0 epic? As in, they are using the serialized representation to do
.equalsand.hashcode, and ignoring the actual.equalsand.hashcode? That's reasonable thing to do, but doing that and requiring roundtrip serialization of all configuration values is insane... I hope that's not what's going on...nedtwigg commentedon Jul 3, 2024
Yeah, looks like there's a fundamental conflict here. Gradle uses the serialized representation of your task to do build cache, it does not respect
.equals. But configuration cache requires your task to support full roundtrip serialization. As argued in #987, the roundtrip serialization has a lot of downsides, and the only upside (cross-machine configuration cache) is imo never going to happen, and if it does it's going to smear secret keys even further across the internet than it already has.As it stands in spotless 7.x, local build cache and local configuration cache both work, but shared build cache is broken. In 6.x, everything worked except that the local configuration cache barfed if you started a new daemon.
nedtwigg commentedon Oct 15, 2024
plugin-gradle 7.0.0.BETA3by Fix remote build cache #2298. Can you confirm @jprinet ?jprinet commentedon Oct 18, 2024
Great job @nedtwigg, the 7.0.0.BETA3 indeed fixes the issue
gendolf3d commentedon Jan 7, 2025
The issue is reproducible with release version of the 7.0.0 plugin and gradle 8.9:
nedtwigg commentedon Jan 7, 2025
@gendolf3d your issue seems like something new, feel free to open an issue if you have a reproducer