Repository navigation
Processing Parameters and Asset Roles - #46
fmigneault wants to merge 6 commits into
Conversation
gadomski
left a comment
There was a problem hiding this comment.
Thanks, this makes more sense to me, but there's a few bits that I find confusing ... I might not be the target audience for these changes, though.
| | ------------------ | ------ | ----------- | ------- | | ||
| | `gdal-calc` | string | A [`gdal_calc.py`](https://gdal.org/programs/gdal_calc.html) expression based on numpy syntax. | `A*logical_or(A<=177,A>=185)` | | ||
| | `openeo` | object | An [openEO process](https://openeo.org/documentation/1.0/developers/api/reference.html#section/Processes) | [Example](https://github.com/raw/Open-EO/openeo-processes/1.2.0/normalized_difference.json) | | ||
| | `ogcapi-processes` | object | An [OGC ProcessDescription](https://docs.ogc.org/is/18-062r2/18-062r2.html#ogc_process_description) | [Example](https://schemas.opengis.net/ogcapi/processes/part1/1.0/examples/json/ProcessDescription.json) | |
There was a problem hiding this comment.
This change (adding ogcapi-processes) might be easier to review in its own PR, since it's technically independent of processing parameters.
There was a problem hiding this comment.
I've added it because the document already mentioned "OGC Process API" in the relation types, but didn't provide any corresponding value here.
There was a problem hiding this comment.
I guess my point is, "Add ogciapi-processes to expression formats" could be its own, light pull request that could be easily reviewed and merged?
There was a problem hiding this comment.
I can cherry-pick it if desired.
| > [!NOTE] | ||
| > To ensure portability, `format` may employ a specific profile URI from a registered naming authority | ||
| > that explicitly describes the represented `expression` without ambiguity. | ||
|
|
There was a problem hiding this comment.
I'm not familiar with a "profile URI" — it could be helpful to link out to an explanation to help the reader understand how to use a profile?
There was a problem hiding this comment.
I agree about adding an example for clarity.
Following is an example: https://www.opengis.net/def/profile/OGC/0/ogc-process-description
It can be any URI. Ideally it resolves to a concept/ontology, but it can technically be even just an well-defined string in a standard document. It can also be a JSON schema URI or some other identifiable resource that strictly characterizes the resource.
The idea of using profile URIs is mainly such that what they refer to is more interpretable and removes potential ambiguity in otherwise generic strings.
| | processing-input | An Asset that represents an individual input (e.g., a configuration file required by the processing expression). Note that an input STAC object should instead use the `derived_from` link [relation type](#relation-types). | | ||
| | processing-parameters | An Asset that embeds the entire processing parametrization contents that were submitted. This is similar to [Parameters](#parameters) but is not limited to `application/json` and can contain much more data. | |
There was a problem hiding this comment.
I don't intuitively understand the different between the input and the parameters? In programming terms, aren't they both "function arguments"?
There was a problem hiding this comment.
The difference would be as follows.
Given a process that takes {"a": <some-geotiff>, "b": <some-parquet>, "c": <some-float>}.
You would have multiple processing-input, each respectively pointing at the href of <some-geotiff> and <some-parquet>. The <some-float> can be embedded in an Asset file as text/plain, JSON, or similar if one really feels the need for it.
The processing-parameters instead represent the structure itself, as submitted to the process, i.e.:
{
"a": {"href": <some-geotiff>},
"b": {"href": <some-parquet>},
"c": <some-float>
}This is dependant of the specific encoding the process takes. Ideally it is also indicated as processing:expression to clarify, though not strictly enforced.
The processing-parameters also allows other formats, so if a multipart/related was used for examples instead of the JSON, it could also be provided as a whole.
I purposely used processing-parameters to align with the suggested (as discussed the meeting):
"processing:expression": {
"format": "https://www.opengis.net/def/profile/OGC/0/ogc-process-description",
"expression": { ... the big process description ... },
"parameters": { ... same as above 'processing-parameters' content ... }
}I think it is also a good idea to have distinct inputs/outputs and parameters/results names because they represent differ document contents. From experience with OGC Processes, "inputs" term as been used to refer to different concepts (description vs execution values), and people always get very confused about the structures. Different terms ensure a clear distinction.
There was a problem hiding this comment.
The processing-parameters instead represent the structure itself, as submitted to the process, i.e.:
If I'm understanding (I'm probably not), the asset hrefs are duplicated in processing-input and processing-parameters? If so, this feels redundant.
Different terms ensure a clear distinction.
To me, it's less clear (inputs are inputs, that seems simplest to me). But again, I'm probably not the target audience for this extension, so I won't block.
There was a problem hiding this comment.
Optionally duplicated. It gives the flexibility. I would say it depends how specific the provenance wants to illustrate the concepts.
I can see at least 2 reasons why processing-parameters remains relevant:
- If a process only used a few floating point values, it is overkill to have it generate multiple distinct asset files with just a few bytes of data. Those files might not even exist also.
- If a provenance chain provides the original submission (in the context of how the openEO, OGC API, etc. produced it), it easy to re-execute it as is. Using
processing-inputswould force a client to manually recompose it (maybe incorrectly) from the Assets.
| | processing-output | An Asset that represents an invidivudal output (possibily amongst multiple) that was generated by the processing expression. These Assets should typically include `data` role as well, although not strictly required. | | ||
| | processing-results | An Asset that embeds the entire processing results contents generated by the processing expression (e.g., an _OGC API - Processes_ [Results](https://schemas.opengis.net/ogcapi/processes/part1/1.0/examples/json/Result.json) document. | |
There was a problem hiding this comment.
Similarly, what's the different between an "output" and a "result"? Aren't these both akin to "the return value of the function"?
There was a problem hiding this comment.
Output/Results happens to align with the same OGC concepts of the big "Results" document containing all of the individual "Outputs".
Beside that, same reason as Inputs/Parameters for distinct naming for different concepts.
| | ---------- | --------------- | ----------- | | ||
| | format | string | **REQUIRED** The type of the expression that is specified in the `expression` property (see [Format](#format) examples). | | ||
| | expression | \* | **REQUIRED** An expression compliant with the `format` specified. The expression can be any data type and depends on the `format` given, e.g. string or object. | | ||
| | parameters | Map<string, \*> | Optional parameters relevant for the context of the `format` specified (e.g., _openEO_ or _OGC API - Processes_ execution inputs, a variable name/file mapping for `gdal-calc`, etc. | |
There was a problem hiding this comment.
Wondering whether this should be of type *, like expression. We don't really know how parameters look like in formats. Could be an array with values or a map with parameter name => argument paris, for example.
There was a problem hiding this comment.
Yeah that's a good point. In OGC Application packages I think parameters are an array.
There was a problem hiding this comment.
I'm fine with that. It is up for the "expression format" implementation to define what it allows.
| | ------------------ | ------ | ----------- | ------- | | ||
| | `gdal-calc` | string | A [`gdal_calc.py`](https://gdal.org/programs/gdal_calc.html) expression based on numpy syntax. | `A*logical_or(A<=177,A>=185)` | | ||
| | `openeo` | object | An [openEO process](https://openeo.org/documentation/1.0/developers/api/reference.html#section/Processes) | [Example](https://github.com/raw/Open-EO/openeo-processes/1.2.0/normalized_difference.json) | | ||
| | `ogcapi-processes` | object | An [OGC ProcessDescription](https://docs.ogc.org/is/18-062r2/18-062r2.html#ogc_process_description) | [Example](https://schemas.opengis.net/ogcapi/processes/part1/1.0/examples/json/ProcessDescription.json) | |
There was a problem hiding this comment.
An OGC API Process is only metadata, not the actual "thing" to run, right? At least that's how the example looks like, if I understand it correctly. I think processing:expression should be something you can executy, not just metadata.
There was a problem hiding this comment.
Depends how you look at it. It is an abstraction just like gdal-calc is an abstraction to some very specific C++ functions invoked. You don't actually need to know the internals of the process.
The example would be something like (YAML and single input for simplicity):
processing:expresssion:
format: ogcapi-processes
expression:
process: https://example.com/ogcapi/processes/Echo
inputs:
message:
schema:
type: string
outputs:
result:
schema:
type: string
jobControlOptions: [execute-sync]
parameters:
inputs:
message: "Hello, World"| _ProcessDecription_ or _Process Graph_ in `expression`, and along it, the `"parameters": {"threshold": 0.5}` inputs | ||
| it employed for its execution. | ||
|
|
||
| ## Asset Roles |
There was a problem hiding this comment.
Should these be Link Relation Types instead? Or why are these assets?
How's the relation to https://github.com/stac-extensions/application#relation-types ? Is there potential for alignment?
There was a problem hiding this comment.
Yeah it doesn't seem obvious to me why processing-expression and processing-execution are link relation types and these ones are assets.
There was a problem hiding this comment.
I started with Assets because I initially only had the processing-output and processing-results, and those would potentially be the actual results for which the STAC definition exists (the whole point of processing in this case is to provide their provenance). So, the idea is that we simply tag the role on top of the Asset already there just to make it clear it is part of the provenance (in contrast for example to some thumbnail or other alternate Asset that could be supplied from elsewhere).
I agree for the processing-input and processing-parameters, it makes more sense to have them as links since they are not the results.
Is there a strong distinction between Asset/Links for those cases? It seems to be walking the fine line between what should be an Asset or a Link.
jsignell
left a comment
There was a problem hiding this comment.
I like the concept. I definitely see the gap in the current extension. It just seems like there might be a few too many ways to say the same thing. Do you already have a need for all these different fields? Otherwise might it make sense to add less in this PR and see if a need becomes clear?
I'd say most often than not, I could "manage" with |
Note
As discussed at the STAC Community Meeting 2026-10-19, and as replacement of:
processing:expressionwithformat: results#44Changes
parametersfield underprocessing:expressionto provide a mapping of named input values employed.processing-input,processing-output,processing-parametersandprocessing-results.profilein Asset using the above roles and examples for common applications.