Use databricks container service for CI/CD #74

Hi,
we are thinking about using container services and generate the docker images as the final artifacts, the same it is usually done in kubernetes or in any modern cloud facility supporting docker.
For that, we create a dockerfile that copies all of our generated jars, including dependencies, into /databricks/jars and publish a docker image in a container registry.
For example: our CI pipeline generates a software version 1.2.0-development.aabbbccdd that gets deployed as the docker image with that version. All of our libraries and dependencies are in /databricks/jars.

If we create a cluster and set the docker container image of that specific version, we can run a scala notebook which has access to our classes, so this is great in terms of CI/CD.

However, we have a "problem". We are launching jobs from Azure Data Factory which in turn, create DATABRICKS JOBS that will run in our cluster. The problem is that it seems mandatory to always add an "additional library". We understand that this requirement comes from the fact that this is emulating and spark-submit and we have to pass the jar with the main class. However, in our containers, we already have the jars inside and are in the classpath.
It seems we can add any file (with any extension) as an additional library, as long as it exists in dbfs. So, we pass a dummy jar, and everything works.

We are worried about defining our CI/CD with something that may break in the future but we totally dislike the way databricks handles libraries and would prefer the docker approach.

So we have a few questions:

• Have you had this problem before and it is something that it is being requested by customers?
• is there any alternative?
• do you suggest to not use containers this way?

best regards