Repository navigation
Initial work to switch to system installed tools - #109
Closed
Brian Kroth (bpkroth) wants to merge 7 commits into
Closed
Brian Kroth (bpkroth) wants to merge 7 commits into
Brian Kroth (bpkroth) wants to merge 7 commits into
Conversation
Brian Kroth (bpkroth)
requested review from
Andreas Mueller (amueller) and
Greg Lapinski (grlap)
September 26, 2020 21:06
| set +x | ||
| else | ||
| echo '/usr/bin/cmake is already available. Use "apt-get update && apt-get upgrade" to update it.' | ||
| fi |
Contributor
Author
There was a problem hiding this comment.
Should add notes about these scripts to the setup docs if we go the route of ditching the "fetch local" method.
This was referenced Oct 3, 2020
Contributor
Author
|
Pulled this into #128. |
Brian Kroth (bpkroth)
added a commit
that referenced
this pull request
Oct 13, 2020
Lots of build/pipeline cleanup with the goal of making and publishing standard Docker images the for Linux developers and the CI pipelines (#36). - Most implementation notes are in `.github/workflows/main.yml`, but here's a brief summary: - Nightly we build fresh docker images for Ubuntu {16.04, 18.04, 20.04}. They are only published if all of the usual build/test jobs pass (using the same logic as for PRs). That also helps prime the local caches for the following: - On each PR, we 1. (re)build the image from a local buildx cache that's keyed on the sha of the PR, and optionally will also pull from the latest "main" build (since it presumably worked). That is initially only stored in a local cache. Part of the goal here is to avoid the need to push unvetted docker images (e.g. tagged by sha) to the public registry and then have to clean them up later (which is not always even possible). 2. For each build/test job, we again "rebuild" the image, using that cache from step 1, so that it goes quickly. Then we load the image into the local runner's docker daemon and start an instance to run our tests. This reload is necessary because individual jobs are assigned to new runners so we can't reuse state between jobs. Jobs are organized into dependency graphs to account for this cache reuse. At the moment, all of the tests run on all versions of Ubuntu, which makes the CI pipeline a bit slow. - DONE: We could consider only running 16.04 and 20.04 in the CI pipelines and then running 18.04 in the nightly scheduled run or else just drop support for it entirely. 3. Once all of the tests pass, and the PR is merged, we execute a final job to update the published version of the job by retagging it and pushing it to various registries. The current registries are all Github services (for convenience to get started): - docker.pkg.github.com: this is nice because it supports the built in `GITHUB_TOKEN` secret so doesn't require any additional work for a forked repo to use for testing (they push to their own path though). However, it doesn't support anonymous pulls and is being deprecated by ... - ghcr.io: this the newly announced Github Container Registry service. It requires a special set of secrets to be assigned to the repo, so needs some additional work to make use of. Also, as of now, the Microsoft org doesn't seem to have it enabled, so for now I've set it to push to the Microsoft-CISL org. Additional work: - Reorg the Dockerfile to use multi-stage builds to allow skipping copying the source code into it so that we only distribute the build tools. We can also optionally build slimmer "python-only" images this way. In a future PR we'll take on the work to switch from straight pip to conda (#83) - TODO: Could optionally prime the `/src/MLOS` folder in the image so that one could `git pull` immediately to get the latest code if they wanted to without a local copy as a bind mount. - Added a `make docker-image` target to simplify repeated local docker image build invocations. - Switch to using tools (cmake, dotnet) installed in the system rather than fetched during build (#109) - These scripts are also nice reusable elements for users to install in their own environments if they wish. - DONE: Update the documentation to reference them instead of relying on the old "fetch" method. - DONE: There is work in ADO that provides a similar `install.python.sh` script to setup the `python3.7` install. Will port that over later as well. - Cleanup _most_ of the build system to match (I have not yet removed the old setup scripts, but they are no longer automatically called). - Added vscode integration support for developing inside the container (#105, #2) - Added support for running as a non-root user so that bind mounts of the repo source tree work without re-chowning files on the host system as root. This required a fair bit of fixups, but I think should cut down on some issues with students/others and makes the vscode integration easier as well. - Add nightly run scheduling of Python long haul tests (#113) - TODO: Fixup docker related documentation (#87)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is just a draft for now.
Initial fixups following efforts for #108.
Originally, we had scripts to fetch some tools into the local repo.
This has some benefits wrt to lowering the environment setup cost for users. However, then the
PATHenvironment needs to be setup to point at those. In build scripts this is easy, but for local debugging it's less awesome.This initially just adds support for just using the system installed tools if they're new enough.
It does not yet, entirely remove the old path. The main reason for this is the current CI pipelines still depend upon them.
As we transition to Docker for everything I will add that too and then we can cleanup the scripts and environment variables considerably.
For instance, I don't think most of the
DOTNET_env variables are necessary anymore so that mess of wrappers can go away.At best we could simply keep a script that checks that all of the installed versions of tools are new enough for us and if not throw a warning.
Not sure how we'd handle forced downgrades, but if we encounter something like that we could probably resurrect the current fetching mechanism at that point.