Hi Peter,

Thanks indeed for all this work. There are bits I don't understand -- for instance, I thought Travis on Github had a hard limit of 30 minutes for a build process, which would prevent us from using it, but perhaps I misunderstood that. Anyway, I'm looking forward to learning more about your vision for the infrastructure.

Cheers,
Martin

On 2018-09-12 02:40 PM, Peter Stadler wrote:

Dear all,

as I reported during our Council meeting on Saturday(?) I wanted to clean up and reorganize our Docker images a little bit. Here’s a little update on my work so far:

a) every DockerHub repo is now turned into an „automated build“. This results in a better GitHub integration since every commit to a GitHub repo will trigger a fresh built of the respective Docker image at DockerHub. Sadly, there is no way of converting an existing repo so I had to nuke the Jenkins and the teidev-docker repos and recreate them. Hence, all stars and pull statistics got lost :(

b) because some repos do not get frequent commits, I added a cron job at my server to regularly trigger a rebuild of those images to keep them updated with their upstream images. Now that I’m writing this, I think that would make a nice Jenkins job. Will try and report back!

c) I was originally thinking that I’d like to alter the teidev-docker image to inherit from our Jenkins image (so we need to maintain only one Dockerfile with the software dependencies) – but when I was working on this it occurred to me, that we actually could be doing everything with the Jenkins image alone (by overwriting the entrypoint and cmd directives) and don’t need a dedicated image for building the Guidelines etc. locally. If you want to give it a try, install Docker, navigate to your Stylesheets directory and enter the command:

`docker run --rm -w /var/odd -it -v `pwd`:/var/odd --entrypoint "make" teic/jenkins:dev test `

This will run the tests on your machine, with your local files, with the software environment from the Jenkins Docker image. That’s cool, ain’t it?

d) I was taking first steps with Travis for RomaJS and it’s great fun, because (again) it integrates with GitHub so seamlessly. What we gain is automated testing for every branch and every pull request, nice badges, and even Slack notifications ;)

So, I added this Travis integration to the Stylesheets repo as well, using the Docker image and the above Docker command for testing. Another really cool thing IMHO, because (thanks to Docker) the commands run are using the very same software environment as our Jenkins does.

If you want to check, have a look at the .travis.yml file in the Stylesheets repo which is driving the integration tests. [1]

BTW, everybody who is a member of the GitHub TEIC organisation should be able to login and see/alter all the stuff at https://travis-ci.org/TEIC.

Of course, these new opportunities bring up a few new ideas. Here’s the proposal bit:

1) since we do not need a dedicated Docker image for building locally, I propose to delete the repo at DockerHub. At GitHub, I propose to archive the repo. Instead, I will add a section to the Jenkins Readme on how to build locally with the Docker image. Any other reference on how to build the Guidelines should be updated as well and have at least a pointer to the Jenkins Readme and the Jenkins Dockerfile. That would serve as our main „infrastructure as code“ documentation on how to set up the build environment.

2) I propose to add Travis integration to the TEI repo as well and set it up to (only) run the Tests (i.e. `make clean validate test`). Since the Guidelines depend on the Stylesheets to build, Travis would need to get them from Jenkins lastSuccesfullBuild (as Jenkins is currently doing)

3) Once we have the Travis integration, I propose to turn our dev branches into "protected“ branches. That means, on the pull request pages on GitHub (and only there) the test results will be added. An example for how this looks like can be seen at [2].

The advantage is for the issuer of the pull request to directly see whether it’s working or not. If the tests are failing the pull request is marked as „blocked“, but as an administrator (and probably all Council members have this role?) you are still able to merge.

4) A mid term goal could be to use Travis for running the tests and have Jenkins only build the artefacts. We could even replace Jenkins completely but then we’d need another place to push our build artefacts to.

Sorry for the lengthy mail.

Best

Peter

[1] https://github.com/TEIC/Stylesheets/blob/dev/.travis.yml

[2] https://github.com/eXist-db/exist/pull/2153