Release post mortem
Hi All, 3.5.0 wasn't entirely smooth. Remaining tasks are reloading OxGarage and Roma Docker containers (request has been sent), doing the Oxygen plugin release, and doing the Debian release. The latter process fails due to an SSL error. I suspect it may have to do with the ancient version of Ant on the TEI server and maybe wonky SSL support in an older Java library. Upgrading Ant might fix everything. I think it worked before because Jenkins wasn't under HTTPS. Notes on updates needed to the release docs are at https://docs.google.com/document/d/1fTrCQrC8tRTZWLGScXKb0R_brwhOoRA6uta4c1hC... . Comments welcome. We got bitten (again!) by the circular dependency between the Stylesheets and Guidelines. The Stylesheets build started failing after the Guidelines were released because it uses the p5subset from the latest release, meaning we're not *actually* testing the Stylesheets against the latest GLs build. Martin spotted a problem with the SimplePrint exemplar post release, which we decided to go back and fix. Moving the att.tableDecoration class into the figure module broke SimplePrint because it references nearly everything directly, bypassing the modules. So we inadvertently lost table attributes by putting them alongside tables (where they should have been in the first place). This is something we'll have to keep an eye out for in the future. but note https://github.com/TEIC/TEI/issues/1853. I think that was the gist of it. Overall, we've had better releases, but it wasn't too bad. Elli may have some followup if I missed anything. Thanks to everyone who helped today! Hugh
I don't have much to add - mostly that we can continue to improve the instructions for the release. Also that the process beyond the actual guidelines compilation is still very complicated and requires a great deal of special knowledge, not to mention access privileges... But it worked, once again. The Raven has spread its wings. best to all --elli On Tue, Jan 29, 2019 at 9:22 PM Hugh Cayless <philomousos@gmail.com> wrote:
Hi All,
3.5.0 wasn't entirely smooth. Remaining tasks are reloading OxGarage and Roma Docker containers (request has been sent), doing the Oxygen plugin release, and doing the Debian release. The latter process fails due to an SSL error. I suspect it may have to do with the ancient version of Ant on the TEI server and maybe wonky SSL support in an older Java library. Upgrading Ant might fix everything. I think it worked before because Jenkins wasn't under HTTPS.
Notes on updates needed to the release docs are at https://docs.google.com/document/d/1fTrCQrC8tRTZWLGScXKb0R_brwhOoRA6uta4c1hC... . Comments welcome.
We got bitten (again!) by the circular dependency between the Stylesheets and Guidelines. The Stylesheets build started failing after the Guidelines were released because it uses the p5subset from the latest release, meaning we're not *actually* testing the Stylesheets against the latest GLs build.
Martin spotted a problem with the SimplePrint exemplar post release, which we decided to go back and fix. Moving the att.tableDecoration class into the figure module broke SimplePrint because it references nearly everything directly, bypassing the modules. So we inadvertently lost table attributes by putting them alongside tables (where they should have been in the first place). This is something we'll have to keep an eye out for in the future. but note https://github.com/TEIC/TEI/issues/1853.
I think that was the gist of it. Overall, we've had better releases, but it wasn't too bad. Elli may have some followup if I missed anything. Thanks to everyone who helped today!
Hugh
_______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council
Regarding the Debian packages: That’s an issue with the Java version (because older Java VMs do not support Let’s encrypt certificates). A working Java (1.8) is installed on the TEI server (and is the default for the `java` cli) but the JAVA_HOME environment variable for the tei user points at some ancient Java version. I updated `~/.bashrc` last time but again someone (some script) altered it and caused the ant task to fail. I updated the Debian packages just now by issuing ``` # export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk # ant ``` I’ll update tcw22 to add a warning for the correct Java version. Regarding Roma: I updated roma2.tei-c.org (which is running at Paderborn) and took a quick look for <object> — it’s there :) But a second pair of eyes to look for more issues would be great! The main roma (on the TEI server) needs to be updated (i.e. nuke, pull and recreate) by the server admin (Christof?) because we do not have the privileges to start and stop Docker containers on our own. We should negotiate that — maybe the tei user can be added to the docker group? Regarding OxGarage I updated oxgarage2.tei-c.org (running at Paderborn) to use the latest Stylesheets and TEI release. Again, the main OxGarage needs to be updated by the server admin on the tei server. Although the respective volumes or only mounted into the container, I remember Ian needed to use the absolute path rather than the "current“ symlink. That’s probably why it’s still using the old ones … Best Peter
Am 30.01.2019 um 05:05 schrieb Mylonas, Elli <elli_mylonas@brown.edu>:
I don't have much to add - mostly that we can continue to improve the instructions for the release.
Also that the process beyond the actual guidelines compilation is still very complicated and requires a great deal of special knowledge, not to mention access privileges... But it worked, once again. The Raven has spread its wings.
best to all --elli
On Tue, Jan 29, 2019 at 9:22 PM Hugh Cayless <philomousos@gmail.com <mailto:philomousos@gmail.com>> wrote: Hi All,
3.5.0 wasn't entirely smooth. Remaining tasks are reloading OxGarage and Roma Docker containers (request has been sent), doing the Oxygen plugin release, and doing the Debian release. The latter process fails due to an SSL error. I suspect it may have to do with the ancient version of Ant on the TEI server and maybe wonky SSL support in an older Java library. Upgrading Ant might fix everything. I think it worked before because Jenkins wasn't under HTTPS.
Notes on updates needed to the release docs are at https://docs.google.com/document/d/1fTrCQrC8tRTZWLGScXKb0R_brwhOoRA6uta4c1hC... <https://docs.google.com/document/d/1fTrCQrC8tRTZWLGScXKb0R_brwhOoRA6uta4c1hCiu4/edit#> . Comments welcome.
We got bitten (again!) by the circular dependency between the Stylesheets and Guidelines. The Stylesheets build started failing after the Guidelines were released because it uses the p5subset from the latest release, meaning we're not *actually* testing the Stylesheets against the latest GLs build.
Martin spotted a problem with the SimplePrint exemplar post release, which we decided to go back and fix. Moving the att.tableDecoration class into the figure module broke SimplePrint because it references nearly everything directly, bypassing the modules. So we inadvertently lost table attributes by putting them alongside tables (where they should have been in the first place). This is something we'll have to keep an eye out for in the future. but note https://github.com/TEIC/TEI/issues/1853 <https://github.com/TEIC/TEI/issues/1853>.
I think that was the gist of it. Overall, we've had better releases, but it wasn't too bad. Elli may have some followup if I missed anything. Thanks to everyone who helped today!
Hugh
_______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org <mailto:Tei-council@lists.tei-c.org> http://lists.lists.tei-c.org/mailman/listinfo/tei-council <http://lists.lists.tei-c.org/mailman/listinfo/tei-council> _______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council
I think in particular we should revisit the question of whether we want to continue to release on Sourceforge and to look seriously at automating releases on GitHub. I’ve been meaning to look into the latter for ages. H
On Jan 29, 2019, at 23:05, Mylonas, Elli <elli_mylonas@brown.edu> wrote:
I don't have much to add - mostly that we can continue to improve the instructions for the release.
Also that the process beyond the actual guidelines compilation is still very complicated and requires a great deal of special knowledge, not to mention access privileges... But it worked, once again. The Raven has spread its wings.
best to all --elli
On Tue, Jan 29, 2019 at 9:22 PM Hugh Cayless <philomousos@gmail.com> wrote: Hi All,
3.5.0 wasn't entirely smooth. Remaining tasks are reloading OxGarage and Roma Docker containers (request has been sent), doing the Oxygen plugin release, and doing the Debian release. The latter process fails due to an SSL error. I suspect it may have to do with the ancient version of Ant on the TEI server and maybe wonky SSL support in an older Java library. Upgrading Ant might fix everything. I think it worked before because Jenkins wasn't under HTTPS.
Notes on updates needed to the release docs are at https://docs.google.com/document/d/1fTrCQrC8tRTZWLGScXKb0R_brwhOoRA6uta4c1hC... . Comments welcome.
We got bitten (again!) by the circular dependency between the Stylesheets and Guidelines. The Stylesheets build started failing after the Guidelines were released because it uses the p5subset from the latest release, meaning we're not *actually* testing the Stylesheets against the latest GLs build.
Martin spotted a problem with the SimplePrint exemplar post release, which we decided to go back and fix. Moving the att.tableDecoration class into the figure module broke SimplePrint because it references nearly everything directly, bypassing the modules. So we inadvertently lost table attributes by putting them alongside tables (where they should have been in the first place). This is something we'll have to keep an eye out for in the future. but note https://github.com/TEIC/TEI/issues/1853.
I think that was the gist of it. Overall, we've had better releases, but it wasn't too bad. Elli may have some followup if I missed anything. Thanks to everyone who helped today!
Hugh
_______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council
Hmm, what is it that we gain by dropping releases on SourceForge? I thought it was a benefit to the user which does not cost us much. Best Peter
Am 30.01.2019 um 13:06 schrieb Hugh Cayless <philomousos@gmail.com>:
I think in particular we should revisit the question of whether we want to continue to release on Sourceforge and to look seriously at automating releases on GitHub. I’ve been meaning to look into the latter for ages.
H
On Jan 29, 2019, at 23:05, Mylonas, Elli <elli_mylonas@brown.edu> wrote:
I don't have much to add - mostly that we can continue to improve the instructions for the release.
Also that the process beyond the actual guidelines compilation is still very complicated and requires a great deal of special knowledge, not to mention access privileges... But it worked, once again. The Raven has spread its wings.
best to all --elli
On Tue, Jan 29, 2019 at 9:22 PM Hugh Cayless <philomousos@gmail.com> wrote: Hi All,
3.5.0 wasn't entirely smooth. Remaining tasks are reloading OxGarage and Roma Docker containers (request has been sent), doing the Oxygen plugin release, and doing the Debian release. The latter process fails due to an SSL error. I suspect it may have to do with the ancient version of Ant on the TEI server and maybe wonky SSL support in an older Java library. Upgrading Ant might fix everything. I think it worked before because Jenkins wasn't under HTTPS.
Notes on updates needed to the release docs are at https://docs.google.com/document/d/1fTrCQrC8tRTZWLGScXKb0R_brwhOoRA6uta4c1hC... . Comments welcome.
We got bitten (again!) by the circular dependency between the Stylesheets and Guidelines. The Stylesheets build started failing after the Guidelines were released because it uses the p5subset from the latest release, meaning we're not *actually* testing the Stylesheets against the latest GLs build.
Martin spotted a problem with the SimplePrint exemplar post release, which we decided to go back and fix. Moving the att.tableDecoration class into the figure module broke SimplePrint because it references nearly everything directly, bypassing the modules. So we inadvertently lost table attributes by putting them alongside tables (where they should have been in the first place). This is something we'll have to keep an eye out for in the future. but note https://github.com/TEIC/TEI/issues/1853.
I think that was the gist of it. Overall, we've had better releases, but it wasn't too bad. Elli may have some followup if I missed anything. Thanks to everyone who helped today!
Hugh
_______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council
Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council
It’s extra work, and requires a good deal of preliminary setup that newer Council members are less likely to find otherwise useful. But I note that SourceForge file releases still get more downloads still than the GitHub ones. I don’t know that most folks get copies of the releases that way, but people are clearly still using it, so I take it back :-).
On Jan 30, 2019, at 07:14, Peter Stadler <stadler@edirom.de> wrote:
Hmm, what is it that we gain by dropping releases on SourceForge? I thought it was a benefit to the user which does not cost us much.
Best Peter
Am 30.01.2019 um 13:06 schrieb Hugh Cayless <philomousos@gmail.com>:
I think in particular we should revisit the question of whether we want to continue to release on Sourceforge and to look seriously at automating releases on GitHub. I’ve been meaning to look into the latter for ages.
H
On Jan 29, 2019, at 23:05, Mylonas, Elli <elli_mylonas@brown.edu> wrote:
I don't have much to add - mostly that we can continue to improve the instructions for the release.
Also that the process beyond the actual guidelines compilation is still very complicated and requires a great deal of special knowledge, not to mention access privileges... But it worked, once again. The Raven has spread its wings.
best to all --elli
On Tue, Jan 29, 2019 at 9:22 PM Hugh Cayless <philomousos@gmail.com> wrote: Hi All,
3.5.0 wasn't entirely smooth. Remaining tasks are reloading OxGarage and Roma Docker containers (request has been sent), doing the Oxygen plugin release, and doing the Debian release. The latter process fails due to an SSL error. I suspect it may have to do with the ancient version of Ant on the TEI server and maybe wonky SSL support in an older Java library. Upgrading Ant might fix everything. I think it worked before because Jenkins wasn't under HTTPS.
Notes on updates needed to the release docs are at https://docs.google.com/document/d/1fTrCQrC8tRTZWLGScXKb0R_brwhOoRA6uta4c1hC... . Comments welcome.
We got bitten (again!) by the circular dependency between the Stylesheets and Guidelines. The Stylesheets build started failing after the Guidelines were released because it uses the p5subset from the latest release, meaning we're not *actually* testing the Stylesheets against the latest GLs build.
Martin spotted a problem with the SimplePrint exemplar post release, which we decided to go back and fix. Moving the att.tableDecoration class into the figure module broke SimplePrint because it references nearly everything directly, bypassing the modules. So we inadvertently lost table attributes by putting them alongside tables (where they should have been in the first place). This is something we'll have to keep an eye out for in the future. but note https://github.com/TEIC/TEI/issues/1853.
I think that was the gist of it. Overall, we've had better releases, but it wasn't too bad. Elli may have some followup if I missed anything. Thanks to everyone who helped today!
Hugh
_______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council
Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council
for the user is there any difference in the access and downloading from SF vs Git? In other words is it habit or is there some kind of affordance that makes SF easier? if just habit, we can start redirecting and slowly wean people of the habit. It does seem strange to have 2 locations for releases. Or is there something else we should be thinking about? --e On Wed, Jan 30, 2019 at 7:39 AM Hugh Cayless <philomousos@gmail.com> wrote:
It’s extra work, and requires a good deal of preliminary setup that newer Council members are less likely to find otherwise useful. But I note that SourceForge file releases still get more downloads still than the GitHub ones. I don’t know that most folks get copies of the releases that way, but people are clearly still using it, so I take it back :-).
On Jan 30, 2019, at 07:14, Peter Stadler <stadler@edirom.de> wrote:
Hmm, what is it that we gain by dropping releases on SourceForge? I thought it was a benefit to the user which does not cost us much.
Best Peter
Am 30.01.2019 um 13:06 schrieb Hugh Cayless <philomousos@gmail.com>:
I think in particular we should revisit the question of whether we want to continue to release on Sourceforge and to look seriously at automating releases on GitHub. I’ve been meaning to look into the latter for ages.
H
On Jan 29, 2019, at 23:05, Mylonas, Elli <elli_mylonas@brown.edu> wrote:
I don't have much to add - mostly that we can continue to improve the instructions for the release.
Also that the process beyond the actual guidelines compilation is still very complicated and requires a great deal of special knowledge, not to mention access privileges... But it worked, once again. The Raven has spread its wings.
best to all --elli
On Tue, Jan 29, 2019 at 9:22 PM Hugh Cayless <philomousos@gmail.com> wrote: Hi All,
3.5.0 wasn't entirely smooth. Remaining tasks are reloading OxGarage and Roma Docker containers (request has been sent), doing the Oxygen plugin release, and doing the Debian release. The latter process fails due to an SSL error. I suspect it may have to do with the ancient version of Ant on the TEI server and maybe wonky SSL support in an older Java library. Upgrading Ant might fix everything. I think it worked before because Jenkins wasn't under HTTPS.
Notes on updates needed to the release docs are at https://docs.google.com/document/d/1fTrCQrC8tRTZWLGScXKb0R_brwhOoRA6uta4c1hC... . Comments welcome.
We got bitten (again!) by the circular dependency between the Stylesheets and Guidelines. The Stylesheets build started failing after the Guidelines were released because it uses the p5subset from the latest release, meaning we're not *actually* testing the Stylesheets against the latest GLs build.
Martin spotted a problem with the SimplePrint exemplar post release, which we decided to go back and fix. Moving the att.tableDecoration class into the figure module broke SimplePrint because it references nearly everything directly, bypassing the modules. So we inadvertently lost table attributes by putting them alongside tables (where they should have been in the first place). This is something we'll have to keep an eye out for in the future. but note https://github.com/TEIC/TEI/issues/1853.
I think that was the gist of it. Overall, we've had better releases, but it wasn't too bad. Elli may have some followup if I missed anything. Thanks to everyone who helped today!
Hugh
_______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council
Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council
_______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council
I've been thinking about the circular dependency thing. What it basically means is that any given version of the Stylesheets MUST be functional for both the previous version of P5 (as released on the site) and the next version of P5 (in the dev branch, or ultimately in the master branch at release time). As long as we're clear on that, we can set up Jenkins build processes that test it. And if we ensure that's the case, then we can make a point of releasing the new Stylesheets version ahead of the P5 version, maybe a day or two before, so that we're not trying to do both at the same time. The only potential problem would be a situation in which a change in P5 requires a corresponding change in the Stylesheets that breaks backward compatibility. But that would surely be a bad thing anyway; we should be coding explicitly to support both old and new TEI structures in the Stylesheets. Cheers, Martin On 2019-01-29 8:05 p.m., Mylonas, Elli wrote:
I don't have much to add - mostly that we can continue to improve the instructions for the release.
Also that the process beyond the actual guidelines compilation is still very complicated and requires a great deal of special knowledge, not to mention access privileges... But it worked, once again. The Raven has spread its wings.
best to all --elli
On Tue, Jan 29, 2019 at 9:22 PM Hugh Cayless <philomousos@gmail.com <mailto:philomousos@gmail.com>> wrote:
Hi All,
3.5.0 wasn't entirely smooth. Remaining tasks are reloading OxGarage and Roma Docker containers (request has been sent), doing the Oxygen plugin release, and doing the Debian release. The latter process fails due to an SSL error. I suspect it may have to do with the ancient version of Ant on the TEI server and maybe wonky SSL support in an older Java library. Upgrading Ant might fix everything. I think it worked before because Jenkins wasn't under HTTPS.
Notes on updates needed to the release docs are at https://docs.google.com/document/d/1fTrCQrC8tRTZWLGScXKb0R_brwhOoRA6uta4c1hC... . Comments welcome.
We got bitten (again!) by the circular dependency between the Stylesheets and Guidelines. The Stylesheets build started failing after the Guidelines were released because it uses the p5subset from the latest release, meaning we're not *actually* testing the Stylesheets against the latest GLs build.
Martin spotted a problem with the SimplePrint exemplar post release, which we decided to go back and fix. Moving the att.tableDecoration class into the figure module broke SimplePrint because it references nearly everything directly, bypassing the modules. So we inadvertently lost table attributes by putting them alongside tables (where they should have been in the first place). This is something we'll have to keep an eye out for in the future. but note https://github.com/TEIC/TEI/issues/1853.
I think that was the gist of it. Overall, we've had better releases, but it wasn't too bad. Elli may have some followup if I missed anything. Thanks to everyone who helped today!
Hugh
_______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org <mailto:Tei-council@lists.tei-c.org> http://lists.lists.tei-c.org/mailman/listinfo/tei-council
At first blush the sounds like a very reasonable analysis, Martin. Strikes me as difficult, but probably possible, to automate the idea that Test2/ runs (by default) on both tei:current and on tei:current-1. The driver script would have to look in http://www.tei-c.org/Vault/P5/ and have some heuristics to determine the latest and latest minus one releases, then run all the tests twice, once for each. (This behavior should, of course, be something that could be overridden at the commandline so you can run the tests against any version you want.)
I've been thinking about the circular dependency thing. What it basically means is that any given version of the Stylesheets MUST be functional for both the previous version of P5 (as released on the site) and the next version of P5 (in the dev branch, or ultimately in the master branch at release time).
As long as we're clear on that, we can set up Jenkins build processes that test it. And if we ensure that's the case, then we can make a point of releasing the new Stylesheets version ahead of the P5 version, maybe a day or two before, so that we're not trying to do both at the same time.
The only potential problem would be a situation in which a change in P5 requires a corresponding change in the Stylesheets that breaks backward compatibility. But that would surely be a bad thing anyway; we should be coding explicitly to support both old and new TEI structures in the Stylesheets.
Wouldn't it be simpler to run the tests against the TEI master branch (which should reflect the current release) and the dev branch (which should reflect the next one)? Cheers, Martin On 2019-01-31 6:44 a.m., Syd Bauman wrote:
At first blush the sounds like a very reasonable analysis, Martin.
Strikes me as difficult, but probably possible, to automate the idea that Test2/ runs (by default) on both tei:current and on tei:current-1.
The driver script would have to look in http://www.tei-c.org/Vault/P5/ and have some heuristics to determine the latest and latest minus one releases, then run all the tests twice, once for each. (This behavior should, of course, be something that could be overridden at the commandline so you can run the tests against any version you want.)
I've been thinking about the circular dependency thing. What it basically means is that any given version of the Stylesheets MUST be functional for both the previous version of P5 (as released on the site) and the next version of P5 (in the dev branch, or ultimately in the master branch at release time).
As long as we're clear on that, we can set up Jenkins build processes that test it. And if we ensure that's the case, then we can make a point of releasing the new Stylesheets version ahead of the P5 version, maybe a day or two before, so that we're not trying to do both at the same time.
The only potential problem would be a situation in which a change in P5 requires a corresponding change in the Stylesheets that breaks backward compatibility. But that would surely be a bad thing anyway; we should be coding explicitly to support both old and new TEI structures in the Stylesheets.
participants (5)
-
Hugh Cayless -
Martin Holmes -
Mylonas, Elli -
Peter Stadler -
Syd Bauman