Re: [Tei-council] shooting the messenger : make still broken

Done - jenkins3 is ready to help Raff On Wed, Jul 3, 2019 at 2:18 AM Peter Stadler wrote:

Ok, as most have probably noticed the builds are stable again :)

Thanks to Hugh for adding the catalog files which map external URLs to local files, thus circumventing the rather conservative TEI server setup which only allowed for a certain amount of requests (per minute?) For the catalogs to work a resolver library is needed on the Jenkins servers which is now in place at Victoria and Paderborn but not at Maryland. @raff could you please update your Jenkins from the tei/jenkins:dev image to pick up the changes?

There was another tiny (but thorny) path issue with the build of the Debian packages for the Stylesheets which I fixed in https://github.com/TEIC/Stylesheets/commit/6e320b5b4bb764bedd5ce4bc89d5abfc4...

Putting it all together, I think we have our build infrastructure prepared for the upcoming release — hurray :) (i.e. `make` should work on the Jenkins servers and locally)

Best Peter

...

Am 01.07.2019 um 14:47 schrieb Hugh Cayless :

On Mon, Jul 1, 2019 at 8:34 AM Peter Stadler < pstadler@mail.uni-paderborn.de> wrote: a) which user is plagued? I thought the issue only surfaced with the Docker image because this features some ‚unconventional‘ paths.

All of them? The build wasn't working for anyone because of the server errors. Plus, we want to be able to build the Guidelines offline anyway (or at least, I do). Anyone who isn't running Linux (maybe Debian, in fact) who might want to set up builds locally without using Docker will be impacted.

b) to me it’s not about the size of the jar but about the maintainability of the repo and the build process. To me, the current folder https://github.com/TEIC/TEI/tree/dev/P5/Utilities/lib is a real mess because I have no idea which of these libraries are actually (still) needed and whether they are included properly or whether some commands still default to the system versions. In short, that’s no dependency management.

I hear you, and I sympathise, but sometimes we have to make compromises. A better solution might be to do dependency management with Maven, or Gradle, vel sim. but that adds its own costs.

I just created https://github.com/TEIC/Jenkins/pull/6 which fixes the issue with the ‚unconventional‘ paths of the Docker image.

I was a little reluctant to take that route for the same sorts of reasons you cite in #b :-).

Bet Peter

...

Am 01.07.2019 um 14:17 schrieb Hugh Cayless :

If the choice is between adding an 82K .jar file to the repo that might be elsewhere on your system or making it harder for users to build the Guidelines successfully without being plagued by nonsensical error messages, I know what I'd choose, but I guess it's up to Council, really.

Hugh

On Mon, Jul 1, 2019 at 8:08 AM Peter Stadler < pstadler@mail.uni-paderborn.de> wrote: Sorry, I don’t understand … I’d say, if your base system (where you want to build the Guidelines) is MacOS you should make sure that your (local) ant has access to the proper (local) libraries. For the Docker image we (as Council) need to make sure it works, hence my proposal to symlink /usr/share/java to ~/.ant/lib. That solution is only intended for the Jenkins Dockerfile.

Best Peter

...

Am 01.07.2019 um 14:02 schrieb Hugh Cayless :

Won't work on Windows or OS X though. Macs have a /usr/share/java, but there's nothing useful in it.

On Mon, Jul 1, 2019 at 7:39 AM Peter Stadler < pstadler@mail.uni-paderborn.de> wrote: In general, I’d say that this is a standard component which should be provided by the respective base system, not by our repository. For the Docker image, we should provide a way to make this available to ant. A rather simple approach would be to symlink /usr/share/java to ~/.ant/lib. (I just tried and `ant -diagnostics` shows it works)

Best Peter

...

Am 01.07.2019 um 13:03 schrieb Hugh Cayless

Can we depend on the resolver being in /usr/share/java on every system though? I'd have thought not.

I suspect the Jenkins Docker build does a not-altogether-standard install of OpenJDK 8, meaning not everything is where ant might expect to find it.

On Mon, Jul 1, 2019 at 6:07 AM Peter Stadler < pstadler@mail.uni-paderborn.de> wrote: tools.jar shouldn’t be an issue because although being in a different place it’s in the java classpath. The resolver.jar indeed is not. It’s installed as xml-resolver-1.2.jar under /usr/share/java/ which is not in the java classpath.

Rather than adding this jar to our Utilities folder (which is already cluttered with external libraries) I’d prefer to fix the Docker image by adding /usr/share/java/ to the java classpath globally.

Best Peter

...

Am 30.06.2019 um 17:50 schrieb Hugh Cayless < philomousos@gmail.com>:

Just trying out the Jenkins Docker, and indeed, tools.jar is not in /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar, but it *is* in /usr/local/openjdk-8/lib/, so something funky is going on with that...

On Sun, Jun 30, 2019 at 11:19 AM Hugh Cayless < philomousos@gmail.com> wrote: The Jenkins Docker build should have a proper JDK installed automatically, but maybe it's not where ant thinks it should be?

On Sun, Jun 30, 2019 at 11:07 AM Lou Burnard < lou.burnard@retired.ox.ac.uk> wrote: Installing open-jdk-8-jdk got rid of those messages for me (on ancient version of ubuntu)

On 30/06/2019 15:53, Hugh Cayless wrote: > I see the following message in the logs: "Unable to locate tools.jar. > Expected to find it in /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar" I > think that means the JDK isn't installed, or isn't completely installed? > > > On Sun, Jun 30, 2019 at 10:45 AM Hugh Cayless > > wrote: > > >> It's still failing on Jenkins though...not sure why. It's still giving the >> "Warning: XML resolver not found; external catalogs will be ignored" >> message, which I no longer get locally, so I'm guessing it's not seeing, or >> is failing to deal with, the resolver-1.2.0.jar library. >> >> On Sun, Jun 30, 2019 at 10:25 AM Lou Burnard >> >> >> wrote: >> >> >>> Thanks Hugh. With those changes, it works fine for me too. Someone should >>> remember to tell Luis not to bother, all the same! >>> On 30/06/2019 14:45, Hugh Cayless wrote: >>> >>> I've had a go at fixing this by adding an XML catalog for the RNGs in >>> Exemplars, and (incidentally) fixing that dratted warning about the missing >>> resolver by adding one. It builds successfully for me locally. We'll see if >>> it works for Jenkins... >>> >>> Hugh >>> >>> On Sun, Jun 30, 2019 at 7:22 AM Lou Burnard >>> >>> >>> wrote: >>> >>> >>> The real question is: why are we including this file via an HTTP copy from >>> the TEI website, when it's already present in the Exemplums directory? A >>> gnomic comment in the ODD source suggests this is something to do with >>> oXGarage: >>> >>> line 70: >>> >>> <moduleRef xmlns= >>> "http://www.tei-c.org/ns/1.0" http://www.tei-c.org/ns/1.0< http://www.tei-c.org/ns/1.0> http://www.tei-c.org/ns/1.0 xmlns:rng=" http://relaxng.org/ns/structure/1.0" <http://relaxng.org/ns/structure/1.0 http://relaxng.org/ns/structure/1.0 < http://relaxng.org/ns/structure/1.0> url=" https://www.tei-c.org/release/xml/tei/custom/schema/relaxng/svg11.rng" < https://www.tei-c.org/release/xml/tei/custom/schema/relaxng/svg11.rng>< https://www.tei-c.org/release/xml/tei/custom/schema/relaxng/svg11.rng> < https://www.tei-c.org/release/xml/tei/custom/schema/relaxng/svg11.rng> >>> > >>> <!-- this needs to be an absolute URI to keep oxGarage happy --> >>> >>> (This is for the inclusion of SVG rng, but the same applies, presumably, >>> to the inclusion of mathml.rng) >>> >>> But >>> >>> (a) is this still really true? is the new oxGarage less demanding? Peter? >>> >>> (b) even if it is true, wouldn't it be better to point to a website run by >>> the people who run SVG and Mathml (respectively) assuming there is such a >>> thing? >>> >>> The copy in the Exemplum directory is exactly the same as the one I just >>> downloaded from the TEI website. Since the former has been languishing >>> there for THIRTEEN YEARS, it's not the most dynamic of data in any case... >>> >>> It seems to me it would be better to try to work within the constraints of >>> having a professionally maintained secure website than bend the rules. I >>> assume Luis set this limit for a purpose, and I'd rather trust his judgment >>> if possible! >>> On 30/06/2019 05:31, Martin Holmes wrote: >>> >>> I think I was wrong about this: the server error is 429, which is "too >>> many requests". I think the tei server may be set up to reject repeated >>> requests for the same files, which is something the build process does as a >>> matter of course. I've written to Luis to ask if there's a limit, and if >>> so, whether it can be lifted. >>> >>> [xslt] >>> /var/lib/jenkins/jobs/Stylesheets-dev/lastSuccessful/archive/dist/xml/tei/stylesheet/odds/teiodds.xsl:1236:20: >>> Fatal Error! I/O error reported by XML parser processinghttps:// >>> www.tei-c.org/release/xml/tei/custom/schema/relaxng/svg11.rng >>> : >>> Server returned HTTP response code: 429 for URL: >>> https://www.tei-c.org/release/xml/tei/custom/schema/relaxng/svg11.rng >>> >>> Cause: java.io.IOException: Server returned HTTP response code: 429 for >>> URL: >>> https://www.tei-c.org/release/xml/tei/custom/schema/relaxng/svg11.rng >>> >>> >>> Cheers, >>> Martin >>> >>> On 2019-06-29 3:16 p.m., Martin Holmes wrote: >>> >>> I think this is caused by the fact that the >>> tei-c.org >>> server is not >>> serving some files correctly. If you wget that URL, you'll see that you get >>> a 301 redirect to >>> google.com >>> . I've reported this to Luis, but if I >>> understand his response correctly, it's some sort of security measure he >>> put in place. I've asked him if he can change the setup so that files like >>> rng are served appropriately as text/xml, and he agreed to look at it. >>> >>> Cheers, >>> Martin >>> >>> On 2019-06-29 1:43 p.m., Lou Burnard wrote: >>> >>> >>> Well, the build is now a lot healthier. The new validation method which >>> checks examples against the schema being documented, rather than TEI All as >>> before threw up a number of bugs in the source of the simplePrint odd. So >>> not all change is bad! >>> >>> The Make still fails though: now with a message the meaning of which >>> defeats me: Anyone got any idea what's going on here? >>> >>> BUILD FAILED >>> /home/lou/Public/TEI/P5/Test/antruntest.xml:147: Fatal error during >>> transformation using >>> /usr/share/xml/tei/stylesheet/profiles/tei/relaxng/to.xsl: I/O error >>> reported by XML parser processinghttps:// >>> www.tei-c.org/release/xml/tei/Exemplars/mathml2-qname-1.mod.rng >>> : >>> Server returned HTTP response code: 429 for URL: >>> https://www.tei-c.org/release/xml/tei/Exemplars/mathml2-qname-1.mod.rng >>> ; >>> SystemID: >>> file:/usr/share/xml/tei/stylesheet/odds/teiodds.xsl >>> ; Line#: >>> 1292; Column#: 21 >>> >>> Total time: 9 seconds >>> Makefile:54: recipe for target 'tei_allPlus.special' failed >>> make: *** [tei_allPlus.special] Error 1 >>> >>> I can live happily without tei_allPlus.special tbh. >>> >>> >>> On 29/06/2019 21:32, Martin Holmes wrote: >>> >>> Hi Lou, >>> >>> Our main objective was to try to get the build working ahead of the >>> release, not to say that what was done in the simplePrint ODD was wrong. I >>> think you're right that there's something broken in the Stylesheets that >>> wasn't broken before, but all our attempts to figure that out have failed >>> so far. You're very welcome to help with that, and I really wish you would, >>> rather than shouting at people. >>> >>> Cheers, >>> Martin >>> >>> On 2019-06-29 8:50 a.m., Lou Burnard wrote: >>> >>> Thinking about this debacle again, it seems to me that the approach taken >>> in issue #370 (oh tei_simpleprint.odd has stopped working: so there must be >>> something wrong with it) is really not defensible. >>> >>> 1. The unmodified tei_simplePrint odd is structured in exactly the same >>> way as countless other ODDs, not excluding the TEI Guidelines. It has free >>> floating globs of specification elements, which are invoked from a single >>> schemaSpec by means of specGrpRefs. >>> >>> 2. The only thing which is slightly different about it is that some >>> objects are declared more than once in multiple specGrps, with >>> @mode=change. This is because the specGrps are organised thematically (for >>> example one contains changes needed to introduce processing models, another >>> contains changes needed to introduce schematron rules). This kind of >>> logical organisation is perfectly reasonable as far as I am aware, indeed >>> why else bother to have specGrps. >>> >>> 3. The only objection I can see to it is that an ODD processor has to >>> decide how to combine two or more @mode=change specs. It's not just a >>> matter of applying one set of changes: there may be a bunch of them. This >>> is kind of the way unification grammars work in NLP: you build up the >>> complete spec one bit at a time. Yes, occasionally you may find >>> contradictions -- one spec adds an attribute x (say) which another one >>> deletes, or gives a different datatype to. But this is not the only place >>> where ODD processing is underspecified and rules of thumb are not hard to >>> imagine (e.g. delete always wins; in the absence of delete, most recent >>> modification always wins; etc.). >>> >>> 4. Moreover, and this is where I am really a bit annoyed, THIS USED TO >>> WORK! So some infrastructural change has made it impossible to run the >>> makefile and impossible to use the simpleprint odd as originally designed. >>> And instead of trying to find what change has caused this, we are shooting >>> the messenger. This my friends is called zealotry. In the immortal words of >>> Michael SpMcQ in another recent context, "please don't". >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Tei-council mailing >>> listTei-council@lists.tei-c.orghttp:// lists.lists.tei-c.org/mailman/listinfo/tei-council >>> >>> >>> _______________________________________________ >>> Tei-council mailing >>> listTei-council@lists.tei-c.orghttp:// lists.lists.tei-c.org/mailman/listinfo/tei-council >>> >>> >>> _______________________________________________ >>> Tei-council mailing >>> listTei-council@lists.tei-c.orghttp:// lists.lists.tei-c.org/mailman/listinfo/tei-council >>> >>> >>> >>> _______________________________________________ >>> Tei-council mailing >>> listTei-council@lists.tei-c.orghttp:// lists.lists.tei-c.org/mailman/listinfo/tei-council >>> >>> >>> _______________________________________________ >>> Tei-council mailing list >>> >>> Tei-council@lists.tei-c.org >>> http://lists.lists.tei-c.org/mailman/listinfo/tei-council >>> >>> >>> > > > _______________________________________________ > Tei-council mailing list > > Tei-council@lists.tei-c.org > http://lists.lists.tei-c.org/mailman/listinfo/tei-council _______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council _______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council

_______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council