But, but ... I don't really speak 'ant' very well, but AFAIK DTDcmopat is a property, and it's set (to a boolean) on line 162. What I've read (e.g. https://stackoverflow.com/questions/5643918/if-and-unless-in-ant) shows the property being referenced in an @if or @unless as a Name, not a variable-like-reference. In truth (as the comment on 164 says) the value of DTDcompat should probably be passed in from the caller, defaulting to "true", but set to "false" for tei_customization. (But that's a bit of a pain to do in the Makefile. :-)
I checked your changes and see that you introduced a new(?) variable ${DTDcompat} in antruntest.xml which you refer to multiple times. Crucial part is in line https://github.com/TEIC/TEI/blob/c293b6275365b5eefc60399fbec60c7879bc7ee8/P5... where the @unless=„DTDcompat“ blocks the test. One obvious thing is that the variable must be referenced by ${DTDcompat} but I have no idea where you would want to inject the proper value?