Hi all,

+1 from me for option 2.5: Test1 —> "Legacy_Tests" and Test2  —> "Test". From what I can tell, it seems like the Tests are useful, but not fully reliable anyway given that I would think the ideal situation would be that Jenkins never fails (i.e. our tests should cover everything that Jenkins does, so that `dev` is seldom in a passing state in GitHub but not in Jenkins)

But agreed with Martin that Test2 should be rationalized and cleaned up before making the switch. That may also be a good opportunity to outline principles and practices for how those tests are structured, named, et cetera (how many things are tested per test, for instance? Personally, I'd be inclined to have more atomic tests—single files that test for one thing only, but perhaps that's a bit idealistic.) 

I feel like there's a few things that would need to be done (not necessarily in this order):

1) Figuring out what Test2 is currently testing
2) Determining structures, policies, best practices for Tests/test coverage (and formalizing those in some way—either in prose or programatically)
3) Figuring out what Test one is testing
4) Out of that list, seeing if there's anything that isn't in Test2 that we think should still be tested and then writing those tests
5) Moving the directories, updating prose, rewriting GitHub action scripts etc
6) Determining if there are things that neither test1 nor test2 test for, but we think ought to be tested for (i.e., visual regression?)
7) Writing out new tests as we see fit

In any case, would this be something that merits (and apologies if my nomenclature is off) a working group?

Best,
Joey

⎯
Joey Takeda
Developer, Digital Humanities Innovation Lab
Simon Fraser University Library
takeda@sfu.ca
Unceded territory of the səl̓ilw̓ətaʔɬ (Tsleil-Waututh), kʷikʷəƛ̓əm (Kwikwetlem), Sḵwx̱wú7mesh Úxwumixw (Squamish), and xʷməθkʷəy̓əm (Musqueam) Nations

On Oct 18, 2023, at 8:41 AM, Martin Holmes <mholmes@uvic.ca> wrote:

Hi Elisa,

When I started Test2, I went through all the old Test files, which are mostly unhelpfully named test1, test2 ... test40, and tried to figure out what the purpose of each test was; if I could figure it out, and if it was still relevant, I added content to one of the (hopefully more helpfully-named) files in Test2. Since then, Syd and I have come across a couple of instances where Test caught something that Test2 didn't, and remedied that, as well as adding some new components for newer features.

However, when I look in Test2 now, I see that there is now a file called test-382.xml which is a "testcase for #382 and PR #475". That filename is unhelpful, so before we make any change we should see if that test can be integrated into any other file, or if not, renamed to reflect its purpose (which seems to be to test that index elements in heads are not turned into TOC entries, I think). The corresponding output file test-382.html should also be renamed. I think this test should actually be integrated into testStructure1.xml, which already checks TOC-building.

I also think we should add some more detailed documentation that would help to avoid the proliferation of anonymous and mysterious test files in the future, as well as helping people to debug issues caught by the tests.

Is anyone else aware of anything missing from Test2?

Cheers,
Martin


On 2023-10-18 05:18, Elisa Beshero-Bondar wrote:
I like solution 3 and “Legacy_Tests” as well, but only if we are sure we’re ready to demote the original test as legacy. Does Test2 cover everything we care about now? Are we missing any tests there that matter and that the old tests cover?
Elisa
Elisa Beshero-Bondar, PhD (she/they)
Chair, TEI Technical Council
Program Chair of Digital Media, Arts, and Technology | Professor of Digital Humanities |  Director of the Digital Humanities Lab at Penn State Erie, The Behrend College
Typeset by hand on my iPhone
On Oct 17, 2023, at 5:42 PM, Bauman, Syd <s.bauman@northeastern.edu> wrote:


Oooh. I like the idea of  Test/ → Legacy_Test/  and  Test2/ → Test/.

------------------------------------------------------------------------

I lean towards 3), though perhaps Test/ can be kept with a new name, such as additional_tests or... Test2/

   Trish & I are doing the p5subset → Stylesheets process. Most of
   our time has been spent handling Test/. I think it is just too
   cumbersome — the amount of time we (Council, collectively) spend
   wrangling with Test/ is likely by now much greater than the amount
   of time it saves by catching bugs early.

   I think we need to do one of three things, toot sweet:

    1. Re-work the Test/Makefile so that the process is /much/
       easier: e.g., when comparisons are deferred, a shell script to
       perform them needs to be generated; this might be hard to
       impossible.
    2. Stop using Test/, it is just not helpful enough to be worth
       it. Leave the directory in the repo so that we can use it when
       there is a particularly gnarly case, or when a human is not
       waiting for it, but remove it from the general test procedure
       we use, e.g. when generating p5subset for the Stylesheets repo.
    3. Remove Test/ from the repo (and various instructions) entirely
       (and rename Test2/ to Test/ :-)

   My vote is for (2). I could be perhaps be talked into (3)
   reasonably easily; it would take some serious convincing to get me
   to vote for (1).

   P.S. BTW, Martin & I intended that we would do (3) when we
   developed Test2/. (I wanted to hang onto Test/, i.e. do (2), for
   awhile to ensure that everything tested by Test/ was also tested
   by Test2/.)


_______________________________________________
Tei-council mailing list
Tei-council@lists.tei-c.org
http://lists.lists.tei-c.org/mailman/listinfo/tei-council
_______________________________________________
Tei-council mailing list
Tei-council@lists.tei-c.org
http://lists.lists.tei-c.org/mailman/listinfo/tei-council

-- 
------------------------------------------
Martin Holmes
UVic Humanities Computing and Media Centre

I acknowledge and respect the lək̓ʷəŋən peoples on whose traditional territory the university stands and the Songhees, Esquimalt and WSÁNEĆ peoples whose historical relationships with the land continue to this day.
_______________________________________________
Tei-council mailing list
Tei-council@lists.tei-c.org
http://lists.lists.tei-c.org/mailman/listinfo/tei-council