Oooh. I like the idea of Test/ → Legacy_Test/ and Test2/ → Test/. ________________________________ I lean towards 3), though perhaps Test/ can be kept with a new name, such as additional_tests or... Test2/ Trish & I are doing the p5subset → Stylesheets process. Most of our time has been spent handling Test/. I think it is just too cumbersome — the amount of time we (Council, collectively) spend wrangling with Test/ is likely by now much greater than the amount of time it saves by catching bugs early. I think we need to do one of three things, toot sweet: 1. Re-work the Test/Makefile so that the process is much easier: e.g., when comparisons are deferred, a shell script to perform them needs to be generated; this might be hard to impossible. 2. Stop using Test/, it is just not helpful enough to be worth it. Leave the directory in the repo so that we can use it when there is a particularly gnarly case, or when a human is not waiting for it, but remove it from the general test procedure we use, e.g. when generating p5subset for the Stylesheets repo. 3. Remove Test/ from the repo (and various instructions) entirely (and rename Test2/ to Test/ :-) My vote is for (2). I could be perhaps be talked into (3) reasonably easily; it would take some serious convincing to get me to vote for (1). P.S. BTW, Martin & I intended that we would do (3) when we developed Test2/. (I wanted to hang onto Test/, i.e. do (2), for awhile to ensure that everything tested by Test/ was also tested by Test2/.)

With new people joining Council, these next couple of months are definitely timely to review and update our standard operating testing procedures. Trish and Syd, thanks for working on this just now! Elisa Elisa Beshero-Bondar, PhD (she/they) Chair, TEI Technical Council Program Chair of Digital Media, Arts, and Technology | Professor of Digital Humanities | Director of the Digital Humanities Lab at Penn State Erie, The Behrend College Typeset by hand on my iPhone

On Oct 18, 2023, at 8:18 AM, Elisa Beshero-Bondar wrote:

I like solution 3 and “Legacy_Tests” as well, but only if we are sure we’re ready to demote the original test as legacy. Does Test2 cover everything we care about now? Are we missing any tests there that matter and that the old tests cover?

Elisa

Elisa Beshero-Bondar, PhD (she/they) Chair, TEI Technical Council Program Chair of Digital Media, Arts, and Technology | Professor of Digital Humanities | Director of the Digital Humanities Lab at Penn State Erie, The Behrend College

Typeset by hand on my iPhone

...

...

On Oct 17, 2023, at 5:42 PM, Bauman, Syd wrote:

 Oooh. I like the idea of Test/ → Legacy_Test/ and Test2/ → Test/.

I lean towards 3), though perhaps Test/ can be kept with a new name, such as additional_tests or... Test2/

Trish & I are doing the p5subset → Stylesheets process. Most of our time has been spent handling Test/. I think it is just too cumbersome — the amount of time we (Council, collectively) spend wrangling with Test/ is likely by now much greater than the amount of time it saves by catching bugs early.

I think we need to do one of three things, toot sweet: Re-work the Test/Makefile so that the process is much easier: e.g., when comparisons are deferred, a shell script to perform them needs to be generated; this might be hard to impossible. Stop using Test/, it is just not helpful enough to be worth it. Leave the directory in the repo so that we can use it when there is a particularly gnarly case, or when a human is not waiting for it, but remove it from the general test procedure we use, e.g. when generating p5subset for the Stylesheets repo. Remove Test/ from the repo (and various instructions) entirely (and rename Test2/ to Test/ :-) My vote is for (2). I could be perhaps be talked into (3) reasonably easily; it would take some serious convincing to get me to vote for (1).

P.S. BTW, Martin & I intended that we would do (3) when we developed Test2/. (I wanted to hang onto Test/, i.e. do (2), for awhile to ensure that everything tested by Test/ was also tested by Test2/.)

_______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council

Hi all, +1 from me for option 2.5: Test1 —> "Legacy_Tests" and Test2 —> "Test". From what I can tell, it seems like the Tests are useful, but not fully reliable anyway given that I would think the ideal situation would be that Jenkins never fails (i.e. our tests should cover everything that Jenkins does, so that `dev` is seldom in a passing state in GitHub but not in Jenkins) But agreed with Martin that Test2 should be rationalized and cleaned up before making the switch. That may also be a good opportunity to outline principles and practices for how those tests are structured, named, et cetera (how many things are tested per test, for instance? Personally, I'd be inclined to have more atomic tests—single files that test for one thing only, but perhaps that's a bit idealistic.) I feel like there's a few things that would need to be done (not necessarily in this order): 1) Figuring out what Test2 is currently testing 2) Determining structures, policies, best practices for Tests/test coverage (and formalizing those in some way—either in prose or programatically) 3) Figuring out what Test one is testing 4) Out of that list, seeing if there's anything that isn't in Test2 that we think should still be tested and then writing those tests 5) Moving the directories, updating prose, rewriting GitHub action scripts etc 6) Determining if there are things that neither test1 nor test2 test for, but we think ought to be tested for (i.e., visual regression?) 7) Writing out new tests as we see fit In any case, would this be something that merits (and apologies if my nomenclature is off) a working group? Best, Joey ⎯ Joey Takeda Developer, Digital Humanities Innovation Lab Simon Fraser University Library takeda@sfu.ca Unceded territory of the səl̓ilw̓ətaʔɬ (Tsleil-Waututh), kʷikʷəƛ̓əm (Kwikwetlem), Sḵwx̱wú7mesh Úxwumixw (Squamish), and xʷməθkʷəy̓əm (Musqueam) Nations On Oct 18, 2023, at 8:41 AM, Martin Holmes wrote: Hi Elisa, When I started Test2, I went through all the old Test files, which are mostly unhelpfully named test1, test2 ... test40, and tried to figure out what the purpose of each test was; if I could figure it out, and if it was still relevant, I added content to one of the (hopefully more helpfully-named) files in Test2. Since then, Syd and I have come across a couple of instances where Test caught something that Test2 didn't, and remedied that, as well as adding some new components for newer features. However, when I look in Test2 now, I see that there is now a file called test-382.xml which is a "testcase for #382 and PR #475". That filename is unhelpful, so before we make any change we should see if that test can be integrated into any other file, or if not, renamed to reflect its purpose (which seems to be to test that index elements in heads are not turned into TOC entries, I think). The corresponding output file test-382.html should also be renamed. I think this test should actually be integrated into testStructure1.xml, which already checks TOC-building. I also think we should add some more detailed documentation that would help to avoid the proliferation of anonymous and mysterious test files in the future, as well as helping people to debug issues caught by the tests. Is anyone else aware of anything missing from Test2? Cheers, Martin On 2023-10-18 05:18, Elisa Beshero-Bondar wrote: I like solution 3 and “Legacy_Tests” as well, but only if we are sure we’re ready to demote the original test as legacy. Does Test2 cover everything we care about now? Are we missing any tests there that matter and that the old tests cover? Elisa Elisa Beshero-Bondar, PhD (she/they) Chair, TEI Technical Council Program Chair of Digital Media, Arts, and Technology | Professor of Digital Humanities | Director of the Digital Humanities Lab at Penn State Erie, The Behrend College Typeset by hand on my iPhone On Oct 17, 2023, at 5:42 PM, Bauman, Syd wrote:  Oooh. I like the idea of Test/ → Legacy_Test/ and Test2/ → Test/. ------------------------------------------------------------------------ I lean towards 3), though perhaps Test/ can be kept with a new name, such as additional_tests or... Test2/ Trish & I are doing the p5subset → Stylesheets process. Most of our time has been spent handling Test/. I think it is just too cumbersome — the amount of time we (Council, collectively) spend wrangling with Test/ is likely by now much greater than the amount of time it saves by catching bugs early. I think we need to do one of three things, toot sweet: 1. Re-work the Test/Makefile so that the process is /much/ easier: e.g., when comparisons are deferred, a shell script to perform them needs to be generated; this might be hard to impossible. 2. Stop using Test/, it is just not helpful enough to be worth it. Leave the directory in the repo so that we can use it when there is a particularly gnarly case, or when a human is not waiting for it, but remove it from the general test procedure we use, e.g. when generating p5subset for the Stylesheets repo. 3. Remove Test/ from the repo (and various instructions) entirely (and rename Test2/ to Test/ :-) My vote is for (2). I could be perhaps be talked into (3) reasonably easily; it would take some serious convincing to get me to vote for (1). P.S. BTW, Martin & I intended that we would do (3) when we developed Test2/. (I wanted to hang onto Test/, i.e. do (2), for awhile to ensure that everything tested by Test/ was also tested by Test2/.) _______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council _______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council -- ------------------------------------------ Martin Holmes UVic Humanities Computing and Media Centre I acknowledge and respect the lək̓ʷəŋən peoples on whose traditional territory the university stands and the Songhees, Esquimalt and WSÁNEĆ peoples whose historical relationships with the land continue to this day. _______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.orgmailto:Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council

If we’re talking about moving Test to a legacy directory, my vote would be to kill it instead. It will only get more and more out of sync over time. It’ll be useless inside 6 months. I think we should do what Martin says and make Test2 thoroughly comprehensible and documented and then we can get rid of Test. Hugh

On Oct 18, 2023, at 11:41, Martin Holmes wrote:

Hi Elisa,

When I started Test2, I went through all the old Test files, which are mostly unhelpfully named test1, test2 ... test40, and tried to figure out what the purpose of each test was; if I could figure it out, and if it was still relevant, I added content to one of the (hopefully more helpfully-named) files in Test2. Since then, Syd and I have come across a couple of instances where Test caught something that Test2 didn't, and remedied that, as well as adding some new components for newer features.

However, when I look in Test2 now, I see that there is now a file called test-382.xml which is a "testcase for #382 and PR #475". That filename is unhelpful, so before we make any change we should see if that test can be integrated into any other file, or if not, renamed to reflect its purpose (which seems to be to test that index elements in heads are not turned into TOC entries, I think). The corresponding output file test-382.html should also be renamed. I think this test should actually be integrated into testStructure1.xml, which already checks TOC-building.

I also think we should add some more detailed documentation that would help to avoid the proliferation of anonymous and mysterious test files in the future, as well as helping people to debug issues caught by the tests.

Is anyone else aware of anything missing from Test2?

Cheers, Martin

...

On 2023-10-18 05:18, Elisa Beshero-Bondar wrote: I like solution 3 and “Legacy_Tests” as well, but only if we are sure we’re ready to demote the original test as legacy. Does Test2 cover everything we care about now? Are we missing any tests there that matter and that the old tests cover? Elisa Elisa Beshero-Bondar, PhD (she/they) Chair, TEI Technical Council Program Chair of Digital Media, Arts, and Technology | Professor of Digital Humanities | Director of the Digital Humanities Lab at Penn State Erie, The Behrend College Typeset by hand on my iPhone

...

...

On Oct 17, 2023, at 5:42 PM, Bauman, Syd wrote:

 Oooh. I like the idea of Test/ → Legacy_Test/ and Test2/ → Test/.

------------------------------------------------------------------------

I lean towards 3), though perhaps Test/ can be kept with a new name, such as additional_tests or... Test2/

Trish & I are doing the p5subset → Stylesheets process. Most of our time has been spent handling Test/. I think it is just too cumbersome — the amount of time we (Council, collectively) spend wrangling with Test/ is likely by now much greater than the amount of time it saves by catching bugs early.

I think we need to do one of three things, toot sweet:

1. Re-work the Test/Makefile so that the process is /much/ easier: e.g., when comparisons are deferred, a shell script to perform them needs to be generated; this might be hard to impossible. 2. Stop using Test/, it is just not helpful enough to be worth it. Leave the directory in the repo so that we can use it when there is a particularly gnarly case, or when a human is not waiting for it, but remove it from the general test procedure we use, e.g. when generating p5subset for the Stylesheets repo. 3. Remove Test/ from the repo (and various instructions) entirely (and rename Test2/ to Test/ :-)

My vote is for (2). I could be perhaps be talked into (3) reasonably easily; it would take some serious convincing to get me to vote for (1).

P.S. BTW, Martin & I intended that we would do (3) when we developed Test2/. (I wanted to hang onto Test/, i.e. do (2), for awhile to ensure that everything tested by Test/ was also tested by Test2/.)

_______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council

---

Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council

-- ------------------------------------------ Martin Holmes UVic Humanities Computing and Media Centre

I acknowledge and respect the lək̓ʷəŋən peoples on whose traditional territory the university stands and the Songhees, Esquimalt and WSÁNEĆ peoples whose historical relationships with the land continue to this day. _______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council

Hello, I agree with Hugh that we have very little to gain by moving Test to a legacy directory (except if we want additional input test files, but even then, the lack of intuitive titles and documentation makes them hard to browse, therefore it might be faster to just create a test file when needed). Concerning things that are missing from Test2, I think that there are a few transformations without tests (e.g. xlsxtotei, csvtotei, teitojson). Best, H. On Thu, 19 Oct 2023 at 01:29, Hugh Cayless wrote:

If we’re talking about moving Test to a legacy directory, my vote would be to kill it instead. It will only get more and more out of sync over time. It’ll be useless inside 6 months.

I think we should do what Martin says and make Test2 thoroughly comprehensible and documented and then we can get rid of Test.

Hugh

...

On Oct 18, 2023, at 11:41, Martin Holmes wrote:

Hi Elisa,

When I started Test2, I went through all the old Test files, which are mostly unhelpfully named test1, test2 ... test40, and tried to figure out what the purpose of each test was; if I could figure it out, and if it was still relevant, I added content to one of the (hopefully more helpfully-named) files in Test2. Since then, Syd and I have come across a couple of instances where Test caught something that Test2 didn't, and remedied that, as well as adding some new components for newer features.

However, when I look in Test2 now, I see that there is now a file called test-382.xml which is a "testcase for #382 and PR #475". That filename is unhelpful, so before we make any change we should see if that test can be integrated into any other file, or if not, renamed to reflect its purpose (which seems to be to test that index elements in heads are not turned into TOC entries, I think). The corresponding output file test-382.html should also be renamed. I think this test should actually be integrated into testStructure1.xml, which already checks TOC-building.

I also think we should add some more detailed documentation that would help to avoid the proliferation of anonymous and mysterious test files in the future, as well as helping people to debug issues caught by the tests.

Is anyone else aware of anything missing from Test2?

Cheers, Martin

...

On 2023-10-18 05:18, Elisa Beshero-Bondar wrote: I like solution 3 and “Legacy_Tests” as well, but only if we are sure we’re ready to demote the original test as legacy. Does Test2 cover everything we care about now? Are we missing any tests there that matter and that the old tests cover? Elisa Elisa Beshero-Bondar, PhD (she/they) Chair, TEI Technical Council Program Chair of Digital Media, Arts, and Technology | Professor of Digital Humanities | Director of the Digital Humanities Lab at Penn State Erie, The Behrend College Typeset by hand on my iPhone

...

...

On Oct 17, 2023, at 5:42 PM, Bauman, Syd wrote:

 Oooh. I like the idea of Test/ → Legacy_Test/ and Test2/ → Test/.

---

...

...

...

I lean towards 3), though perhaps Test/ can be kept with a new name,

such as additional_tests or... Test2/

...

Trish & I are doing the p5subset → Stylesheets process. Most of our time has been spent handling Test/. I think it is just too cumbersome — the amount of time we (Council, collectively) spend wrangling with Test/ is likely by now much greater than the amount of time it saves by catching bugs early.

I think we need to do one of three things, toot sweet:

1. Re-work the Test/Makefile so that the process is /much/ easier: e.g., when comparisons are deferred, a shell script to perform them needs to be generated; this might be hard to impossible. 2. Stop using Test/, it is just not helpful enough to be worth it. Leave the directory in the repo so that we can use it when there is a particularly gnarly case, or when a human is not waiting for it, but remove it from the general test procedure we use, e.g. when generating p5subset for the Stylesheets repo. 3. Remove Test/ from the repo (and various instructions) entirely (and rename Test2/ to Test/ :-)

My vote is for (2). I could be perhaps be talked into (3) reasonably easily; it would take some serious convincing to get me to vote for (1).

P.S. BTW, Martin & I intended that we would do (3) when we developed Test2/. (I wanted to hang onto Test/, i.e. do (2), for awhile to ensure that everything tested by Test/ was also tested by Test2/.)

_______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council

_______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council

-- ------------------------------------------ Martin Holmes UVic Humanities Computing and Media Centre

I acknowledge and respect the lək̓ʷəŋən peoples on whose traditional territory the university stands and the Songhees, Esquimalt and WSÁNEĆ peoples whose historical relationships with the land continue to this day. _______________________________________________ Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council

---

Tei-council mailing list Tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council