Standarize the notation of subtests in TAP

@Leont

This issue was originally reported in TestAnything/testanything.github.io#20, and the conversation has been moved here since it is related to the TAP specification.

It concerns the subtests in TAP and a formal standard to be followed by TAP consumer and producers. In the latest TAP specification (TAP 13) there is no formal definition of subtests. There were proposals and draft documents on this, some tools like Test::More and tap4j support with limitations.

Please feel free to chime in and suggest enhancements for this standard. A message is being written to the mailing list pointing to this issue too. The mailing list has several users that participated in the IETF draft and have lots of experience with TAP and its implementations.

/cc @Leont @xmojmr

One thing that did not seem to be mentioned on the TAP mailing list was that the potential for a sub test to look like this:

ok 1 - something
ok 1 -     something
ok 2 - something

or

ok 1 - something
ok 1 -- something
ok 2 - something

I believe most libraries would support a failure just fine for both of those, its simpler that requiring an update to the libraries and forces stricter reasoning of tests which is a bonus.

We really want to have this. I'm not fond of the current mechanism, but given it's out there and there's a lot of stuff outputting it already it may be a passed station.

I believe most libraries would support a failure just fine for both of those, its simpler that requiring an update to the libraries and forces stricter reasoning of tests which is a bonus.

Both parsers I've worked on would consider those outputs an error. Subtests really should be something legacy parsers consider comments, which fortunately still gives a lot of space.

@jonathanKingston

@jonathanKingston I'm happy with the old proposal for subtests

TAP 13
1..1
ok 1 - Subtest 1
    ok 1 - subtest 2a
    ok 2 - subtest 2b

I'd be fine with other formats, but since there are tools in Perl, Java, Node, which output something similar to that, I'd be more keen to simply formalize it in a TAP 14 specification.

@Leont

We really want to have this. I'm not fond of the current mechanism, but given it's out there and there's a lot of stuff outputting it already it may be a passed station.

+1 I believe we have the formal TAP 13 specification, and a meta TAP 13.5, or TAP 13 + proposals/custom code. Considering the number of users in different programming languages outputting subtests, I think a proposal similar to what most tools produce (and that got consensus here and in the mailing list) would be a good approach.

As a side note, there is an open issue in tap4j where the TAP producer emits the subtests first. The current state machine in tap4j is not able to consume a subtest before. Quoting the example in that issue below:

    ok 1 - subtest 1a
    ok 2 - subtest 1b
    1..2
ok 1 - Subtest 1
    ok 1 - subtest 2a
    ok 2 - subtest 2b
    1..2
ok 2 - Subtest 2
1..2
ok

A perl test that will produce this output is as follows:

use Test::More;

subtest 'Subtest 1' => sub {
    ok(1,'subtest 1a');
    ok(1,'subtest 1b');
};
subtest 'Subtest 2' => sub {
    ok(1,'subtest 2a');
    ok(1,'subtest 2b');
};
done_testing();

It'd be nice to have the specification in cases like this to guide the developer :-)

We should not be afraid to break old implementations and if anything that should be the correct behaviour. Old implementations should not ever ignore sub tests. This is a testing setup, ignoring tests would be a mistake. Due to the non standard nature of all the current use of subtests it is possible subtests could fail and the parent succeed - this should be caught or be a parse error; either for me is fine.
This brings me to another ticket I need to make about how TAP software should respond if it can't be parsed.

I'm a little against looking to satisfy older implementations which we might end up calling 13.5 erroneously. This is a similar situation to what Markdown has been through and where possible I would prefer to go for breaking behaviour and errors will be spotted sooner rather than skipped over.

This was my theory around the formats above which should be treated as safe in most parsers where extra white space would be ignored around the comment.

I would therefore aim for:

False negative changes only
Try not to risk being interpreted by any parser as YAML
Discover the most TAP consumers which can cope with the two points above

I'm actually a fan of the tap4j as it would aid streaming output of the tests before they have completed running.

The other issue with the old proposal was that there is no acknowledgement of sub test count which I think is an issue.

Do sub tests count as the main count?
Sub tests counts help consumers fail earlier from a streaming output of tests

One last point, the number of main consumers and producers is still actually small, patching all of them to support a newer version shouldn't be a large piece of work.

We should not be afraid to break old implementations and if anything that should be the correct behaviour. Old implementations should not ever ignore sub tests. This is a testing setup, ignoring tests would be a mistake.

The way it was designed means that even if the subtests are ignored the correct conclusion can still be made. It leads to a data-loss, not not to incorrect operation.

Due to the non standard nature of all the current use of subtests it is possible subtests could fail and the parent succeed - this should be caught or be a parse error; either for me is fine.

In the parser I wrote a discrepancy generates a non-fatal parse error.

I'm a little against looking to satisfy older implementations which we might end up calling 13.5 erroneously. This is a similar situation to what Markdown has been through and where possible I would prefer to go for breaking behaviour and errors will be spotted sooner rather than skipped over.

Do we really have that problem?

This was my theory around the formats above which should be treated as safe in most parsers where extra white space would be ignored around the comment.

I don't follow you. They don't have extra whitespace at the start of the line, so they won't be ignored even on legacy parsers.

@kinow

The point I am making is we are expecting the producer to be error free by always returning a not ok parent test. I would prefer if both producer and consumer were more fragile rather than the 'Just cope' attitude the philosophy suggests.

This was my theory around the formats above which should be treated as safe in most parsers where extra white space would be ignored around the comment.

I don't follow you. They don't have extra whitespace at the start of the line, so they won't be ignored even on legacy parsers.

That is my point, why would you want them ignored. I want a testing framework to do one of two things when it doesn't know what to do:

Fatal error
Add a parse error as if it were a failed test it encountered

Older parsers may or may not be able to cope with invalid subtests. The following will not fail with some of the parsers I have seen:

ok 1 - test
  not ok 1 - test
ok 2 - test

However both these would:

ok 1 - test
not ok 1 -    test
ok 2 - test

and

ok 1 - test
not ok 1 -- test
ok 2 - test

The way it was designed means that even if the subtests are ignored the correct conclusion can still be made. It leads to a data-loss, not not to incorrect operation.

This is unlikely to be true for all producers but I take your point that the parent test should fail however I would prefer the correct behaviour for parsers is that they fail.

In the parser I wrote a discrepancy generates a non-fatal parse error.

This is unlikely to be true for all consumers, in fact the TAP philosophy (which likely needs addressing) suggests any such issue should be skipped.

The only way I would support this with the subtests being before like @kinow mentioned. Even then the format isn't as easy to read.

Older parsers may or may not be able to cope with invalid subtests. The following will not fail with some of the parsers I have seen:

Even as OKs they would fail because the numbering is incorrect.

This is unlikely to be true for all producers but I take your point that the parent test should fail however I would prefer the correct behaviour for parsers is that they fail.

Agreed. Parsing only the parent test should clearly be considered legacy behavior, but legacy has its uses.

This is unlikely to be true for all consumers, in fact the TAP philosophy (which likely needs addressing) suggests any such issue should be skipped.

My take on the philosophy would be: "report errors, but when possibly make them non-fatal", as that way the user can be presented with as much information as possible.

Very true however subtests could be part of the same numbering potentially.

Is there other useful meta data that subtests have?

Yeah I will raise an issue also with that as I certainly don't think it is clear enough.

@Ovid

@Ovid I saw some interesting comments on the mailing list of yours. (http://www.ietf.org/mail-archive/web/tap/current/msg00530.html) Would you be able to mention your concerns here?

I believe a way of demarcating subtests is needed and it should satisfy these conditions:

Subtests be tested by older parsers as if it was a parent or fail in most cases
YAML doesn't become a nightmare to parse
Continue to be as human readable as possible
Allow for test numbering
- Allow for sub test counts 1..20
- Allow for sub test numbering 1.1.1 and 1

Allow for sub test numbering 1.1.1 and 1

I'm not sure what you mean with that exactly.

ok 1 - something
ok 1.1 -- something
ok 1.1.1 --- something
ok 2 - something

and

ok 1 - something
ok 1 -- something
ok 1 --- something
ok 2 - something

Obviously numbering itself should be optional also.

Node-tap uses a 2-space indented yaml blocks delimited by --- and ..., which are always attached to the previous test point.

The problem is, that means you either have two separate events (which is undesirable if you want to process it in a sensible way; together), or you have to wait for the next line until you can emit the most recent test result (which is undesirable if you want to show updates as they come in). We need some kind of marker of there's more coming up if we don't want to lose TAP's streaming nature.

The problem is, that means you either have two separate events (which is undesirable if you want to process it in a sensible way; together)

Great suggestion for TAP15 ;)

I mean, having written parsers that have to handle this and maintaining node-tap for 11 years now, I can say yes, you're 100% right, that sucks. And also, it's the kind of thing you do once, and isn't really that hard from then on.

But we all do this now, undesirable or not, and there's value in having a spec that is minimal, descriptive, and accurate, at least to start. Then there'll be a firmer foundation upon which to build new ideas, which has been the biggest obstacle to progress in this space thus far.

Great suggestion for TAP15 ;)

Fair enough

@exodist

@exodist @mblayman @Leont Please go take a look: #25

For the yaml blocks, under a subtest would a yaml block have 6 space indentation for the --- and ...? To indicate the yaml block belongs to an event inside the subtest?

@exodist

@exodist Yes, when we add child tests to this specification, we'll have to specify that the YAML indentation is 2 spaces more than the indentation of the test point to which it belongs. Similarly, a child test inside of a child test would be indented 8 spaces (4 + the parent), one inside of that would be 12, a yaml block for the 12-space indented test point would be 14 spaces, etc.

If subtests will be indented, then they should use the more accessible tabs¹²³⁴, because unlike with spaces, it’s exactly one tab per level.

That said, I personally prefer the 1.2.3 format combined with including the supertest name(s) delimited with >.

@ExE-Boss

@ExE-Boss Current implementations use 4 spaces for indenting child tests. The goal of this exercise is to describe what is, so that we can make sensible progress towards what ought to be. Or we spend another 7 years debating everything and going nowhere.

I should point out that the Test2::Harness TAP parser actually does not require 4 space indentation. Any indentation at all with proper TAP indented counts, and it considers a change in indentation one way or the other to indicate an additional subtest or and end of a subtest. it uses perls regex '\s' wildcard to capture space, that includes tabs or spaces.

I am not likely to change this any time soon, it does work for the 4-space standard this wants to set, but it is more relaxed on what it accepts.

@exodist

@exodist Parsers rarely get in trouble by following Postel's rule, I think that's fine. ;) That being said, though, when we get to that part, we should specify "Harnesses may accept any level of indentation / should only parse subtests indented exactly 4 spaces", but also "producers must indent subtests 4 spaces", or something like that.

@exodist

@exodist I'm curious what would it do with something like this?

TAP version 13
1..2
    1..1
    ok
  1..1
  ok
ok
ok

I seem to recall edge cases like that are why I flipped the table and just said it had to be 4 always in node-tap.

The way the parser works last I modified it (a while ago to be sure) was that it had an array indexed by indentation level, every time indentation gets deeper it adds one to the right element (4 for 4 spaces, 3 for 3 spaces, etc), every time it gets more shallow it will close out all deeper subtests until the shallow level is closed. So it would handle this case just fine, and I believe I may even have a unit test for this specific edge case.

Something I am less sure of from memory is if my parser considers a tab to be 4 spaces, or a single space (IE does it convert tabs to spaces, or does it just count it as a single space character?) I have to look into that.

I should also say my parser is a couple-layered thing.
https://metacpan.org/dist/Test2-Harness/source/lib/Test2/Harness/Collector/TapParser.pm parses line at a time, and returns what it parsed as well as the indentation. This part of the parser does not care about subtests, it is simply "What does this TAP line say, and how much indentation came first)
This basically turns the TAP stream into the harnesses internal event stream format which then goes to https://metacpan.org/dist/Test2-Harness/source/lib/Test2/Harness/Auditor/Watcher.pm which does the work of validating the event stream, including verifying subtests and indentations are all sane.

Looking at this again I see the "current" implementation does not actually index into an array, I have a feature branch somewhere that does, but this is still set up such that it should handle your edge case.

Anyway, the point is that I did not find it difficult to handle edge cases like yours, and I found that the parsing is much easier if the lines are first parsed into objects, then a separate watcher handles the object stream, the watcher then spawns a child-watcher for each subtest, etc. This simplifies the process greatly..

oh, also my parser ignores the YAML stuff, but if we get it standardized I can add it in.

@Leont

Anyway, the point is that I did not find it difficult to handle edge cases like yours, and I found that the parsing is much easier if the lines are first parsed into objects, then a separate watcher handles the object stream, the watcher then spawns a child-watcher for each subtest, etc. This simplifies the process greatly..

Yeah, I can see how that'd simplify things. http://npm.im/tap-parser creates a child Parser object for each indented block, and then when it sees a test point at the parent level, it closes out that child and continues on with the parent, iirc. It was also a while ago since I touched it last though, I could be completely wrong 😅 It does have to not emit the test point until the next line, in case it's yaml, which is sometimes a bit weird (that case @Leont is talking about).

I don't think we'd need diagnostics to come before the test point, but it'd be nice if there was a marker on the test point line to indicate whether yaml is coming or not. Something like:

TAP version 15
1..2
not ok 1 - this has more info # diagnostics
  ---
  found: 1
  wanted: 2
  ...
ok 2

Then it could be a little less stateful and still fully interruptible.

Navigation Menu

Repository navigation

Standarize the notation of subtests in TAP #2

61 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Standarize the notation of subtests in TAP #2

Description

Activity

jonathanKingston commented on Jan 6, 2015

Leont commented on Jan 6, 2015

Leont commented on Jan 6, 2015

kinow commented on Jan 6, 2015

jonathanKingston commented on Jan 6, 2015

Leont commented on Jan 7, 2015

jonathanKingston commented on Jan 7, 2015

Leont commented on Jan 9, 2015

jonathanKingston commented on Jan 11, 2015

jonathanKingston commented on Jan 14, 2015

Leont commented on Jan 15, 2015

jonathanKingston commented on Jan 18, 2015

61 remaining items

Leont commented on Mar 1, 2022

isaacs commented on Mar 1, 2022

Leont commented on Mar 1, 2022

isaacs commented on Mar 1, 2022

exodist commented on Mar 1, 2022

isaacs commented on Mar 1, 2022

ExE-Boss commented on Mar 1, 2022

Footnotes

isaacs commented on Mar 1, 2022

exodist commented on Mar 1, 2022

isaacs commented on Mar 1, 2022

isaacs commented on Mar 1, 2022

exodist commented on Mar 2, 2022

exodist commented on Mar 2, 2022

exodist commented on Mar 2, 2022

isaacs commented on Mar 2, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions