This page runs the checks that the Checked block of the
Evaluation chapter claims: the shapes S1 to S8
over the measles evaluation, M2 to M5 over the model graph, thirty-three
counterexamples over the record and two model counterexamples that must fail. A reader need
not take the chapter’s word for it. Every line of output below is produced by
the code above it, the gate re-executes this notebook and fails if the
committed outputs differ, and each claim is an assert that stops the
notebook when it does not hold.
import sys; sys.path[:0] = ["notebooks", "."] # the shared module lives beside this notebook
import checked
# "S1 to S8" in the chapter's Checked block names every shape of those
# steps in shapes/epo.shapes.ttl; they are listed here one by one so that a
# renamed or removed shape raises an error instead of passing silently.
RECORD_SHAPES = [
"S1-DsoRelease",
"S2-RequirementSet", "S2-Requirement", "S2-AcceptanceCriterion",
"S3-TestPlan", "S3-PlanApproval", "S3-PlanDeviation", "S3-Probe", "S3-Strategy",
"S4-Session", "S4-Turn", "S4-TestSuite",
"S5-Response", "S5-Evidence",
"S6-Determination", "S6-Attestation",
"S7-Report", "S7-ConformanceVerdict", "S7-CoverageComputation", "S7-ReportApproval",
"S8-Recommendation", "S8-Delivery",
]
# "M2 to M5" likewise names every shape of those groups in shapes/model.shapes.ttl.
MODEL_SHAPES = [
"M2-Wire", "M2-Part",
"M3-RecordHub",
"M4-Steps", "M4-Nesting", "M4-ProcessDag", "M4-EveryItemProduced",
"M5-Roles", "M5-Cardinality", "M5-PortsBelongToRoles",
]1The record conforms to S1 to S8¶
The record is track/measles-evaluation.ttl with the EPO vocabulary and the
model graph, loaded as the test suite loads it: one default graph, since the
shapes derive an item’s step through the model (sheet 10-33). The shapes
graph holds exactly the twenty-two named shapes copied from
shapes/epo.shapes.ttl, one line each below. The chapter’s sentences map
onto them: the DSO release approved before any probe (S1); the requirement
set, its requirements and criteria, approved by the sponsor’s signatory
(S2); the plan approved and consistent, every departure recorded (S3);
sessions against the one test item the envelope binds, turns and suites
(S4); responses and evidence (S5); determinations and attestations under
the judgment rules (S6); the verdict on the record before the final report
and coverage from attestations alone (S7); the recommendation and the
delivery traced (S8).
record = checked.record()
S = checked.shapes("shapes/epo.shapes.ttl", RECORD_SHAPES)
conforms, fired = checked.report("track/measles-evaluation.ttl", record, S)
assert conforms and not fired
checked.passed("the measles evaluation conforms to the twenty-two S1 to S8 shapes")track/measles-evaluation.ttl: conforms = True
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
ok: the measles evaluation conforms to the twenty-two S1 to S8 shapes
2Thirty-three counterexamples over the record¶
Each is the measles evaluation with one change, written by
scripts/render_counterexamples.py with the fault in its header (sheet
10-19). The files under counterexamples/ and the shape each must fail
among the twenty-two here, as tests/test_shacl.py pins them (a fault that
ripples to a shape outside S1 to S8, such as S0-Independence or
S9-Acceptance, is pinned there too, and printed, not hidden):
| The fault | File | Must fail |
|---|---|---|
| an attestation without a determination behind it, so the recommendation reaches no evidence through it | attestation-without-evidence.ttl | S6-Attestation, S8-Recommendation |
| one aggregating a determination for another criterion | attestation-off-plan.ttl | S6-Attestation |
| the same fault across two turns of a strategy-driven session | attestation-off-turn.ttl | S6-Attestation |
| a probe derived before the requirements | probe-before-requirements.ttl | S2-RequirementSet |
| a recommendation resting on nothing | recommendation-untraced.ttl | S8-Recommendation |
| a session run by the domain expert | expert-administers-tests.ttl | S4-Session |
| an attestation by the authorized representative | executive-attests.ttl | S6-Attestation |
| one before its determination | attestation-before-determination.ttl | S6-Attestation |
| a probe checked after its first turn | consistency-check-after-turn.ttl | S3-Probe |
| a plan approved after the session | plan-approved-after-session.ttl | S3-PlanApproval |
| a delivery before its approval | delivery-before-approval.ttl | S8-Delivery |
| a misstated coverage | report-coverage-misstated.ttl | S7-Report |
| padded rates | report-rates-padded.ttl | S7-Report |
| an interview after the DSO release | dso-before-stakeholder-input.ttl | S1-DsoRelease |
| a superseding attestation that drops the judgment it supersedes without saying why (sheet 10-14) | cherry-picked-determination.ttl | S6-Attestation |
| failed on insufficient evidence (sheet 10-12) | insufficient-yet-failed.ttl | S6-Attestation |
| a requirement set the sponsor never approved (sheet 10-01) | requirement-set-unapproved.ttl | S2-RequirementSet |
| a criterion left out with no deviation recorded (sheet 10-16), the stored coverage then wrong | deviation-unrecorded.ttl | S3-TestPlan, S7-Report |
| a deviation the domain expert wrote | deviation-by-expert.ttl | S3-PlanDeviation |
| a session against another build (sheet 10-17), its responses then from an agent it did not run against | session-on-another-item.ttl | S4-Session, S5-Response |
| a final report without a verdict (sheet 10-41) | final-report-without-verdict.ttl | S7-Report |
| a verdict naming no digests (sheet 10-18) | verdict-without-digests.ttl | S7-ConformanceVerdict |
| a coverage computation naming no digests (sheet 10-18) | coverage-without-digests.ttl | S7-CoverageComputation |
| a recommendation nobody approved (sheet 10-11) | recommendation-unapproved.ttl | S8-Recommendation |
| a draft without its gap flagged (sheet 10-41) | draft-without-gaps.ttl | S7-Report |
| a draft storing the final report’s rates (sheet 10-48) | draft-rates-from-later.ttl | S7-Report |
| an approval of the draft, not the final report (sheet 10-41) | draft-approved.ttl | S7-ReportApproval |
| a recommendation fit to deploy over a failed criterion (sheet 10-48) | fit-despite-failure.ttl | S8-Recommendation |
| a DSO approval with no date (drift pass 4) | dso-approval-undated.ttl | S1-DsoRelease |
| an appropriateness assessment after the sessions (drift pass 4) | assessment-after-session.ttl | S2-RequirementSet |
| a session run after the access period ended (sheet 10-04; drift pass 4) | session-outside-access-period.ttl | S4-Session |
| a response from another build (drift pass 4) | response-from-another-build.ttl | S5-Response |
| an attestation on another item (sheet 10-17; drift pass 4) | attestation-on-another-item.ttl | S6-Attestation |
Each is run against the same twenty-two shapes. The assert in each cell says that the file does not conform, that the named shape fires, and that no other of the twenty-two does.
cx = checked.counterexample("attestation-without-evidence.ttl")
conforms, fired = checked.report("counterexamples/attestation-without-evidence.ttl", cx, S)
assert not conforms and set(fired) == {"S6-Attestation", "S8-Recommendation"}
checked.passed("attestation-without-evidence.ttl fails S6-Attestation and, through it, S8-Recommendation, and no other of the twenty-two")counterexamples/attestation-without-evidence.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation FAIL
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation FAIL
S6-Attestation at ev:attestation-1:
S6 closure rule: an attestation with outcome passed or failed must aggregate at least one determination that rules on evidence; with none it can only say cantTell.
S6-Attestation at ev:attestation-1:
S6 no cherry-picking (sheet 10-14): an attestation uses every determination of the record on its criterion dated no later than it, or names the determination it left out with epo:excludes and gives its epo:reason.
S8-Recommendation at ev:recommendation-1:
S8: every attestation a recommendation rests on must itself rest on evidence the recommendation can reach.
ok: attestation-without-evidence.ttl fails S6-Attestation and, through it, S8-Recommendation, and no other of the twenty-two
cx = checked.counterexample("attestation-off-plan.ttl")
conforms, fired = checked.report("counterexamples/attestation-off-plan.ttl", cx, S)
assert not conforms and set(fired) == {"S6-Attestation"}
checked.passed("attestation-off-plan.ttl fails S6-Attestation and no other of the twenty-two")counterexamples/attestation-off-plan.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation FAIL
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S6-Attestation at ev:attestation-1:
S6 chain rule (R-12, R-18, R-20): every determination an attestation aggregates tests the attested criterion, and every evidence item it rules on bears on that criterion under a plan that has it as an objective.
ok: attestation-off-plan.ttl fails S6-Attestation and no other of the twenty-two
cx = checked.counterexample("attestation-off-turn.ttl")
conforms, fired = checked.report("counterexamples/attestation-off-turn.ttl", cx, S)
assert not conforms and set(fired) == {"S6-Attestation"}
checked.passed("attestation-off-turn.ttl fails S6-Attestation and no other of the twenty-two")counterexamples/attestation-off-turn.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation FAIL
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S6-Attestation at ev:attestation-2:
S6 chain rule (R-12, R-18, R-20): every determination an attestation aggregates tests the attested criterion, and every evidence item it rules on bears on that criterion under a plan that has it as an objective.
ok: attestation-off-turn.ttl fails S6-Attestation and no other of the twenty-two
cx = checked.counterexample("probe-before-requirements.ttl")
conforms, fired = checked.report("counterexamples/probe-before-requirements.ttl", cx, S)
assert not conforms and set(fired) == {"S2-RequirementSet"}
checked.passed("probe-before-requirements.ttl fails S2-RequirementSet and no other of the twenty-two")counterexamples/probe-before-requirements.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet FAIL
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S2-RequirementSet at ev:requirement-set:
S2: a probe run started, or a probe was generated, before the requirement set was declared.
ok: probe-before-requirements.ttl fails S2-RequirementSet and no other of the twenty-two
cx = checked.counterexample("recommendation-untraced.ttl")
conforms, fired = checked.report("counterexamples/recommendation-untraced.ttl", cx, S)
assert not conforms and set(fired) == {"S8-Recommendation"}
checked.passed("recommendation-untraced.ttl fails S8-Recommendation and no other of the twenty-two")counterexamples/recommendation-untraced.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation FAIL
S8-Recommendation at ev:recommendation-1:
S8 no cherry-picking (sheet 10-11): a recommendation derives from every attestation the record's coverage computation used, or names the attestation it left out with epo:excludes and gives its epo:reason.
S8-Recommendation at ev:recommendation-1:
S8: a recommendation must derive from at least one attestation.
ok: recommendation-untraced.ttl fails S8-Recommendation and no other of the twenty-two
cx = checked.counterexample("expert-administers-tests.ttl")
conforms, fired = checked.report("counterexamples/expert-administers-tests.ttl", cx, S)
assert not conforms and set(fired) == {"S4-Session"}
checked.passed("expert-administers-tests.ttl fails S4-Session and no other of the twenty-two")counterexamples/expert-administers-tests.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session FAIL
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S4-Session at ev:session-1:
S4: a session must be associated with a named person in the evaluation operator role; administering tests is the operator's activity, not the domain expert's (R-10, R-23).
ok: expert-administers-tests.ttl fails S4-Session and no other of the twenty-two
cx = checked.counterexample("executive-attests.ttl")
conforms, fired = checked.report("counterexamples/executive-attests.ttl", cx, S)
assert not conforms and set(fired) == {"S6-Attestation"}
checked.passed("executive-attests.ttl fails S6-Attestation and no other of the twenty-two")counterexamples/executive-attests.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation FAIL
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S6-Attestation at ev:attestation-1:
S6: the attesting person holds the domain expert role; the operator collects and determines, the authorized representative signs and delivers, only the domain expert attests (R-21, R-23).
ok: executive-attests.ttl fails S6-Attestation and no other of the twenty-two
cx = checked.counterexample("attestation-before-determination.ttl")
conforms, fired = checked.report("counterexamples/attestation-before-determination.ttl", cx, S)
assert not conforms and set(fired) == {"S6-Attestation"}
checked.passed("attestation-before-determination.ttl fails S6-Attestation and no other of the twenty-two")counterexamples/attestation-before-determination.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation FAIL
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S6-Attestation at ev:attestation-1:
S6 order (sheet 10-19): an attestation is dated no earlier than every determination it aggregates.
ok: attestation-before-determination.ttl fails S6-Attestation and no other of the twenty-two
cx = checked.counterexample("consistency-check-after-turn.ttl")
conforms, fired = checked.report("counterexamples/consistency-check-after-turn.ttl", cx, S)
assert not conforms and set(fired) == {"S3-Probe"}
checked.passed("consistency-check-after-turn.ttl fails S3-Probe and no other of the twenty-two")counterexamples/consistency-check-after-turn.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe FAIL
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S3-Probe at ev:probe-1:
S3 order (sheet 10-19): the probe's passed consistency check is dated before the first turn that used the probe started.
ok: consistency-check-after-turn.ttl fails S3-Probe and no other of the twenty-two
cx = checked.counterexample("plan-approved-after-session.ttl")
conforms, fired = checked.report("counterexamples/plan-approved-after-session.ttl", cx, S)
assert not conforms and set(fired) == {"S3-PlanApproval"}
checked.passed("plan-approved-after-session.ttl fails S3-PlanApproval and no other of the twenty-two")counterexamples/plan-approved-after-session.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval FAIL
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S3-PlanApproval at ev:plan-approval-1:
S3 order (sheet 10-19): the plan is approved before any session whose turns use the plan's probes starts.
ok: plan-approved-after-session.ttl fails S3-PlanApproval and no other of the twenty-two
cx = checked.counterexample("delivery-before-approval.ttl")
conforms, fired = checked.report("counterexamples/delivery-before-approval.ttl", cx, S)
assert not conforms and set(fired) == {"S8-Delivery"}
checked.passed("delivery-before-approval.ttl fails S8-Delivery and no other of the twenty-two")counterexamples/delivery-before-approval.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery FAIL
S8-Recommendation pass
S8-Delivery at ev:delivery-1:
S8 order (sheet 10-19): the delivery is made no earlier than the report approval it carries is dated.
ok: delivery-before-approval.ttl fails S8-Delivery and no other of the twenty-two
cx = checked.counterexample("report-coverage-misstated.ttl")
conforms, fired = checked.report("counterexamples/report-coverage-misstated.ttl", cx, S)
assert not conforms and set(fired) == {"S7-Report"}
checked.passed("report-coverage-misstated.ttl fails S7-Report and no other of the twenty-two")counterexamples/report-coverage-misstated.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report FAIL
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S7-Report at ev:report:
S7: the stored coverage equals the coverage recomputed from the record as it stood when the report was generated: the weighted share of acceptance criteria that carry at least one attestation with an outcome, dated no later than the report (walkthrough B5, R-48; a draft is a snapshot, sheet 10-48).
ok: report-coverage-misstated.ttl fails S7-Report and no other of the twenty-two
cx = checked.counterexample("report-rates-padded.ttl")
conforms, fired = checked.report("counterexamples/report-rates-padded.ttl", cx, S)
assert not conforms and set(fired) == {"S7-Report"}
checked.passed("report-rates-padded.ttl fails S7-Report and no other of the twenty-two")counterexamples/report-rates-padded.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report FAIL
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S7-Report at ev:report:
S7 (SCI-08, sheet 10-19): the stored pass, fail and cannot-tell rates equal the rates recomputed from the record's attestations dated no later than the report was generated, over the covered criteria, as queries/coverage.rq computes them (a criterion counts once: failed if any attestation failed, else passed if any passed, else cannot tell); a final report has every attestation behind it, a draft the ones of its day.
ok: report-rates-padded.ttl fails S7-Report and no other of the twenty-two
cx = checked.counterexample("dso-before-stakeholder-input.ttl")
conforms, fired = checked.report("counterexamples/dso-before-stakeholder-input.ttl", cx, S)
assert not conforms and set(fired) == {"S1-DsoRelease"}
checked.passed("dso-before-stakeholder-input.ttl fails S1-DsoRelease and no other of the twenty-two")counterexamples/dso-before-stakeholder-input.ttl: conforms = False
S1-DsoRelease FAIL
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S1-DsoRelease at ev:dso-apollo-sv-r1:
S1 precondition (sheet 08, R-49): every affected population's representation is available before the DSO release is approved: the population's stakeholder representation was generated before the release's approval time, epo:approvedAt (a population with no representation at all is S0's finding).
ok: dso-before-stakeholder-input.ttl fails S1-DsoRelease and no other of the twenty-two
cx = checked.counterexample("cherry-picked-determination.ttl")
conforms, fired = checked.report("counterexamples/cherry-picked-determination.ttl", cx, S)
assert not conforms and set(fired) == {"S6-Attestation"}
checked.passed("cherry-picked-determination.ttl fails S6-Attestation and no other of the twenty-two")counterexamples/cherry-picked-determination.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation FAIL
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S6-Attestation at ev:attestation-6:
S6 no cherry-picking (sheet 10-14): an attestation uses every determination of the record on its criterion dated no later than it, or names the determination it left out with epo:excludes and gives its epo:reason.
ok: cherry-picked-determination.ttl fails S6-Attestation and no other of the twenty-two
cx = checked.counterexample("insufficient-yet-failed.ttl")
conforms, fired = checked.report("counterexamples/insufficient-yet-failed.ttl", cx, S)
assert not conforms and set(fired) == {"S6-Attestation"}
checked.passed("insufficient-yet-failed.ttl fails S6-Attestation and no other of the twenty-two")counterexamples/insufficient-yet-failed.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation FAIL
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S6-Attestation at ev:attestation-3:
S6 judgment rules (sheet 10-12): passed or failed requires sufficient evidence; insufficient evidence implies cannot tell; an inappropriate context allows only cannot tell or inapplicable.
ok: insufficient-yet-failed.ttl fails S6-Attestation and no other of the twenty-two
cx = checked.counterexample("requirement-set-unapproved.ttl")
conforms, fired = checked.report("counterexamples/requirement-set-unapproved.ttl", cx, S)
assert not conforms and set(fired) == {"S2-RequirementSet"}
checked.passed("requirement-set-unapproved.ttl fails S2-RequirementSet and no other of the twenty-two")counterexamples/requirement-set-unapproved.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet FAIL
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S2-RequirementSet at ev:requirement-set:
S2: the sponsor approves the requirement set (sheet 10-01): an approval attributed to the sponsor's signatory, dated after the set was declared and before any session of the record started.
ok: requirement-set-unapproved.ttl fails S2-RequirementSet and no other of the twenty-two
cx = checked.counterexample("deviation-unrecorded.ttl")
conforms, fired = checked.report("counterexamples/deviation-unrecorded.ttl", cx, S)
assert not conforms and set(fired) == {"S3-TestPlan", "S7-Report"}
checked.passed("deviation-unrecorded.ttl fails S3-TestPlan and, the coverage no longer supported, S7-Report, and no other of the twenty-two")
counterexamples/deviation-unrecorded.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan FAIL
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report FAIL
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S3-TestPlan at ev:test-plan:
S3 completion (sheet 10-16): every acceptance criterion of the record the plan leaves out, and every objective of the plan without an attestation, has a plan deviation with its reason that deviates from this plan and concerns that criterion.
S7-Report at ev:report:
S7: the stored coverage equals the coverage recomputed from the record as it stood when the report was generated: the weighted share of acceptance criteria that carry at least one attestation with an outcome, dated no later than the report (walkthrough B5, R-48; a draft is a snapshot, sheet 10-48).
S7-Report at ev:report-draft:
S7: the stored coverage equals the coverage recomputed from the record as it stood when the report was generated: the weighted share of acceptance criteria that carry at least one attestation with an outcome, dated no later than the report (walkthrough B5, R-48; a draft is a snapshot, sheet 10-48).
ok: deviation-unrecorded.ttl fails S3-TestPlan and, the coverage no longer supported, S7-Report, and no other of the twenty-two
cx = checked.counterexample("deviation-by-expert.ttl")
conforms, fired = checked.report("counterexamples/deviation-by-expert.ttl", cx, S)
assert not conforms and set(fired) == {"S3-PlanDeviation"}
checked.passed("deviation-by-expert.ttl fails S3-PlanDeviation and no other of the twenty-two")counterexamples/deviation-by-expert.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation FAIL
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S3-PlanDeviation at ev:plan-deviation-1:
S3: a plan deviation is the evaluation operator's, and is an item of the plan step or the report step (the step is derived through the model graph, sheet 10-33).
ok: deviation-by-expert.ttl fails S3-PlanDeviation and no other of the twenty-two
cx = checked.counterexample("session-on-another-item.ttl")
conforms, fired = checked.report("counterexamples/session-on-another-item.ttl", cx, S)
assert not conforms and set(fired) == {"S4-Session", "S5-Response"}
checked.passed("session-on-another-item.ttl fails S4-Session and, through its responses, S5-Response, and no other of the twenty-two")counterexamples/session-on-another-item.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session FAIL
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response FAIL
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S4-Session at ev:session-1:
S4 binding (sheet 10-17): the session's software agent is the system under test the record's requirement set binds, and every determination on evidence from this session names that agent as its subject.
S5-Response at ev:response-1:
S5 (drift pass 4, QA 10): the software agent a response is attributed to is the one its session ran against, the agent the session is associated with, reached through the turn that generated the response; a response from another build is not evidence about this test item.
S5-Response at ev:response-2:
S5 (drift pass 4, QA 10): the software agent a response is attributed to is the one its session ran against, the agent the session is associated with, reached through the turn that generated the response; a response from another build is not evidence about this test item.
S5-Response at ev:response-3:
S5 (drift pass 4, QA 10): the software agent a response is attributed to is the one its session ran against, the agent the session is associated with, reached through the turn that generated the response; a response from another build is not evidence about this test item.
ok: session-on-another-item.ttl fails S4-Session and, through its responses, S5-Response, and no other of the twenty-two
cx = checked.counterexample("final-report-without-verdict.ttl")
conforms, fired = checked.report("counterexamples/final-report-without-verdict.ttl", cx, S)
assert not conforms and set(fired) == {"S7-Report"}
checked.passed("final-report-without-verdict.ttl fails S7-Report and no other of the twenty-two")counterexamples/final-report-without-verdict.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report FAIL
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S7-Report at ev:report:
S7 draft and final (sheet 10-41, Z: conformance is a prerequisite for compiling the final report; a draft may carry flagged gaps): a draft report flags its gaps; a final report derives from a passed conformance verdict on its record dated no later than the report was generated.
ok: final-report-without-verdict.ttl fails S7-Report and no other of the twenty-two
cx = checked.counterexample("verdict-without-digests.ttl")
conforms, fired = checked.report("counterexamples/verdict-without-digests.ttl", cx, S)
assert not conforms and set(fired) == {"S7-ConformanceVerdict"}
checked.passed("verdict-without-digests.ttl fails S7-ConformanceVerdict and no other of the twenty-two")counterexamples/verdict-without-digests.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict FAIL
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S7-ConformanceVerdict at ev:conformance-verdict-1:
The verdict names the ontology it ran by its sha256 digest (sheet 10-18).
S7-ConformanceVerdict at ev:conformance-verdict-1:
The verdict names the record it judged by the sha256 digest of its canonical member triples as they stood at the verdict (round four, KG 8); `ogc doctor` recomputes it.
S7-ConformanceVerdict at ev:conformance-verdict-1:
The verdict names the shapes it ran by their sha256 digest (tool qualification, sheet 10-18).
ok: verdict-without-digests.ttl fails S7-ConformanceVerdict and no other of the twenty-two
cx = checked.counterexample("coverage-without-digests.ttl")
conforms, fired = checked.report("counterexamples/coverage-without-digests.ttl", cx, S)
assert not conforms and set(fired) == {"S7-CoverageComputation"}
checked.passed("coverage-without-digests.ttl fails S7-CoverageComputation and no other of the twenty-two")counterexamples/coverage-without-digests.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation FAIL
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S7-CoverageComputation at ev:coverage-computation:
The coverage computation names the coverage query it ran by its sha256 digest (sheet 10-18; round four, KG 8: the query is the assembler's tool, not the checker's).
S7-CoverageComputation at ev:coverage-computation:
The coverage computation names the ontology it ran by its sha256 digest (sheet 10-18).
S7-CoverageComputation at ev:coverage-computation:
The coverage computation names the shapes it ran by their sha256 digest (sheet 10-18).
ok: coverage-without-digests.ttl fails S7-CoverageComputation and no other of the twenty-two
cx = checked.counterexample("recommendation-unapproved.ttl")
conforms, fired = checked.report("counterexamples/recommendation-unapproved.ttl", cx, S)
assert not conforms and set(fired) == {"S8-Recommendation"}
checked.passed("recommendation-unapproved.ttl fails S8-Recommendation and no other of the twenty-two")counterexamples/recommendation-unapproved.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation FAIL
S8-Recommendation at ev:recommendation-1:
S8: the domain expert's report approval owns the recommendation: it derives from a report approval (sheet 10-11).
ok: recommendation-unapproved.ttl fails S8-Recommendation and no other of the twenty-two
cx = checked.counterexample("draft-without-gaps.ttl")
conforms, fired = checked.report("counterexamples/draft-without-gaps.ttl", cx, S)
assert not conforms and set(fired) == {"S7-Report"}
checked.passed("draft-without-gaps.ttl fails S7-Report and no other of the twenty-two")
counterexamples/draft-without-gaps.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report FAIL
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S7-Report at ev:report-draft:
S7 draft and final (sheet 10-41, Z: conformance is a prerequisite for compiling the final report; a draft may carry flagged gaps): a draft report flags its gaps; a final report derives from a passed conformance verdict on its record dated no later than the report was generated.
ok: draft-without-gaps.ttl fails S7-Report and no other of the twenty-two
cx = checked.counterexample("draft-rates-from-later.ttl")
conforms, fired = checked.report("counterexamples/draft-rates-from-later.ttl", cx, S)
assert not conforms and set(fired) == {"S7-Report"}
checked.passed("draft-rates-from-later.ttl fails S7-Report and no other of the twenty-two")
counterexamples/draft-rates-from-later.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report FAIL
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S7-Report at ev:report-draft:
S7 (SCI-08, sheet 10-19): the stored pass, fail and cannot-tell rates equal the rates recomputed from the record's attestations dated no later than the report was generated, over the covered criteria, as queries/coverage.rq computes them (a criterion counts once: failed if any attestation failed, else passed if any passed, else cannot tell); a final report has every attestation behind it, a draft the ones of its day.
ok: draft-rates-from-later.ttl fails S7-Report and no other of the twenty-two
cx = checked.counterexample("draft-approved.ttl")
conforms, fired = checked.report("counterexamples/draft-approved.ttl", cx, S)
assert not conforms and set(fired) == {"S7-ReportApproval"}
checked.passed("draft-approved.ttl fails S7-ReportApproval and no other of the twenty-two (S9-Acceptance, outside them, is pinned by the tests)")
counterexamples/draft-approved.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval FAIL
S8-Delivery pass
S8-Recommendation pass
S7-ReportApproval at ev:report-approval-1:
S7: the final report's contents are approved by a domain expert, after a passed conformance verdict on the record: correct construction is not enough, and a draft is not approved (walkthrough, R-49; sheet 10-41).
ok: draft-approved.ttl fails S7-ReportApproval and no other of the twenty-two (S9-Acceptance, outside them, is pinned by the tests)
cx = checked.counterexample("fit-despite-failure.ttl")
conforms, fired = checked.report("counterexamples/fit-despite-failure.ttl", cx, S)
assert not conforms and set(fired) == {"S8-Recommendation"}
checked.passed("fit-despite-failure.ttl fails S8-Recommendation and no other of the twenty-two")
counterexamples/fit-despite-failure.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation FAIL
S8-Recommendation at ev:recommendation-1:
S8 (sheet 10-48): a recommendation that says fit to deploy rests on no failed attestation; a criterion attested failed leaves fit with conditions or not fit.
ok: fit-despite-failure.ttl fails S8-Recommendation and no other of the twenty-two
cx = checked.counterexample("dso-approval-undated.ttl")
conforms, fired = checked.report("counterexamples/dso-approval-undated.ttl", cx, S)
assert not conforms and set(fired) == {"S1-DsoRelease"}
checked.passed("dso-approval-undated.ttl fails S1-DsoRelease and no other of the twenty-two")counterexamples/dso-approval-undated.ttl: conforms = False
S1-DsoRelease FAIL
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S1-DsoRelease at ev:dso-apollo-sv-r1:
A DSO release's approval is dated (drift pass 4, QA 9): epo:approvedAt is the time the rules on the release order against, not the time the release was generated.
ok: dso-approval-undated.ttl fails S1-DsoRelease and no other of the twenty-two
cx = checked.counterexample("assessment-after-session.ttl")
conforms, fired = checked.report("counterexamples/assessment-after-session.ttl", cx, S)
assert not conforms and set(fired) == {"S2-RequirementSet"}
checked.passed("assessment-after-session.ttl fails S2-RequirementSet and no other of the twenty-two")counterexamples/assessment-after-session.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet FAIL
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S2-RequirementSet at ev:requirement-set:
S2 order (drift pass 4, QA 9): the appropriateness assessment of the requirement set is dated before any session of the record started; the envelope is judged appropriate before it is tested, as the plan is approved before its probes are used (S3).
ok: assessment-after-session.ttl fails S2-RequirementSet and no other of the twenty-two
cx = checked.counterexample("session-outside-access-period.ttl")
conforms, fired = checked.report("counterexamples/session-outside-access-period.ttl", cx, S)
assert not conforms and set(fired) == {"S4-Session"}
checked.passed("session-outside-access-period.ttl fails S4-Session and no other of the twenty-two")counterexamples/session-outside-access-period.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session FAIL
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S4-Session at ev:session-4:
S4 access period (sheet 10-04; drift pass 4, contracting officer 4): a session runs within the period the record's access grant states for the session's test item: it starts no earlier than epo:accessFrom, and neither starts nor ends after epo:accessUntil. The period the contract fixed binds the sessions.
ok: session-outside-access-period.ttl fails S4-Session and no other of the twenty-two
cx = checked.counterexample("response-from-another-build.ttl")
conforms, fired = checked.report("counterexamples/response-from-another-build.ttl", cx, S)
assert not conforms and set(fired) == {"S5-Response"}
checked.passed("response-from-another-build.ttl fails S5-Response and no other of the twenty-two")counterexamples/response-from-another-build.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response FAIL
S6-Attestation pass
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S5-Response at ev:response-1:
S5 (drift pass 4, QA 10): the software agent a response is attributed to is the one its session ran against, the agent the session is associated with, reached through the turn that generated the response; a response from another build is not evidence about this test item.
ok: response-from-another-build.ttl fails S5-Response and no other of the twenty-two
cx = checked.counterexample("attestation-on-another-item.ttl")
conforms, fired = checked.report("counterexamples/attestation-on-another-item.ttl", cx, S)
assert not conforms and set(fired) == {"S6-Attestation"}
checked.passed("attestation-on-another-item.ttl fails S6-Attestation and no other of the twenty-two")counterexamples/attestation-on-another-item.ttl: conforms = False
S1-DsoRelease pass
S2-AcceptanceCriterion pass
S2-Requirement pass
S2-RequirementSet pass
S3-PlanApproval pass
S3-PlanDeviation pass
S3-Probe pass
S3-Strategy pass
S3-TestPlan pass
S4-Session pass
S4-TestSuite pass
S4-Turn pass
S5-Evidence pass
S5-Response pass
S6-Attestation FAIL
S6-Determination pass
S7-ConformanceVerdict pass
S7-CoverageComputation pass
S7-Report pass
S7-ReportApproval pass
S8-Delivery pass
S8-Recommendation pass
S6-Attestation at ev:attestation-1:
S6 binding (sheet 10-17; drift pass 4, QA 19): the subject of an attestation is the system under test the record's requirement set binds; an attestation on another item says nothing about this evaluation, as S4 already asks of every determination.
ok: attestation-on-another-item.ttl fails S6-Attestation and no other of the twenty-two
3The model graph conforms to M2 to M5¶
The model graph is model/og-caie.model.ttl, the pruned RDF rendering of the
SysML source (R-22). The ten named M-shapes are copied from
shapes/model.shapes.ttl: every wire local (M2), every item kind reaching
the recorder (M3), the steps in order forming a DAG that produces every item
kind (M4), the three roles in their slots (M5).
model = checked.model_graph()
M = checked.shapes("shapes/model.shapes.ttl", MODEL_SHAPES)
conforms, fired = checked.report("model/og-caie.model.ttl", model, M)
assert conforms and not fired
checked.passed("the model graph conforms to the ten M2 to M5 shapes")model/og-caie.model.ttl: conforms = True
M2-Part pass
M2-Wire pass
M3-RecordHub pass
M4-EveryItemProduced pass
M4-Nesting pass
M4-ProcessDag pass
M4-Steps pass
M5-Cardinality pass
M5-PortsBelongToRoles pass
M5-Roles pass
ok: the model graph conforms to the ten M2 to M5 shapes
4Two model counterexamples¶
Each is built to RDF through the same pipeline as the canonical graph (the
pinned OpenSysML converter, then scripts/prune_model.py) and run against
the same ten shapes. tests/test_model_graph.py pins the shape each must
fail; the claim is that shape, and the assert says so. What else fires is
printed, not hidden.
| The chapter says | File | Must fail |
|---|---|---|
| a port no seam reaches | model/unwired-port.sysml | M2-Part |
| a domain expert who applies probes | model/expert-administers-tests.sysml | M5-PortsBelongToRoles |
In unwired-port.sysml the recorder’s probe port is on no wire, so
M2-Part fires at the recorder; M3-RecordHub fires as well, since the same
missing wire means the probe item kind never reaches the recorder. In
expert-administers-tests.sysml the domain expert carries a port that
applies probes to the test item, so M5-PortsBelongToRoles fires at that
port; M2-Part fires as well at the test item, whose input port is then on
two wires, the operator’s and the expert’s, where an input wire must be
unique (R-25).
cx = checked.model_counterexample("unwired-port.sysml")
conforms, fired = checked.report("counterexamples/model/unwired-port.sysml", cx, M)
assert not conforms and "M2-Part" in fired
checked.passed("unwired-port.sysml does not conform and fails M2-Part")counterexamples/model/unwired-port.sysml: conforms = False
M2-Part FAIL
M2-Wire pass
M3-RecordHub FAIL
M4-EveryItemProduced pass
M4-Nesting pass
M4-ProcessDag pass
M4-Steps pass
M5-Cardinality FAIL
M5-PortsBelongToRoles pass
M5-Roles FAIL
M2-Part at elmt:UnwiredPort__Recorder:
At the part: every input port is present on exactly one wire (input wires are unique, R-25, R-26).
M3-RecordHub at elmt:UnwiredPort__ProbeWrite:
Every item kind that crosses a seam reaches the recorder: the recorder owns a conjugated port of this type (R-17).
M5-Cardinality at elmt:UnwiredPort__SponsorOrganization:
The sponsor organization holds exactly one signatory (SponsorSignatory), the named person who signs, approves the requirement set and accepts on its behalf (sheet 10-06, R-50).
M5-Cardinality at elmt:UnwiredPort__TestingOrganization:
The testing organization holds exactly one authorized representative (AuthorizedRepresentative) (R-23).
M5-Roles at elmt:UnwiredPort:
The testing organization holds an authorized representative (AuthorizedRepresentative) and a team of at least one domain expert and one evaluation operator.
M5-Roles at elmt:UnwiredPort:
The three actor categories are distinct part definitions of Person: DomainExpert, EvaluationOperator, AuthorizedRepresentative (R-10, R-23).
ok: unwired-port.sysml does not conform and fails M2-Part
cx = checked.model_counterexample("expert-administers-tests.sysml")
conforms, fired = checked.report("counterexamples/model/expert-administers-tests.sysml", cx, M)
assert not conforms and "M5-PortsBelongToRoles" in fired
checked.passed("expert-administers-tests.sysml does not conform and fails M5-PortsBelongToRoles")counterexamples/model/expert-administers-tests.sysml: conforms = False
M2-Part FAIL
M2-Wire pass
M3-RecordHub pass
M4-EveryItemProduced pass
M4-Nesting pass
M4-ProcessDag pass
M4-Steps pass
M5-Cardinality FAIL
M5-PortsBelongToRoles FAIL
M5-Roles FAIL
M2-Part at elmt:ExpertAdministersTests__TestItem:
At the part: every input port is present on exactly one wire (input wires are unique, R-25, R-26).
M5-Cardinality at elmt:ExpertAdministersTests__SponsorOrganization:
The sponsor organization holds exactly one signatory (SponsorSignatory), the named person who signs, approves the requirement set and accepts on its behalf (sheet 10-06, R-50).
M5-Cardinality at elmt:ExpertAdministersTests__TestingOrganization:
The testing organization holds exactly one authorized representative (AuthorizedRepresentative) (R-23).
M5-PortsBelongToRoles at elmt:ExpertAdministersTests__DomainExpert__probesToItem:
Evidence is supplied only by the evaluation operator, and a person applies probes to the test item only as the evaluation operator (R-12, R-23).
M5-Roles at elmt:ExpertAdministersTests:
The testing organization holds an authorized representative (AuthorizedRepresentative) and a team of at least one domain expert and one evaluation operator.
M5-Roles at elmt:ExpertAdministersTests:
The three actor categories are distinct part definitions of Person: DomainExpert, EvaluationOperator, AuthorizedRepresentative (R-10, R-23).
ok: expert-administers-tests.sysml does not conform and fails M5-PortsBelongToRoles
5Verdict¶
One line for the reader and for the gate. It is printed only when every cell above ran and every assert held.
checked.verdict()claims checked: 37
the measles evaluation conforms to the twenty-two S1 to S8 shapes
attestation-without-evidence.ttl fails S6-Attestation and, through it, S8-Recommendation, and no other of the twenty-two
attestation-off-plan.ttl fails S6-Attestation and no other of the twenty-two
attestation-off-turn.ttl fails S6-Attestation and no other of the twenty-two
probe-before-requirements.ttl fails S2-RequirementSet and no other of the twenty-two
recommendation-untraced.ttl fails S8-Recommendation and no other of the twenty-two
expert-administers-tests.ttl fails S4-Session and no other of the twenty-two
executive-attests.ttl fails S6-Attestation and no other of the twenty-two
attestation-before-determination.ttl fails S6-Attestation and no other of the twenty-two
consistency-check-after-turn.ttl fails S3-Probe and no other of the twenty-two
plan-approved-after-session.ttl fails S3-PlanApproval and no other of the twenty-two
delivery-before-approval.ttl fails S8-Delivery and no other of the twenty-two
report-coverage-misstated.ttl fails S7-Report and no other of the twenty-two
report-rates-padded.ttl fails S7-Report and no other of the twenty-two
dso-before-stakeholder-input.ttl fails S1-DsoRelease and no other of the twenty-two
cherry-picked-determination.ttl fails S6-Attestation and no other of the twenty-two
insufficient-yet-failed.ttl fails S6-Attestation and no other of the twenty-two
requirement-set-unapproved.ttl fails S2-RequirementSet and no other of the twenty-two
deviation-unrecorded.ttl fails S3-TestPlan and, the coverage no longer supported, S7-Report, and no other of the twenty-two
deviation-by-expert.ttl fails S3-PlanDeviation and no other of the twenty-two
session-on-another-item.ttl fails S4-Session and, through its responses, S5-Response, and no other of the twenty-two
final-report-without-verdict.ttl fails S7-Report and no other of the twenty-two
verdict-without-digests.ttl fails S7-ConformanceVerdict and no other of the twenty-two
coverage-without-digests.ttl fails S7-CoverageComputation and no other of the twenty-two
recommendation-unapproved.ttl fails S8-Recommendation and no other of the twenty-two
draft-without-gaps.ttl fails S7-Report and no other of the twenty-two
draft-rates-from-later.ttl fails S7-Report and no other of the twenty-two
draft-approved.ttl fails S7-ReportApproval and no other of the twenty-two (S9-Acceptance, outside them, is pinned by the tests)
fit-despite-failure.ttl fails S8-Recommendation and no other of the twenty-two
dso-approval-undated.ttl fails S1-DsoRelease and no other of the twenty-two
assessment-after-session.ttl fails S2-RequirementSet and no other of the twenty-two
session-outside-access-period.ttl fails S4-Session and no other of the twenty-two
response-from-another-build.ttl fails S5-Response and no other of the twenty-two
attestation-on-another-item.ttl fails S6-Attestation and no other of the twenty-two
the model graph conforms to the ten M2 to M5 shapes
unwired-port.sysml does not conform and fails M2-Part
expert-administers-tests.sysml does not conform and fails M5-PortsBelongToRoles
NOTEBOOK: PASS