An evaluation is work done under a contract. Before any test runs, a customer has stated a need, a provider has answered it, the two have signed, and the test item’s provider has opened it up. After the last attestation, the provider delivers and the customer accepts. This chapter is that outer cycle: the parties, the steps, what the contract pins, and the measles case.
1What the standards say¶
Three organizations are parties. The sponsor is ISO 9000’s customer, the testing organization its provider, and the organization that provides the test item is accountable for it as its provider, ISO/IEC 17000’s first party. The sponsor’s obligation to the affected populations is its own and the contract does not transfer it. A sponsor with a user interest in the test item performs a second-party activity when it commissions and accepts the evaluation; the independent testing organization performs a third-party activity whoever commissions it. Affected stakeholders are populations, spoken for, sometimes interviewed, who sign nothing. The six steps are the standards’ own; each quote carries its tag, defined on the vocabulary page:
The outer cycle, contracting through delivery: Guide to the Systems Engineering Body of Knowledge, Stakeholder Needs (Acquisition and Supply), p. 353: “These needs and requirements are expressed in agreements between acquirers and suppliers.”.
| Step | Matches | Also |
|---|---|---|
| C1 need: the sponsor states its mission and obligations towards the affected populations, the problem, and what success would look like | Guide to the Systems Engineering Body of Knowledge, Business or Mission Analysis, p. 543: “The purpose of Business or Mission Analysis is to understand a mission or market problem, threat, or opportunity, and to establish the goals, objectives and measures of success of a potential solution class.” | ISO 9000:2026(en) Quality management — Fundamentals and vocabulary, 3.5.1 requirement: “need or expectation that is stated, generally implied or obligatory” |
| C2 propose: the sponsor seeks the service and the testing organization responds | Guide to the Systems Engineering Body of Knowledge, Enterprise Systems Engineering, contract products and services, p. 935: “Contract products and services often demand tailor-made system/service solutions which are typically specified by a single customer to whom the solution is provided. The supplier responds with proposed solutions.” | ISO/IEC 17000:2020(en) Conformity assessment — Vocabulary and general principles, 4.10 access: “opportunity for an applicant to obtain a conformity assessment service from a body under a conformity assessment scheme”; IEEE Computer Society, proposal, p. 329 (ISO/IEC/IEEE 24765:2014, SEVOCAB tag 24765c): “supplier’s offer to provide a system or service, usually including benefits, costs, risks, opportunities, and other factors applicable to decisions”; IEEE Computer Society, request for proposal (RFP), p. 350 (ISO/IEC/IEEE 24765:2017): “document used by the acquirer as the means to announce its intention to potential bidders to acquire a specified system, software product, or software service” |
| C3 agree: the service agreement fixes the counterparties, the test item, the requirements’ frame and the method | Guide to the Systems Engineering Body of Knowledge, Stakeholder Responsibilities, Acquirer/Supplier Agreements, p. 353 (INCOSE 2012, the acquisition process): “The acquisition process includes activities to identify, select, and reach commercial agreements with a product or service supplier.” | ISO 9000:2026(en) Quality management — Fundamentals and vocabulary, 3.3.13: “binding agreement”; ISO/IEC 17000:2020(en) Conformity assessment — Vocabulary and general principles, 4.9 conformity assessment scheme: “set of rules and procedures that describes the objects of conformity assessment, identifies the specified requirements and provides the methodology for performing conformity assessment” |
| C4 access: the test item provider (ISO/IEC 17000 first party) provides the test item and the test environment | IEEE Computer Society, test environment and data management process, p. 434 (ISO/IEC/IEEE 29119-2:2021, 3.37): “test process for establishing and maintaining a required test environment and corresponding test data” | ISO/IEC 17000:2020(en) Conformity assessment — Vocabulary and general principles, 4.2 object of conformity assessment: “entity to which specified requirements apply”; ISO/IEC 17000:2020(en) Conformity assessment — Vocabulary and general principles, Annex A, A.2 Selection (the functional approach: planning and preparation to obtain the inputs for determination); IEEE Computer Society, test environment, p. 434 (ISO/IEC/IEEE 29119-2:2021, 3.34): “environment containing facilities, hardware, software, firmware, and procedures needed to conduct a test”; IEEE Computer Society, test item transmittal report, p. 435 (ISO/IEC/IEEE 24765:2017): “document identifying test items” |
| C5 deliver: the authorized representative delivers the final report, its approval and the recommendation to the sponsor | IEEE Computer Society, test completion report, p. 433 (ISO/IEC/IEEE 29119-2:2021, 3.26): “report that provides a summary of the testing that was performed” | ISO/IEC 17000:2020(en) Conformity assessment — Vocabulary and general principles, 7.3 attestation, Note 1 (fragment of the note): “is intended to convey the assurance that the specified requirements have been fulfilled”; IEEE Computer Society, delivery, p. 120 (ISO/IEC/IEEE 24765:2017): “release of a system or component to its customer or intended user” |
| C6 accept: the sponsor accepts the delivery as performance of the agreement | IEEE Computer Society, acceptance, p. 4 (ISO/IEC/IEEE 24748-5:2017, 3.1): “action by an authorized representative of the acquirer by which the acquirer assumes ownership of products as partial or complete performance of an agreement” | Guide to the Systems Engineering Body of Knowledge, Acceptance Criteria (glossary), p. 1437 (INCOSE 2011, Section 6.1.15): “The procurement specification, in the context of the overall agreement, should clearly state the criteria by which the acquirer will accept delivery from the supplier.” |
SEVOCAB definitions: Copyright © 2021 IEEE. Used by permission.
A seventh step, fulfil, the whole evaluation, sits between access and
delivery; the nested lifecycle chapter draws it.
2The specification¶
The model states the cycle as parts, ports and wires: a part is a party or
a role inside the testing organization, a port the typed point where an
item leaves or enters a part, and a wire (a seam, in the model) joins an
output port to an input port and carries one item kind. The sponsor holds
an obligation towards the affected populations, states its mission and
then its need, and receives the proposal. Where the sponsor signs, a
named person signs for it: its signatory signs the agreement, declares the
user interest, approves the requirement set and accepts delivery. The
authorized representative answers, countersigns, declares the testing
organization’s independence and delivers. The agreement
accepts the proposal; the statement of work, the access and the delivery
are performed under it. The test item provider grants access for a stated
version and period in a named environment, under an instrument outside
this agreement that the record names.
Every item reaches the recorder, the machine keeping the record; three
more machines act: the test driver derives probes, the conformance
checker checks them and the record, the report assembler assembles the
report. Input wires are unique, output wires fan out. One item sits on the boundary
between the contract and its fulfilment: the statement of work,
the sponsor’s decision, for each affected population, whether
it is interviewed before it is spoken for. A team member speaks for every
population; an interview, when decided, feeds that representation. In
the model this cycle is the outer action def, SysML’s definition of a
process as steps in succession; its fulfil step is the evaluation as a
black box.
View contracting: the contracting slice. In focus: the parties to the contract and the two parts of the testing organization they touch, the items pinned at the contract braided into one bundle per pair of parts, and the sponsor’s obligation to the affected populations, a relation that carries no item. Left out: the evaluation team, the machines and the test item; the evaluation items; the seam names and the ports; the recorder’s wires to the machines. Legend: rounded green, a person; double-boxed pink, a machine; dashed amber, an affected population; a plain box, an organization; a solid arrow bundles the items that flow from one part to another; a dotted arrow is a relation that carries no item.
What the contract pins is the first layer of assumptions: the counterparties, the test item, the frame of the requirements and the method. The evaluation takes them as given and cannot change them. The record states the level at which independence between roles is required; the measles evaluation is at the person level: nobody determines alone on evidence from a session they ran, and nobody assesses a requirement set they wrote. The report is the record’s delivered export; the record itself is not delivered but is independently auditable, access to it a special case, publication not assumed. Acceptance is the sponsor’s act on receipt of the deliverables, recognizing completion of the contract’s obligations; a correctly constructed record is necessary and not sufficient for it, and it is not the acceptance criteria the evaluation tests. The essentials are the statements the specification must keep, SCI-01 to SCI-13, each naming the shapes that check it, a shape being one machine-checked rule written in SHACL, the W3C Shapes Constraint Language. The essentials this chapter states:
| ID | Statement | Checked by |
|---|---|---|
| SCI-10 | The sponsor, the testing organization and the test item provider are named; the sponsor’s mission and obligations towards the affected populations are recorded before the need, and its statement of work decides, per population, interview or representation; the agreement is signed for the sponsor by its signatory and for the testing organization by its authorized representative; every affected population is spoken for by a named representative on the team, and interviewed where the statement of work decided so. | M1-Obligation, M1-Parties, S0-Member, S0-Mission, S0-Need, S0-Parties, S0-Population, S0-Proposal, S0-Record, S0-StatementOfWork, S8-Delivery, S9-Acceptance |
| SCI-12 | The domain expert who approves the DSO, assesses appropriateness and attests is not the evaluation operator who administers tests and collects evidence, and neither is the authorized representative who signs the agreement; one authorized representative, one or more of each expert. | M5-Cardinality, M5-PortsBelongToRoles, M5-Roles, S0-Independence, S0-Roles, S1-DsoRelease, S4-Session, S6-Attestation, S6-Determination |
| SCI-13 | What the contract pins (the counterparties, the test item, the requirements’ frame, the method) is recorded and closed before what the evaluation pins (the environment, the requirement set, the DSO release, the plan) is declared; the evaluation documents the first layer and cannot change it. | M4-Nesting, S0-Layers |
The first of them as ogc, the reader of the repository’s graphs, prints
it:
$ uv run -q ogc sci SCI-10
# ogc sci SCI-10 @ <sha>
## SCI-10 PartiesNamedAndAgreed (machine)
The sponsor, the testing organization and the test item provider are named; the sponsor's
mission and obligations towards the affected populations are recorded before the need, and its
statement of work decides, per population, interview or representation; the agreement is
signed for the sponsor by its signatory and for the testing organization by its authorized
representative; every affected population is spoken for by a named representative on the team,
and interviewed where the statement of work decided so.
checked by: M1-Obligation, M1-Parties, S0-Member, S0-Mission, S0-Need, S0-Parties,
S0-Population, S0-Proposal, S0-Record, S0-StatementOfWork, S8-Delivery, S9-Acceptance
terms: authorized representative, contract, customer, evaluation customer, evaluation service
provider, first-party conformity assessment activity, mission, organization, provider,
second-party conformity assessment activity, stakeholder, statement of work, test item
customer, test item provider, third-party conformity assessment activity
rests on: R-21, R-23, iso-9000-2026, iso-iec-17000-2020
(exit 0)
3The walkthrough¶
The county public-health office exists to protect the health of residents and of the people passing through the county, and must inform the public accurately during an outbreak; that mission opens the record. It needed to know whether its chatbot could give measles advice to the public. Humane Intelligence proposed an OG-CAIE evaluation; Mala, its authorized representative, signed for it on 31 July and declared it independent of the vendor; Dana Okafor, the county’s health officer, signed for the county and declared its user interest in the chatbot; the vendor opened API access to version 1 the same day, for August, under its deployment contract with the county. Two populations were affected: commuters, interviewed and then spoken for by Theo, and county residents, spoken for by Annie, the domain expert, without an interview, as the county’s statement of work decided. After the evaluation, Mala delivered the final report, its approval and the recommendation, fit to deploy once the vaccination question is asked before any advice, on 12 August, and Dana Okafor accepted for the county on 14 August. The case is synthetic: the names are borrowed from real colleagues whose roles they recognise, the health officer is invented, and no signature or attestation here was made by them.
| Step | Item | What it says | Who | When |
|---|---|---|---|---|
| C3 agree | engagement-commuters (EngagementDecision) | commuters: interviewed, their needs being underdocumented | ||
| C3 agree | engagement-residents (EngagementDecision) | county residents: represented by the domain expert | ||
| C1 need | mission-1 (Mission) | the county public-health office exists to protect the health of county residents and of the people who pass through the county, and is obliged to inform the public accurately during an outbreak | county public-health office (sponsor: evaluation customer, and test item customer as the chatbot’s deployer) | 2026-07-28 |
| C1 need | need-1 (Need) | the county public-health office needs to know whether its chatbot may give measles advice to the public during the outbreak | county public-health office (sponsor: evaluation customer, and test item customer as the chatbot’s deployer) | 2026-07-28 |
| C2 propose | proposal-1 (Proposal) | Humane Intelligence proposes an OG-CAIE evaluation of the chatbot under the Apollo-SV DSO | Mala (authorized representative) | 2026-07-30 |
| C3 agree | service-agreement (ServiceAgreement) | agreement to evaluate the public-health chatbot during the measles outbreak, as a service; accepts the proposal | Dana Okafor (county health officer, sponsor signatory); Mala (authorized representative) | 2026-07-31 |
| C3 agree | statement-of-work (StatementOfWork) | scope of work under the agreement: the commuters are interviewed, the county residents are represented by the domain expert | Dana Okafor (county health officer, sponsor signatory) | 2026-07-31 |
| C3 agree | independence-declaration-1 (IndependenceDeclaration) | Humane Intelligence declares that it is independent of the chatbot vendor and has no user interest in the chatbot | Mala (authorized representative) | 2026-07-31 |
| C3 agree | user-interest-declaration-1 (UserInterestDeclaration) | the county public-health office declares that it has a user interest in the chatbot, which it deploys to the public | Dana Okafor (county health officer, sponsor signatory) | 2026-07-31 |
| C4 access | test-item-access (TestItemAccess) | API access to chatbot v1 for the evaluation, August 2026, under the vendor’s deployment contract with the county | Meridian Health Software, the chatbot’s vendor (test item provider; an invented company) | 2026-07-31 |
| C5 deliver | delivery-1 (Delivery) | final report, its approval and the recommendation delivered to the county public-health office | Mala (authorized representative) | 2026-08-12 |
| C6 accept | acceptance-1 (Acceptance) | the county health officer accepts the delivery on behalf of the county public-health office | Dana Okafor (county health officer, sponsor signatory) | 2026-08-14 |
The statement of work as the tool reads it, with its canonical quote and the ruling that made it an item of its own:
$ uv run -q ogc term 'statement of work'
# ogc term 'statement of work' @ <sha>
## statement of work (statement-of-work)
The sponsor's decisions about the work to be performed under the contract; here, for each
affected population, whether it is interviewed, with its stakeholder needs documented as
input, or represented by a member of the team. A scope-of-work judgment: representation is the
common case, interviews are reserved for underrepresented stakeholders or underdocumented
needs because of the effort they cost. Decided with the agreement, pinned at the contract,
consumed by the scope step, and traceable either way.
class: adopted
also: SOW; scope of work
canonical:
sevocab (rank 2, heldLocally) statement of work, p. 406 (ISO/IEC 33202:2024, 3.25) [machine]
"statement of the expected outcomes and outline of the work required to achieve the
outcomes"
scope note:
Ruling R-40: a boundary item between the contract and its fulfilment; the sponsor decides, per
affected population, interview or representation; the record must show the decision and the
evaluation must realize it.
binding: sysml item def StatementOfWork, port def StatementOfWorkWrite, seams
statementOfWorkSeam and statementOfWorkToExecutiveSeam; epo:StatementOfWork with epo:decides
engagement decisions
related: contract (contract); evaluation customer (evaluation-customer); mission (mission);
stakeholder (stakeholder)
matches: exactMatch SEVOCAB statement of work, p. 406
EPO classes naming it: epo:EngagementDecision, epo:StatementOfWork
derives from rulings: R-40 (the sponsor's decision to interview or represent each affected
population had no item of its own)
concerns naming it: C-46 (ruled) the sponsor's decision to interview or represent each affected
population had no item of its own
essentials stated in it: SCI-10
Popper crosswalk: auxiliary assumption, first layer (pinned at the contract)
permission: Copyright © 2021 IEEE. Used by permission.
(exit 0)
Two further steps are observed in practice and are not part of the contracting lifecycle: the county tells the vendor to change the chatbot, and may then require new testing. The standards call the repeat monitoring, the status of the system determined again at a later stage; a new version of the test item opens a new record linked to the old one, and a record that traces every finding to its evidence is what a second evaluation is measured against.
4Checked¶
Shapes S0-Record, S0-Member, S0-Parties, S0-Roles, S0-Independence, S0-Access, S0-Population, S0-StatementOfWork, S0-Mission, S0-Need, S0-Proposal, S0-Layers, S8-Delivery and S9-Acceptance run over the record; M1-Parties, M1-Obligation and M5-Cardinality run over the model graph. Fifteen counterexamples, fourteen records and a model broken on purpose, must fail: among them a requirement set declared before the agreement, a population nobody speaks for, an untagged member, a person holding two roles, an independence nobody declared, an acceptance by the organization instead of its signatory, and a model with no obligation between the sponsor and the populations.
5There is more in the model¶
6Sources cited¶
ISO, ISO 9000:2026 Quality management — Fundamentals and vocabulary (3.1.9; 3.11.3; 3.3.13; 3.5.1 requirement; 3.9.1; essentials SCI-10, SCI-12) International Organization for Standardization, 2026
IEEE Computer Society, Software and Systems Engineering Vocabulary (SEVOCAB), PDF export created 2026-09-02 (481 pp.) (acceptance criteria, p. 4 (ISO/IEC 33202:2024, 3.1); acceptance, p. 4 (ISO/IEC/IEEE 24748-5:2017, 3.1); artificial intelligence-based system, p. 28 (ISO/IEC TR 29119-11:2020); delivery, p. 120 (ISO/IEC/IEEE 24765:2017); proposal, p. 329 (ISO/IEC/IEEE 24765:2014, SEVOCAB tag 24765c); recommendation, p. 340 (ISO/IEC 14143-2:2011, 3.9); request for proposal (RFP), p. 350 (ISO/IEC/IEEE 24765:2017); statement of work, p. 406 (ISO/IEC 33202:2024, 3.25); system under test (SUT), p. 423 (ISO/IEC 14756:1999); system-of-interest (SOI), p. 423 (ISO/IEC/IEEE 15288:2023); test completion report, p. 433 (ISO/IEC/IEEE 29119-2:2021, 3.26); test environment and data management process, p. 434 (ISO/IEC/IEEE 29119-2:2021, 3.37); test environment, p. 434 (ISO/IEC/IEEE 29119-2:2021, 3.34); test item transmittal report, p. 435 (ISO/IEC/IEEE 24765:2017); test item, p. 435 (ISO/IEC/IEEE 29119-2:2021); essentials SCI-12) IEEE Computer Society and ISO/IEC JTC 1/SC 7, 2026
NIST, Assessing Risks and Impacts of AI (ARIA): ARIA 0.1 Pilot Evaluation Report (Appendix A, Application, p. 16 (fragment of the entry)) Amironesei et al., 2025
ISO/IEC, ISO/IEC 17000:2020 Conformity assessment — Vocabulary and general principles (4.10 access; 4.2 object of conformity assessment; 4.3 first-party conformity assessment activity; 4.4 second-party conformity assessment activity; 4.9 conformity assessment scheme; 7.3 attestation, Note 1 (fragment of the note); 8.1 surveillance; Annex A, A.2 Selection (the functional approach: planning and preparation to obtain the inputs for determination); essentials SCI-10, SCI-13) ISO/IEC, 2020
INCOSE, Guide to Writing Requirements v4, Summary Sheet (C7 Verifiable, p. 2) INCOSE, 2023
Guide to the Systems Engineering Body of Knowledge (SEBoK) v2.14, released 18 May 2026 (Acceptance Criteria (glossary), p. 1437 (INCOSE 2011, Section 6.1.15); Business or Mission Analysis, p. 543; Enterprise Systems Engineering, contract products and services, p. 935; Stakeholder Needs (Acquisition and Supply), p. 353; Stakeholder Responsibilities, Acquirer/Supplier Agreements, p. 353 (INCOSE 2012, the acquisition process); essentials SCI-13) SEBoK Editorial Board, 2026
Hollek and Zargham, Building Scientific Approaches to Generative AI (essentials SCI-13) Hollek & Zargham, 2026
- International Organization for Standardization. (2026). ISO 9000:2026 Quality management — Fundamentals and vocabulary. International Organization for Standardization. https://www.iso.org/obp/ui/#iso:std:iso:9000:ed-5:v1:en
- IEEE Computer Society and ISO/IEC JTC 1/SC 7. (2026). IEEE Computer Society, Software and Systems Engineering Vocabulary (SEVOCAB), PDF export created 2026-09-02 (481 pp.). IEEE Computer Society and ISO/IEC JTC 1/SC 7. https://www.computer.org/sevocab
- Amironesei, R., Godil, A., Greenberg, C., Greene, K., Hall, P., Jensen, T., Fiscus, J., & Schulman, N. (2025). Assessing Risks and Impacts of AI (ARIA): ARIA 0.1 Pilot Evaluation Report (Techreport NIST AI 700-2). National Institute of Standards and Technology. 10.6028/NIST.AI.700-2
- ISO/IEC. (2020). ISO/IEC 17000:2020 Conformity assessment — Vocabulary and general principles. ISO/IEC. https://www.iso.org/obp/ui/en/#iso:std:73029:en
- INCOSE. (2023). Guide to Writing Requirements v4, Summary Sheet. INCOSE.
- SEBoK Editorial Board. (2026). Guide to the Systems Engineering Body of Knowledge (SEBoK) v2.14, released 18 May 2026 (N. Hutchison, Ed.). The Trustees of the Stevens Institute of Technology. https://sebokwiki.org/
- Hollek, J., & Zargham, M. (2026). Building Scientific Approaches to Generative AI. Birds-of-a-Feather session, SciPy 2026.