# Events, metrics and scorecards

Day 2, hours 1 to 3 of the [training path](TRAINING.md). By the end, the engine keeps a journal of the events your
scenario emits, computes metrics from it, and the Training plugin scores the run against a scorecard: live while
it runs, and the same again from its recording.

You need [day 1](TRAINING-FIRST-SCENARIO.md), a scenario that emits events, and an engine with the Training plugin
(`argus plugin list` lists `training`). The examples are Crossing Drive's, whose finished files are in the sample:
`journal.json` and `training/crossing-drive.scorecard.json`.

How the pieces fit:

| Piece | File | Who computes it |
|---|---|---|
| **Events**: what your behaviours emit | your scripts | your behaviours, `ctx.events.emit` |
| **Kinds**: which events the journal keeps, with their fields | `journal.json` | you declare them |
| **Journal**: every declared event, at its tick | (in the session, and in its recording) | the engine |
| **Metrics**: numbers computed from the journal and from entity signals | `journal.json` | the engine, every tick |
| **Scorecard**: what is scored, how, and what passes | `training/<name>.scorecard.json` | the Training plugin |

## 1. Add the Training plugin

> **Ask:** "Add the Training plugin to this project."

**By hand**, from a terminal in the project folder:

```sh
argus plugin add training
```

On Windows the command is `argus.exe` in the engine's `bin` folder ([day 1](TRAINING-FIRST-SCENARIO.md), "Before you start").
It prints what the plugin may do, then what it writes:

```text
argus plugin add: adds "training" (<your project folder>)
  it may:
    net.http.serve (serves pages and an API under /plugins/training/ on the engine's local query server (this machine only)): serves the instructor console page (loopback)
    session.control (pauses, steps and scales time, injects events and takes over entities): pause, inject events, take over vehicles and characters, change the environment
    fs.project.write (writes files in the project folders it names: output/training/): writes reports and exports
  entry: {"id":"training"}
  plugins, in load order: training
  copies training/sample.scorecard.json
wrote training/sample.scorecard.json, argus.project.json
```

That is `"plugins": [{ "id": "training" }]` in `argus.project.json`, and a sample scorecard in
`training/sample.scorecard.json`. It never overwrites a file you have. `--dry-run` prints the same and writes
nothing.

The sample scorecard is a template. Its measures name metrics like `training.speed`, which your project does not
have. Section 5 writes Crossing Drive's own, so delete the template or keep it to read.

**In the editor:** **Project Settings** ▸ **Plugins**, then **Add** beside Training. It shows the same list and adds the
plugin when you confirm. More in [Plugins](PLUGINS.md).

## 2. Declare the events

The journal keeps only the events a **kind** declares. Kinds go in `journal.json` at the project root, which applies
to every scene. A `scenes/<scene>.journal.json` beside a scene file applies to that scene only.

```json
{ "kinds": [ ... ], "metrics": [ ... ] }
```

Crossing Drive declares the eight events its scripts emit for scoring. Its `journal.json` starts with them:

```json
{"kind": "stop-line", "label": "Stop line", "severity": "info", "roles": ["subject"], "fields": {"stopped": {"type": "bool"}, "min_kmh": {"type": "number", "unit": "km/h"}, "speed_kmh": {"type": "number", "unit": "km/h"}}},
{"kind": "crossing-entered", "label": "Entered the crossing", "severity": "info", "roles": ["subject"], "fields": {}},
{"kind": "destination-reached", "label": "Destination reached", "severity": "info", "roles": ["subject"], "fields": {"seconds": {"type": "number", "unit": "s"}}},
{"kind": "trainee-collision", "label": "Collision", "severity": "critical", "roles": ["subject", "other"], "fields": {"speed_kmh": {"type": "number", "unit": "km/h"}}},
{"kind": "pedestrian-stepped-out", "label": "Pedestrian steps out", "severity": "warn", "roles": ["pedestrian"], "fields": {"distance_m": {"type": "number", "unit": "m"}}},
{"kind": "speed-over", "label": "Over the speed limit", "severity": "warn", "roles": ["subject"], "fields": {"speed_kmh": {"type": "number", "unit": "km/h"}, "limit_kmh": {"type": "number", "unit": "km/h"}}},
{"kind": "brake-applied", "label": "Brake applied", "severity": "info", "roles": ["subject"], "fields": {"speed_kmh": {"type": "number", "unit": "km/h"}}},
{"kind": "send-pedestrian", "label": "Send the pedestrian", "severity": "critical", "injectable": true, "target": "world", "fields": {}}
```

(`goal-reached`, which the scripts also emit, is the self-test's. It is not declared, so the journal does not keep
it.)

| Field | Meaning |
|---|---|
| `kind` | the event's name, exactly as the script emits it |
| `label` | what the console and the report call it |
| `severity` | `info`, `warn` or `critical`: how the timeline and the event list mark it |
| `roles` | data fields that name entities. A `subject` field in the event names the entity the event is about |
| `fields` | the other data fields, each with a `type` (`number`, `int`, `bool`, `enum`, `entity`, `vec3`, `string`) and optionally `unit`, `min`, `max`, `default`. An `enum` lists its options: `{ "type": "enum", "enum": ["pedestrian", "debris"], "default": "pedestrian" }` |
| `injectable` | the instructor may raise it from the console's palette (an injected kind needs a behaviour that reacts to it) |
| `target` | for an injectable kind: `world`, or `entity` when the instructor picks which entity |

Nothing changes in your scripts. `ctx.events.emit("stop-line", { subject: "car", stopped: true, ... })` already sends
the event, and with the kind declared the journal keeps it. This is the entry from a real run of the sample:

```json
{ "seq": 2, "tick": 547, "kind": "stop-line", "severity": "info", "entities": { "emitter": "car", "subject": "car" },
  "data": { "min_kmh": 75.6, "speed_kmh": 78.1, "stopped": false, "subject": "car" }, "source": "script" }
```

A script's event may carry fields its kind does not declare: the journal keeps them as they are. An injected
event may carry only declared fields, because the console builds its settings from them: an undeclared one is
refused (`bad-params`).

`emitter` is the entity whose behaviour emitted it. `source` is `script` for your events, `command` for an
instructor's (a pause, an injection, with `by`, the console that sent it), and `engine` for what the engine adds
itself (`metric.crossed`, for one). An event of a kind nobody declared is not journaled.

A vehicle on the `simulation` model ([Vehicles](VEHICLES.md)) also journals its driver aids and its grip as they start
and end, under the engine's own kinds: `vehicle.abs.start` and `vehicle.abs.end` for the anti-lock, and the same for
`vehicle.tc` (traction control), `vehicle.esc` (stability control), `vehicle.wheel_lock` and `vehicle.slide`. The
vehicle is the `subject`; a start carries `speed_kmh`, an end `duration_ticks`. A metric counts them like any kind
(`"count": {"kind": "vehicle.abs.start"}`, section 3). Crossing Drive's car is the `arcade` model, which has no aids,
so its journal has none of them.

## 3. Metrics

A metric is a number the engine computes every tick, from the journal and from entities. It is computed inside the
simulation, so a recording played back gives the same values to the last bit.

| `kind` | Value | Crossing Drive uses it for |
|---|---|---|
| `counter` | how many entries match `count: { kind }` | times over the limit, collisions |
| `timer` | seconds while running: from one kind to another, or `while` a threshold is over | time over the limit |
| `interval` | seconds from each `from` entry to the next `to` entry | reaction to the pedestrian |
| `gauge` | a sampled `signal: { entity or tag, field, scale? }` every `every` ticks, or a `field` of each `from` entry | speed; the slowest before the line; time to the destination |
| `threshold` | 1 while metric `of` is `over` or `under` a value for `for_ticks` ticks, else 0; each change journals `metric.crossed` | over the limit |
| `window` | `count`, `sum`, `min` or `max` of a metric over the last `ticks` ticks | (none) |

Any metric can have `per: "<role>"` (one value per entity in that role: one per trainee, with several trainees;
a gauge on a `tag` needs it), `between: { start, end }` (counted only between two kinds) and `reset_on: <kind>`.

Crossing Drive's eight, in the order the engine computes them. A metric can use only the ones before it:

```json
{"id": "speed", "label": "Speed", "unit": "km/h", "kind": "gauge", "per": "subject", "signal": {"tag": "trainee", "field": "speed", "scale": 3.6}, "every": 10},
{"id": "over-limit", "label": "Over the limit", "kind": "threshold", "of": "speed", "over": 52, "for_ticks": 30, "per": "subject"},
{"id": "time-over-limit", "label": "Time over the limit", "unit": "s", "kind": "timer", "while": "over-limit", "per": "subject"},
{"id": "speeding", "label": "Times over the limit", "kind": "counter", "count": {"kind": "speed-over"}},
{"id": "stop-speed", "label": "Slowest before the stop line", "unit": "km/h", "kind": "gauge", "from": {"kind": "stop-line"}, "field": "min_kmh"},
{"id": "reaction", "label": "Reaction to the pedestrian", "unit": "s", "kind": "interval", "from": {"kind": "pedestrian-stepped-out"}, "to": {"kind": "brake-applied"}, "close": "oldest"},
{"id": "collisions", "label": "Collisions", "kind": "counter", "count": {"kind": "trainee-collision"}},
{"id": "route-time", "label": "Time to destination", "unit": "s", "kind": "gauge", "from": {"kind": "destination-reached"}, "field": "seconds"}
```

- `speed` samples the speed of every entity tagged `trainee` every 10 ticks, one value per trainee, and `scale: 3.6`
  turns the engine's metres a second into km/h.
- `over-limit` is 1 while `speed` has been over 52 km/h for half a second (30 ticks), per trainee.
- `time-over-limit` counts the seconds `over-limit` is 1.
- `speeding` counts the `speed-over` events, and `collisions` the `trainee-collision` ones.
- `stop-speed` takes `min_kmh` from the `stop-line` event: the slowest the car went in the 10 m before the line.
- `reaction` is the time from the pedestrian stepping out to the next press of the brake.
- `route-time` takes `seconds` from `destination-reached`.

Each metric also keeps aggregates over the run, which a scorecard reads: `final`, `min`, `max`, `mean`, `count` and
`p95`.

## 4. Check the journal and the metrics

Serve the scenario and ask it:

```sh
argus run . --headless --serve --port 7795 --ws-port 7796
# in another terminal, while it runs:
curl http://127.0.0.1:7795/journal
curl http://127.0.0.1:7795/metrics
curl "http://127.0.0.1:7795/journal/kinds?injectable=true"    # the instructor's palette
curl "http://127.0.0.1:7795/metrics/speed/series?entity=car"  # a metric's history (a per-trainee one needs entity)
```

`--ws-port` is the session's live stream. Give it a free port of its own, or it takes 7701.

`--headless` runs with no window and no input, so the car sits still. To see the events of a drive, drive it in the
editor with Play and watch the console ([The instructor](TRAINING-INSTRUCTOR.md)). Or run the self-test and keep its
recording:

```sh
argus run . --headless --self-test crossing-drive --save-replay run.replay
```

The recording is JSON lines, about one a tick. A line with `"j"` holds that tick's journal entries, and one with `"m"`
the metric values that changed, each as `[metric, entity, bits]`: the 64-bit floating-point value's bits in hex, so a
playback compares them exactly. In Python, `struct.unpack(">d", bytes.fromhex(bits))[0]` reads one. The review
(next guides) shows them as numbers.

**If a metric never changes**, read the first lines the run prints. A definition the engine refuses is named
there with the reason (`journal-defs-invalid`), for example a metric that uses one defined after it.

## 5. The scorecard

A scorecard says what is scored, how it is measured, and what passes. It is one file matching `**/*.scorecard.json`,
usually under `training/`. The console uses the scorecard whose `scenario` names the running scene.

Crossing Drive's:

```json
{
 "schema": 1,
 "id": "crossing-drive",
 "title": "Crossing Drive",
 "scenario": "scenes/main.scene.json",
 "passMark": 75,
 "sections": [
  {
   "id": "speed",
   "title": "Speed",
   "weight": 2,
   "items": [
    {
     "id": "max-speed",
     "label": "Highest speed in the 50 zone",
     "measure": {
      "metric": "speed",
      "agg": "max"
     },
     "unit": "km/h",
     "pass": {
      "op": "<=",
      "value": 55
     },
     "grade": {
      "target": 50,
      "limit": 65
     }
    },
    {
     "id": "time-over",
     "label": "Time over the limit",
     "measure": {
      "metric": "time-over-limit"
     },
     "unit": "s",
     "pass": {
      "op": "<=",
      "value": 5
     },
     "grade": {
      "target": 0,
      "limit": 20
     }
    }
   ]
  },
  {
   "id": "junction",
   "title": "Stop line",
   "items": [
    {
     "id": "full-stop",
     "label": "Full stop at the line",
     "measure": {
      "metric": "stop-speed",
      "agg": "max"
     },
     "unit": "km/h",
     "pass": {
      "op": "<=",
      "value": 2
     },
     "note": "A full stop is under 2 km/h in the last 10 m before the line"
    }
   ]
  },
  {
   "id": "hazard",
   "title": "Pedestrian",
   "weight": 2,
   "items": [
    {
     "id": "reaction",
     "label": "Reaction to the pedestrian",
     "measure": {
      "metric": "reaction",
      "agg": "max"
     },
     "unit": "s",
     "pass": {
      "op": "<=",
      "value": 1.5
     },
     "grade": {
      "target": 0.75,
      "limit": 2.5
     }
    },
    {
     "id": "collisions",
     "label": "Collisions",
     "measure": {
      "metric": "collisions"
     },
     "pass": {
      "op": "==",
      "value": 0
     },
     "critical": true
    }
   ]
  },
  {
   "id": "route",
   "title": "Route",
   "items": [
    {
     "id": "route-time",
     "label": "Time to the destination",
     "measure": {
      "metric": "route-time"
     },
     "unit": "s",
     "pass": {
      "op": "<=",
      "value": 120
     },
     "grade": {
      "target": 80,
      "limit": 160
     }
    }
   ]
  }
 ],
 "display": {
  "subjectTag": "trainee",
  "gauge": {
   "metric": "speed",
   "label": "Speed",
   "unit": "km/h",
   "limit": 50
  },
  "graphs": [
   {
    "metric": "speed",
    "label": "Speed",
    "unit": "km/h",
    "max": 70,
    "limit": 50
   }
  ],
  "stats": [
   {
    "metric": "reaction",
    "label": "Reaction",
    "unit": "s",
    "limit": 1.5
   },
   {
    "metric": "time-over-limit",
    "label": "Over the limit",
    "unit": "s"
   },
   {
    "metric": "stop-speed",
    "label": "At the stop line",
    "unit": "km/h",
    "limit": 2
   }
  ],
  "mapTags": [
   "trainee",
   "pedestrian",
   "traffic"
  ],
  "icons": {
   "send-pedestrian": "person"
  }
 }
}
```

| Field | Meaning |
|---|---|
| `passMark` | the run's pass mark, 0 to 100 (a section may have its own) |
| `sections`, `items` | what the report lists, in order. `weight` weighs an item in its section and a section in the run (default 1) |
| `measure` | `{ "metric": id, "agg"? }` (`agg`: `final` by default, `min`, `max`, `mean`, `count`, `p95`); `{ "expr": "speeding / route-time:final * 60" }` (`+ - * /`, `min()`, `max()`, `abs()`, `id:agg`); or `{ "manual": key }` for something the instructor marks |
| `pass` | `{ "op": "<=", "value": 55 }`; `op` is `<=`, `<`, `>=`, `>`, `==` or `between` (with `max`) |
| `grade` | `{ "target", "limit" }`: points by how close, full at the target and none at the limit. Without it an item is all or nothing |
| `critical` | a failed critical item fails the run, whatever the score |
| `onMissing` | when there is no data at the end (no hazard happened): `fail` (the default) or `skip` |
| `display` | what the console shows: the speed gauge, the graphs under the view, the numbers beside them, what is on the map, palette icons. It never changes a score |

Read Crossing Drive's like this:

- **Speed**, worth twice the others: the highest speed must be at most 55 km/h, with full points at 50 and none at
  65. Time over the limit must be at most 5 s.
- **Stop line:** the slowest speed before the line must be at most 2 km/h.
- **Pedestrian**, worth twice: the reaction must be at most 1.5 s. Any collision fails the run.
- **Route:** at most 120 s to the destination.

While the run is live, the score is provisional. An item with no data yet is pending, and one that can no longer pass
(a maximum already over its limit) is settled. Scoring the recording afterwards gives the same result as the end of
the live run.

> **Ask:** "Write a scorecard for this scenario from its metrics: pass when ..., and check it." Your assistant writes
> it, then checks it with the Training plugin's `training.scorecard_validate` tool. The tool also checks the metric
> ids against the running session.

**By hand,** the Training plugin's `score.mjs` checks a scorecard against your `journal.json`. It names every
problem by its place in the file, and every metric id the journal does not declare. It runs on Node (the engine's
SDK carries its own, in `sdk/node/`), from the plugin's `dist` folder: `sdk/plugins/training/dist/` in an installed
engine, `plugins/training/dist/` in a checkout (after `pnpm --filter ./plugins/training build`). It needs no engine
running and no browser.

```sh
node <plugin dist>/score.mjs --validate training/crossing-drive.scorecard.json --journal journal.json
```

On the template it copied, it names the three `training.*` ids your project does not have:

```text
error /sections/0/items/0/measure/metric scorecard.unknown-metric: no metric "training.collisions" is declared in this project
```

## 6. The report

At the end of a run, the console's **Report** page shows the scorecard filled in, on a sheet you can print, and exports
it as CSV (one row per item), PDF, or xAPI statements written to a file. Nothing is sent anywhere. The Training
plugin's `training.report` tool writes the same files to `output/training/`. The console is the next guide:
[The instructor](TRAINING-INSTRUCTOR.md).

**By hand,** the same `score.mjs` scores a recording ([Reviewing a run](TRAINING-REVIEW.md) says how to record
one) from the file alone. The recording holds the metric values and their definitions, so the numbers are the
console's:

```sh
node <plugin dist>/score.mjs runs/trainee-1.replay --scorecard training/crossing-drive.scorecard.json
node <plugin dist>/score.mjs runs/trainee-1.replay --scorecard training/crossing-drive.scorecard.json --out output/training
```

A careful drive (the sample's self-test, recorded with `--save-replay`) scores:

```text
Crossing Drive · car: PASS 100/100 (pass mark 75)
  Speed  100
    pass    Highest speed in the 50 zone           48.73 km/h   ≤ 55 km/h
    pass    Time over the limit                           0 s   ≤ 5 s
  Stop line  100
    pass    Full stop at the line                      0 km/h   ≤ 2 km/h
  Pedestrian  100
    pass    Reaction to the pedestrian                 0.75 s   ≤ 1.5 s
    pass    Collisions                                      0   = 0
  Route  100
    pass    Time to the destination                    58.7 s   ≤ 120 s
```

`--out` writes the report as JSON, CSV and PDF there (`--formats json,csv,pdf,xapi` to choose), and `--json` prints
it as JSON. It exits 0 when it scored or the scorecard is valid, 1 for a wrong scorecard or argument, 2 for a file it
cannot read.

## Your own measures

| To score | Metric | Then the scorecard |
|---|---|---|
| a reaction time | `interval` from the hazard's kind to the response's kind | `agg: max` (the slowest) or `mean` |
| how long something lasted | `timer` from one kind to another, or `while` a threshold | `final` |
| how often | `counter` of a kind (with `match` on its data) | `final`, `pass` `==` 0 |
| a peak or a low | `gauge` on a signal | `agg: max` or `min` |
| staying within a limit | `threshold` on a gauge, then a `timer` `while` it | `final` |
| a value at a moment | `gauge` with `from` a kind and its `field` | `final` |
| a judgement only a person can make | none | `{ "manual": "key" }` |

Keep the rules in your behaviours: they emit what happened, and the journal, the metrics and the scorecard decide
what it is worth. You can then change a pass mark without touching a script, and score an old recording again with
a new scorecard.