Events, metrics and scorecards

Day 2, hours 1 to 3 of the training path. By the end, the engine keeps a journal of the events your scenario emits, computes metrics from it, and the Training plugin scores the run against a scorecard: live while it runs, and the same again from its recording.

You need day 1, a scenario that emits events, and an engine with the Training plugin (argus plugin list lists training). The examples are Crossing Drive's, whose finished files are in the sample: journal.json and training/crossing-drive.scorecard.json.

How the pieces fit:

1. Add the Training plugin

Ask: "Add the Training plugin to this project."

By hand, from a terminal in the project folder:

argus plugin add training

On Windows the command is argus.exe in the engine's bin folder (day 1, "Before you start"). It prints what the plugin may do, then what it writes:

argus plugin add: adds "training" (<your project folder>)
  it may:
    net.http.serve (serves pages and an API under /plugins/training/ on the engine's local query server (this machine only)): serves the instructor console page (loopback)
    session.control (pauses, steps and scales time, injects events and takes over entities): pause, inject events, take over vehicles and characters, change the environment
    fs.project.write (writes files in the project folders it names: output/training/): writes reports and exports
  entry: {"id":"training"}
  plugins, in load order: training
  copies training/sample.scorecard.json
wrote training/sample.scorecard.json, argus.project.json

That is "plugins": [{ "id": "training" }] in argus.project.json, and a sample scorecard in training/sample.scorecard.json. It never overwrites a file you have. --dry-run prints the same and writes nothing.

The sample scorecard is a template. Its measures name metrics like training.speed, which your project does not have. Section 5 writes Crossing Drive's own, so delete the template or keep it to read.

In the editor: Project Settings ▸ Plugins, then Add beside Training. It shows the same list and adds the plugin when you confirm. More in Plugins.

2. Declare the events

The journal keeps only the events a kind declares. Kinds go in journal.json at the project root, which applies to every scene. A scenes/<scene>.journal.json beside a scene file applies to that scene only.

{ "kinds": [ ... ], "metrics": [ ... ] }

Crossing Drive declares the eight events its scripts emit for scoring. Its journal.json starts with them:

{"kind": "stop-line", "label": "Stop line", "severity": "info", "roles": ["subject"], "fields": {"stopped": {"type": "bool"}, "min_kmh": {"type": "number", "unit": "km/h"}, "speed_kmh": {"type": "number", "unit": "km/h"}}},
{"kind": "crossing-entered", "label": "Entered the crossing", "severity": "info", "roles": ["subject"], "fields": {}},
{"kind": "destination-reached", "label": "Destination reached", "severity": "info", "roles": ["subject"], "fields": {"seconds": {"type": "number", "unit": "s"}}},
{"kind": "trainee-collision", "label": "Collision", "severity": "critical", "roles": ["subject", "other"], "fields": {"speed_kmh": {"type": "number", "unit": "km/h"}}},
{"kind": "pedestrian-stepped-out", "label": "Pedestrian steps out", "severity": "warn", "roles": ["pedestrian"], "fields": {"distance_m": {"type": "number", "unit": "m"}}},
{"kind": "speed-over", "label": "Over the speed limit", "severity": "warn", "roles": ["subject"], "fields": {"speed_kmh": {"type": "number", "unit": "km/h"}, "limit_kmh": {"type": "number", "unit": "km/h"}}},
{"kind": "brake-applied", "label": "Brake applied", "severity": "info", "roles": ["subject"], "fields": {"speed_kmh": {"type": "number", "unit": "km/h"}}},
{"kind": "send-pedestrian", "label": "Send the pedestrian", "severity": "critical", "injectable": true, "target": "world", "fields": {}}

(goal-reached, which the scripts also emit, is the self-test's. It is not declared, so the journal does not keep it.)

Nothing changes in your scripts. ctx.events.emit("stop-line", { subject: "car", stopped: true, ... }) already sends the event, and with the kind declared the journal keeps it. This is the entry from a real run of the sample:

{ "seq": 2, "tick": 547, "kind": "stop-line", "severity": "info", "entities": { "emitter": "car", "subject": "car" },
  "data": { "min_kmh": 75.6, "speed_kmh": 78.1, "stopped": false, "subject": "car" }, "source": "script" }

A script's event may carry fields its kind does not declare: the journal keeps them as they are. An injected event may carry only declared fields, because the console builds its settings from them: an undeclared one is refused (bad-params).

emitter is the entity whose behaviour emitted it. source is script for your events, command for an instructor's (a pause, an injection, with by, the console that sent it), and engine for what the engine adds itself (metric.crossed, for one). An event of a kind nobody declared is not journaled.

A vehicle on the simulation model (Vehicles) also journals its driver aids and its grip as they start and end, under the engine's own kinds: vehicle.abs.start and vehicle.abs.end for the anti-lock, and the same for vehicle.tc (traction control), vehicle.esc (stability control), vehicle.wheel_lock and vehicle.slide. The vehicle is the subject; a start carries speed_kmh, an end duration_ticks. A metric counts them like any kind ("count": {"kind": "vehicle.abs.start"}, section 3). Crossing Drive's car is the arcade model, which has no aids, so its journal has none of them.

3. Metrics

A metric is a number the engine computes every tick, from the journal and from entities. It is computed inside the simulation, so a recording played back gives the same values to the last bit.

Any metric can have per: "<role>" (one value per entity in that role: one per trainee, with several trainees; a gauge on a tag needs it), between: { start, end } (counted only between two kinds) and reset_on: <kind>.

Crossing Drive's eight, in the order the engine computes them. A metric can use only the ones before it:

{"id": "speed", "label": "Speed", "unit": "km/h", "kind": "gauge", "per": "subject", "signal": {"tag": "trainee", "field": "speed", "scale": 3.6}, "every": 10},
{"id": "over-limit", "label": "Over the limit", "kind": "threshold", "of": "speed", "over": 52, "for_ticks": 30, "per": "subject"},
{"id": "time-over-limit", "label": "Time over the limit", "unit": "s", "kind": "timer", "while": "over-limit", "per": "subject"},
{"id": "speeding", "label": "Times over the limit", "kind": "counter", "count": {"kind": "speed-over"}},
{"id": "stop-speed", "label": "Slowest before the stop line", "unit": "km/h", "kind": "gauge", "from": {"kind": "stop-line"}, "field": "min_kmh"},
{"id": "reaction", "label": "Reaction to the pedestrian", "unit": "s", "kind": "interval", "from": {"kind": "pedestrian-stepped-out"}, "to": {"kind": "brake-applied"}, "close": "oldest"},
{"id": "collisions", "label": "Collisions", "kind": "counter", "count": {"kind": "trainee-collision"}},
{"id": "route-time", "label": "Time to destination", "unit": "s", "kind": "gauge", "from": {"kind": "destination-reached"}, "field": "seconds"}

• speed samples the speed of every entity tagged trainee every 10 ticks, one value per trainee, and scale: 3.6 turns the engine's metres a second into km/h.
• over-limit is 1 while speed has been over 52 km/h for half a second (30 ticks), per trainee.
• time-over-limit counts the seconds over-limit is 1.
• speeding counts the speed-over events, and collisions the trainee-collision ones.
• stop-speed takes min_kmh from the stop-line event: the slowest the car went in the 10 m before the line.
• reaction is the time from the pedestrian stepping out to the next press of the brake.
• route-time takes seconds from destination-reached.

Each metric also keeps aggregates over the run, which a scorecard reads: final, min, max, mean, count and p95.

4. Check the journal and the metrics

Serve the scenario and ask it:

argus run . --headless --serve --port 7795 --ws-port 7796
# in another terminal, while it runs:
curl http://127.0.0.1:7795/journal
curl http://127.0.0.1:7795/metrics
curl "http://127.0.0.1:7795/journal/kinds?injectable=true"    # the instructor's palette
curl "http://127.0.0.1:7795/metrics/speed/series?entity=car"  # a metric's history (a per-trainee one needs entity)

--ws-port is the session's live stream. Give it a free port of its own, or it takes 7701.

--headless runs with no window and no input, so the car sits still. To see the events of a drive, drive it in the editor with Play and watch the console (The instructor). Or run the self-test and keep its recording:

argus run . --headless --self-test crossing-drive --save-replay run.replay

The recording is JSON lines, about one a tick. A line with "j" holds that tick's journal entries, and one with "m" the metric values that changed, each as [metric, entity, bits]: the 64-bit floating-point value's bits in hex, so a playback compares them exactly. In Python, struct.unpack(">d", bytes.fromhex(bits))[0] reads one. The review (next guides) shows them as numbers.

If a metric never changes, read the first lines the run prints. A definition the engine refuses is named there with the reason (journal-defs-invalid), for example a metric that uses one defined after it.

5. The scorecard

A scorecard says what is scored, how it is measured, and what passes. It is one file matching **/*.scorecard.json, usually under training/. The console uses the scorecard whose scenario names the running scene.

Crossing Drive's:

{
 "schema": 1,
 "id": "crossing-drive",
 "title": "Crossing Drive",
 "scenario": "scenes/main.scene.json",
 "passMark": 75,
 "sections": [
  {
   "id": "speed",
   "title": "Speed",
   "weight": 2,
   "items": [
    {
     "id": "max-speed",
     "label": "Highest speed in the 50 zone",
     "measure": {
      "metric": "speed",
      "agg": "max"
     },
     "unit": "km/h",
     "pass": {
      "op": "<=",
      "value": 55
     },
     "grade": {
      "target": 50,
      "limit": 65
     }
    },
    {
     "id": "time-over",
     "label": "Time over the limit",
     "measure": {
      "metric": "time-over-limit"
     },
     "unit": "s",
     "pass": {
      "op": "<=",
      "value": 5
     },
     "grade": {
      "target": 0,
      "limit": 20
     }
    }
   ]
  },
  {
   "id": "junction",
   "title": "Stop line",
   "items": [
    {
     "id": "full-stop",
     "label": "Full stop at the line",
     "measure": {
      "metric": "stop-speed",
      "agg": "max"
     },
     "unit": "km/h",
     "pass": {
      "op": "<=",
      "value": 2
     },
     "note": "A full stop is under 2 km/h in the last 10 m before the line"
    }
   ]
  },
  {
   "id": "hazard",
   "title": "Pedestrian",
   "weight": 2,
   "items": [
    {
     "id": "reaction",
     "label": "Reaction to the pedestrian",
     "measure": {
      "metric": "reaction",
      "agg": "max"
     },
     "unit": "s",
     "pass": {
      "op": "<=",
      "value": 1.5
     },
     "grade": {
      "target": 0.75,
      "limit": 2.5
     }
    },
    {
     "id": "collisions",
     "label": "Collisions",
     "measure": {
      "metric": "collisions"
     },
     "pass": {
      "op": "==",
      "value": 0
     },
     "critical": true
    }
   ]
  },
  {
   "id": "route",
   "title": "Route",
   "items": [
    {
     "id": "route-time",
     "label": "Time to the destination",
     "measure": {
      "metric": "route-time"
     },
     "unit": "s",
     "pass": {
      "op": "<=",
      "value": 120
     },
     "grade": {
      "target": 80,
      "limit": 160
     }
    }
   ]
  }
 ],
 "display": {
  "subjectTag": "trainee",
  "gauge": {
   "metric": "speed",
   "label": "Speed",
   "unit": "km/h",
   "limit": 50
  },
  "graphs": [
   {
    "metric": "speed",
    "label": "Speed",
    "unit": "km/h",
    "max": 70,
    "limit": 50
   }
  ],
  "stats": [
   {
    "metric": "reaction",
    "label": "Reaction",
    "unit": "s",
    "limit": 1.5
   },
   {
    "metric": "time-over-limit",
    "label": "Over the limit",
    "unit": "s"
   },
   {
    "metric": "stop-speed",
    "label": "At the stop line",
    "unit": "km/h",
    "limit": 2
   }
  ],
  "mapTags": [
   "trainee",
   "pedestrian",
   "traffic"
  ],
  "icons": {
   "send-pedestrian": "person"
  }
 }
}

Read Crossing Drive's like this:

• Speed, worth twice the others: the highest speed must be at most 55 km/h, with full points at 50 and none at
  1. Time over the limit must be at most 5 s.
• Stop line: the slowest speed before the line must be at most 2 km/h.
• Pedestrian, worth twice: the reaction must be at most 1.5 s. Any collision fails the run.
• Route: at most 120 s to the destination.

While the run is live, the score is provisional. An item with no data yet is pending, and one that can no longer pass (a maximum already over its limit) is settled. Scoring the recording afterwards gives the same result as the end of the live run.

Ask: "Write a scorecard for this scenario from its metrics: pass when ..., and check it." Your assistant writes it, then checks it with the Training plugin's training.scorecard_validate tool. The tool also checks the metric ids against the running session.

By hand, the Training plugin's score.mjs checks a scorecard against your journal.json. It names every problem by its place in the file, and every metric id the journal does not declare. It runs on Node (the engine's SDK carries its own, in sdk/node/), from the plugin's dist folder: sdk/plugins/training/dist/ in an installed engine, plugins/training/dist/ in a checkout (after pnpm --filter ./plugins/training build). It needs no engine running and no browser.

node <plugin dist>/score.mjs --validate training/crossing-drive.scorecard.json --journal journal.json

On the template it copied, it names the three training.* ids your project does not have:

error /sections/0/items/0/measure/metric scorecard.unknown-metric: no metric "training.collisions" is declared in this project

6. The report

At the end of a run, the console's Report page shows the scorecard filled in, on a sheet you can print, and exports it as CSV (one row per item), PDF, or xAPI statements written to a file. Nothing is sent anywhere. The Training plugin's training.report tool writes the same files to output/training/. The console is the next guide: The instructor.

By hand, the same score.mjs scores a recording (Reviewing a run says how to record one) from the file alone. The recording holds the metric values and their definitions, so the numbers are the console's:

node <plugin dist>/score.mjs runs/trainee-1.replay --scorecard training/crossing-drive.scorecard.json
node <plugin dist>/score.mjs runs/trainee-1.replay --scorecard training/crossing-drive.scorecard.json --out output/training

A careful drive (the sample's self-test, recorded with --save-replay) scores:

Crossing Drive · car: PASS 100/100 (pass mark 75)
  Speed  100
    pass    Highest speed in the 50 zone           48.73 km/h   ≤ 55 km/h
    pass    Time over the limit                           0 s   ≤ 5 s
  Stop line  100
    pass    Full stop at the line                      0 km/h   ≤ 2 km/h
  Pedestrian  100
    pass    Reaction to the pedestrian                 0.75 s   ≤ 1.5 s
    pass    Collisions                                      0   = 0
  Route  100
    pass    Time to the destination                    58.7 s   ≤ 120 s

--out writes the report as JSON, CSV and PDF there (--formats json,csv,pdf,xapi to choose), and --json prints it as JSON. It exits 0 when it scored or the scorecard is valid, 1 for a wrong scorecard or argument, 2 for a file it cannot read.

Your own measures

Keep the rules in your behaviours: they emit what happened, and the journal, the metrics and the scorecard decide what it is worth. You can then change a pass mark without touching a script, and score an old recording again with a new scorecard.