Core Utils Runner
The Core Utils Runner provides foundational utilities for data pipelines within HEAT.
It focuses on agnostic conversions and remapping operations that standardize data before analytics or dashboard aggregation.
Available Node Templates
1. tabular-remap (Transform Node)
This node remaps, transforms, and restructures tabular data according to configuration rules provided at runtime.
It supports column renaming, derived fields through Python expressions, and creation of constant or computed columns.
Typical usage: Normalize disparate CSV inputs into a consistent schema before feeding them into downstream processing or aggregation nodes. See the tabular-remap guide.
2. hydrate-protobuf (Transform Node)
Extracts a chosen protobuf type from HEAT capture bundles and outputs JSON, CSV, or raw binary. Supports multiple inputs and optional aggregation of simulation frames.
Typical usage: Turn protobuf payloads (e.g. GazeComponent, OpenDIS PDU, FolderUploadData) into structured data for downstream nodes. See the hydrate-protobuf guide.
3. tabular-query (Transform Node)
Loads tabular input into an in-memory SQLite table and runs configured SQL queries; returns a JSON object of result tables. Optional time-column and base-capture-time settings allow deriving elapsed_ms and captured_at for time-based queries.
Typical usage: Run SQL over CSV/TSV data without leaving the pipeline. Runner-level documentation is in
runners/core-utils/docs/tabular_query.md.
4. json-template (Transform Node)
Builds an output JSON object from a template: string leaves in the template are JMESPath expressions evaluated against the input; objects and arrays are recursed; literals are copied as-is.
Typical usage: Reshape or extract JSON from upstream nodes into the structure required by dashboards or other consumers. See the json-template guide.
5. json-flatten (Transform Node)
Flattens JSON into CSV or TSV. Optionally targets an array via JMESPath (root); nested keys become dotted columns; array handling is configurable (index or join).
Typical usage: Convert nested JSON from APIs or transforms into tabular form for analytics or tabular-remap. See the json-flatten guide.
Integration in a Session Template
- Insert the appropriate Core Utils node (e.g.
tabular-remap,hydrate-protobuf,json-template,json-flatten, ortabular-query) after your data source or upstream node. - Configure the node using its configuration schema (see the guides linked above).
- Connect downstream nodes that consume the transformed output (analytical or dashboard nodes).
Supported Formats
| Input Type | Output Type | Notes |
|---|---|---|
| CSV | CSV / TSV | Auto-detects delimiters (auto mode) |
| TSV | CSV / TSV | Preserves or converts delimiter as configured |
All transformations occur in-memory and respect field quoting per RFC4180.