Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,29 @@

All notable changes to RushTI are documented in this file.

## Unreleased

- **Per-workflow `tm1_instance` setting** for results push and auto-load.
Set it inside a JSON taskfile's `settings` block to override the
`settings.ini` default per workflow. Resolution chain (highest wins):
CLI `--tm1-instance` > taskfile `settings.tm1_instance` >
`settings.ini [tm1_integration].tm1_instance` >
`settings.ini [tm1_integration].default_tm1_instance` (deprecated).
RushTI now logs which tier supplied the target at push time.
- **Behavioural change:** `push_results` and `auto_load_results` set
inside taskfile JSON now take effect. They were silently ignored in
the `run` path before this release. If you have divergent values
between taskfile JSON and `settings.ini`, reconcile them before
upgrading — the taskfile value will now win.
- **Soft deprecation:** `settings.ini [tm1_integration].default_tm1_instance`.
Rename to `tm1_instance`. The old key is honoured indefinitely as the
final fallback in the resolution chain; a one-shot `DEPRECATION:`
warning fires at settings load only when it's the value actually being
used.
- **Soft deprecation:** `rushti tasks push --target-tm1-instance`. Use
`--tm1-instance` instead. The legacy flag is aliased and continues to
work; a `DEPRECATION:` warning fires on use.

## Unreleased — `feat/issue-146-detailed-results`

- Add `--detailed-results` for per-execution cube rows (closes #146).
Expand Down
106 changes: 106 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# RushTI — Domain Context

This file captures the canonical vocabulary used in RushTI. Terms here are
load-bearing: they appear in code, docs, settings, and user-facing messages.
When a term feels ambiguous in conversation, anchor it to a definition here.

> Scope: single context. No `CONTEXT-MAP.md` — the whole repository shares
> this vocabulary.

---

## Core terms

### Taskfile
The structured input that drives a `rushti run`. Materialised as either:
- a **JSON task file** (`.json`) parsed by `parse_json_taskfile`, OR
- a **TXT task file** (`.txt`) converted on read by `convert_txt_to_json`, OR
- a **TM1 cube taskfile** read by `read_taskfile_from_tm1` (an MDX query
against the `rushti` cube using the `Input` element of `rushti_run_id`).

The in-memory shape is the `Taskfile` dataclass (`taskfile.py`) — a container
with `version`, `metadata`, `settings`, and `tasks`. Always prefer
**"taskfile"** (one word, lowercase) in docs; **"JSON task file"** is also
acceptable when the format matters.

Do **not** call it `workflow.json` — that name implies the file *is* a
workflow, which conflates the container with one of its metadata fields.

### Workflow
The logical identifier for a run, stored in `metadata.workflow` of a taskfile
and as an element of the `rushti_workflow` dimension. A taskfile contains
tasks *for* one workflow.

Use **"workflow"** as an adjective for scope ("workflow-level setting",
"per-workflow override") rather than as a synonym for "taskfile".

### Task
A single TI process execution inside a taskfile. Each task has:
- `id` — a positive integer string, used as an element name in
`rushti_task_id`.
- `instance` — **the TM1 instance where this task executes** (task-level).
- `process` — the TI process name on that instance.
- `parameters`, `predecessors`, `stage`, `timeout`, etc.

### TM1 instance
A named TM1 server defined in `config.ini`. Three roles can apply
contextually — context disambiguates, not the name:

| Role | Meaning | Where it shows up |
|---|---|---|
| **Source** | Where a taskfile is read *from* (cube source) | `rushti run --tm1-instance X` |
| **Execution target** | Where a task executes | task-level `instance` field |
| **Results target** | Where execution results are pushed | settings.ini or taskfile `tm1_instance`; also CLI `--tm1-instance` as fallback |

The canonical setting key for the results target is `tm1_instance` (in both
`settings.ini [tm1_integration]` and the taskfile `settings` block). The
deprecated alias is `default_tm1_instance`.

There is no separate "source TM1 instance" override — `--tm1-instance` is
both the source and (in absence of higher-precedence overrides) the results
target for a `run` invocation.

### Source vs target — naming convention
RushTI does not prefix instance names with `source_` or `target_`. The
*command* or the *config section* makes the role obvious:
- `rushti run --tm1-instance X` — X is the source for cube-read, and the
fallback results target if nothing else is set.
- `rushti tasks push --tm1-instance X` — X is the target of the push
(canonical form). The legacy alias `--target-tm1-instance` is deprecated.
- `[tm1_integration].tm1_instance` — the section header makes "target" clear.

This convention is intentional: prefixes don't scale across surfaces and
fight against contextual disambiguation. See
[[adr/0001-tm1-instance-resolution]] for the full rationale.

---

## Settings precedence

The effective value for a settings-driven knob is resolved in this order
(highest wins):

1. **CLI arguments** — e.g. `--max-workers`, `--tm1-instance`.
2. **Taskfile settings block** — e.g. `{"settings": {"max_workers": 8}}`
in JSON. Applied via `_apply_json_settings` inside
`get_effective_settings` *after* the taskfile is parsed.
3. **`settings.ini`** — the canonical defaults file at
`config/settings.ini`.
4. **Built-in defaults** — the dataclass field defaults in
`settings.py`.

Three knobs are *not* settings-driven and don't follow this chain:
- Per-task `instance` and `process` — taskfile-only, no fallback.
- TM1 connection parameters — `config.ini` only.

---

## Files of record

| File | Purpose |
|---|---|
| `config/settings.ini` | Execution defaults (user-editable). |
| `config/settings.ini.template` | Documented example, shipped in repo. |
| `config/config.ini` | TM1 connection parameters per instance. |
| `archive/{workflow}/{run_id}.json` | Snapshot of the taskfile actually executed, for audit + DAG reconstruction. |
| `data/rushti_stats.db` | Local SQLite stats database (when `[stats] enabled = true`). |
17 changes: 13 additions & 4 deletions config/settings.ini.template
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@
# To set up TM1 integration:
# 1. Run: python rushti.py build --tm1-instance tm1srv01
# This creates the required dimensions and cube automatically.
# 2. Set push_results = true and configure default_tm1_instance below
# 2. Set push_results = true and configure tm1_instance below
#
# Default: false
# push_results = false
Expand All @@ -126,9 +126,18 @@
# Default: false
# auto_load_results = false

# Default TM1 instance for reading taskfiles and writing results
# Used when --tm1-instance is specified without an instance, and for auto-upload
# Must be defined in config.ini
# TM1 instance used as the results-push target when no higher-precedence
# override is set. Resolved against a 4-tier chain (highest wins):
# 1. CLI --tm1-instance
# 2. Taskfile JSON settings.tm1_instance
# 3. This key
# 4. default_tm1_instance below (deprecated)
# Must be defined in config.ini.
# tm1_instance = tm1srv01

# DEPRECATED: use tm1_instance above. Still honoured indefinitely as the
# final fallback; a one-shot DEPRECATION warning fires at settings load
# *only* when tm1_instance is unset and this key is the value being used.
# default_tm1_instance = tm1srv01

# Default cube name for task definitions and results
Expand Down
4 changes: 2 additions & 2 deletions docs/advanced/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,8 +235,8 @@ rushti tasks push --tasks daily-etl.json --tm1-instance tm1srv01
| Option | Short | Type | Description |
|--------|-------|------|-------------|
| `--tasks` | `-t` | PATH | Local JSON task file to push (*required*) |
| `--tm1-instance` | | STR | Source TM1 instance (if loading from TM1) |
| `--target-tm1-instance` | | STR | Target TM1 instance for the push |
| `--tm1-instance` | | STR | TM1 instance to push the taskfile to. Context disambiguates the role — on `tasks push` this is the destination. |
| `--target-tm1-instance` | | STR | **Deprecated alias** for `--tm1-instance`. Still works; emits a `DEPRECATION:` warning when used. |
| `--settings` | `-s` | PATH | Path to `settings.ini` |

---
Expand Down
12 changes: 9 additions & 3 deletions docs/advanced/settings-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,8 @@ Controls TM1-based read/write integration: reading task files from a TM1 cube an
| `push_results` | bool | `false` | Upload the results CSV to the TM1 Applications folder after each run. The file is named `rushti_{workflow}_{run_id}.csv` (with `.blb` extension for TM1 < v12). |
| `auto_load_results` | bool | `false` | After uploading results (requires `push_results = true`), call the `}rushti.load.results` TI process on the target TM1 instance to load the CSV into the rushti cube. Passes `pSourceFile` and `pTargetCube` parameters. The process must exist on the target instance. |
| `detailed_results` | bool | `false` | Emit one cube row per executed TI when pushing results, instead of summarizing expanded tasks (only meaningful with `push_results = true`). Each row gets a fresh sequential `task_id`; the original IDs are preserved in the new `original_task_id` measure. See [TM1 integration: detailed results](../features/tm1-integration.md#detailed-results). |
| `default_tm1_instance` | str | *(none)* | Default TM1 instance name (from `config.ini`) used for reading task files and writing results. Required when `push_results` is enabled. |
| `tm1_instance` | str | *(none)* | TM1 instance (from `config.ini`) used as the **results-push target** when no higher-precedence override is set. Resolved per [the 4-tier chain](../features/tm1-integration.md#per-workflow-target-instance): CLI `--tm1-instance` > taskfile `settings.tm1_instance` > this key > `default_tm1_instance` (deprecated). Required when `push_results` is enabled and no higher tier is set. |
| `default_tm1_instance` | str | *(none)* | **Deprecated** — use `tm1_instance`. Honoured indefinitely as the final fallback in the resolution chain; a one-shot `DEPRECATION:` warning fires at settings load *only* when it's actually being used (i.e. `tm1_instance` is unset). |
| `default_rushti_cube` | str | `rushti` | Name of the TM1 cube for task definitions and execution results. Created by the `rushti build` command. |
| `default_workflow_dim` | str | `rushti_workflow` | Dimension name for workflow identifiers. |
| `default_task_id_dim` | str | `rushti_task_id` | Dimension name for task sequence elements (1--5000 default elements). |
Expand All @@ -131,7 +132,7 @@ Controls TM1-based read/write integration: reading task files from a TM1 cube an

**Overridable via CLI:** `--detailed-results`

**Overridable via JSON task file:** `push_results`, `auto_load_results`, `detailed_results`
**Overridable via JSON task file:** `push_results`, `auto_load_results`, `detailed_results`, `tm1_instance`

!!! info "Setting Up TM1 Integration"
Run `rushti build --tm1-instance <instance>` to create the required dimensions and cube automatically before enabling `push_results`.
Expand Down Expand Up @@ -293,7 +294,7 @@ Copy this template to `config/settings.ini` and uncomment the settings you want
# To set up TM1 integration:
# 1. Run: rushti build --tm1-instance tm1srv01
# This creates the required dimensions and cube automatically.
# 2. Set push_results = true and configure default_tm1_instance below
# 2. Set push_results = true and configure tm1_instance below
#
# Default: false
# push_results = false
Expand All @@ -308,6 +309,11 @@ Copy this template to `config/settings.ini` and uncomment the settings you want

# Default TM1 instance for reading taskfiles and writing results
# Must be defined in config.ini
# tm1_instance = tm1srv01

# DEPRECATED: use tm1_instance above. Honoured indefinitely as the final
# fallback in the resolution chain; a one-shot DEPRECATION warning fires
# only when this key is the value actually being used.
# default_tm1_instance = tm1srv01

# Default cube name for task definitions and results
Expand Down
58 changes: 55 additions & 3 deletions docs/features/tm1-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ retention_days = 90

[tm1_integration]
push_results = true
default_tm1_instance = tm1srv01
tm1_instance = tm1srv01
default_rushti_cube = rushti
```

Expand All @@ -163,7 +163,7 @@ To automatically load the CSV into the cube after each push, enable `auto_load_r
[tm1_integration]
push_results = true
auto_load_results = true
default_tm1_instance = tm1srv01
tm1_instance = tm1srv01
default_rushti_cube = rushti
```

Expand Down Expand Up @@ -198,14 +198,66 @@ All TM1 integration settings live in the `[tm1_integration]` section of `config/
|---------|---------|-------------|
| `push_results` | `false` | Push a results CSV to TM1 after each run |
| `auto_load_results` | `false` | Automatically call `}rushti.load.results` to load the CSV into the cube |
| `default_tm1_instance` | *(none)* | TM1 instance to use for reading/writing cube data |
| `tm1_instance` | *(none)* | Default results-push target. Resolved against the [4-tier chain](#per-workflow-target-instance) — CLI `--tm1-instance` > taskfile `settings.tm1_instance` > this key > `default_tm1_instance` (deprecated). |
| `default_tm1_instance` | *(none)* | **Deprecated** alias for `tm1_instance`. Honoured as the final fallback; warns at load only when actually used. |
| `default_rushti_cube` | `rushti` | Name of the cube to use |
| `dim_workflow` | `rushti_workflow` | Workflow dimension name |
| `dim_task_id` | `rushti_task_id` | Task ID dimension name |
| `dim_run_id` | `rushti_run_id` | Run ID dimension name |

---

### Per-workflow target instance

The `tm1_instance` value that receives the results push is resolved against a four-tier precedence chain. The first non-empty value wins:

| Tier | Source | Where it's set |
|------|--------|----------------|
| 1 | CLI | `rushti run --tm1-instance <name>` |
| 2 | Taskfile JSON | `settings.tm1_instance` inside the JSON task file |
| 3 | settings.ini (canonical) | `[tm1_integration].tm1_instance` |
| 4 | settings.ini (deprecated) | `[tm1_integration].default_tm1_instance` |

Tier 2 is the new workflow-level override — drop `tm1_instance` into any JSON task file's `settings` block to push that workflow's results to a specific instance without touching `settings.ini`:

```json
{
"version": "2.0",
"metadata": { "workflow": "daily-finance-close" },
"settings": {
"push_results": true,
"auto_load_results": true,
"tm1_instance": "tm1prod"
},
"tasks": [ /* ... */ ]
}
```

When the upload runs, RushTI logs which tier supplied the value:

```
INFO Result upload target: tm1prod (source: taskfile)
```

If all four tiers are empty and `push_results` is on, the run still succeeds — RushTI logs a warning and skips the upload:

```
WARNING push_results enabled but no TM1 instance configured (CLI --tm1-instance, taskfile settings.tm1_instance, or settings.ini tm1_instance).
```

!!! info "Task-level `instance` vs workflow-level `tm1_instance`"
Two `instance`-flavoured fields live at different nesting levels inside a JSON task file:

- **`tasks[*].instance`** — *task-level execution target.* Where the individual TI process runs. Required on every task.
- **`settings.tm1_instance`** — *workflow-level results target.* Where the run's results are pushed at the end. Optional; overrides settings.ini.

They are independent. A workflow can execute its tasks across several instances (one per task) and push the consolidated results back to a single target instance.

!!! warning "Cube-sourced taskfiles have no tier 2"
When a taskfile is loaded from the TM1 cube via `rushti run --tm1-instance X --workflow Y`, the cube schema does not carry a `settings` block — only task definitions. Resolution skips tier 2 and falls through directly to settings.ini (tiers 3 and 4). The CLI value (tier 1) still acts as both the source for the cube read and, in absence of other settings, the fallback results target.

---

## Build Command Options

```bash
Expand Down
Loading
Loading