Automations - first roundtrip for forecasts

[Flix6x]

Description

First roundtrip for automations (milestone 1 of #2288): recurring tasks defined per asset — for now, computing forecasts.

• Data model: new Automation model (id, created_at, asset_id, type, name, cronstr, active, generator_id, parameters JSONB), with an __acl__ that lets account admins and consultants update/delete, and a migration. New dependencies: croniter (cron matching/validation) and cron-descriptor (natural-language recurrence descriptions).
• CLI: flexmeasures add automation (active by default; validates parameters with the forecast parameter schema; stores the forecaster config on a data source), flexmeasures edit automation (name, cron string, activation status) and flexmeasures delete automation. All record to the asset's audit log.
• Running automations: flexmeasures jobs run-automations queues forecasting jobs for all automations due this minute — to be run once per minute, e.g. via cron. A Redis-based guard prevents duplicate runs within the same minute.
• Job provenance: data generators record how their queued jobs got created (CLI, API or automation) as job meta data; the status page's jobs table shows this in a new Created Via column.
• API: GET /assets/<id>/automations (list, without generator/parameters) and GET /assets/<id>/automations/<automation_id> (details incl. parameters, generator info and job counts per status), documented in the OpenAPI specs.
• UI: /assets/<id>/automations page with a tabbed view (Schedules and Reports tabs prepared but deactivated), listing Name, Created At (human-readable, ISO string in title tag), Active, Recurrence (human-readable, cron string in title tag), plus per-row job counts and a Details modal loaded asynchronously. Added to the breadcrumbs dropdown; links to the status page.

Look & feel

See the new Automations page under an asset's breadcrumbs dropdown.

How to test

flexmeasures db upgrade
flexmeasures add automation --asset <id> --name "Daily PV forecasts" --cron "0 6 * * *" --sensor <sensor-id>
flexmeasures jobs run-automations  # run once per minute via cron

Tests: pytest flexmeasures/cli/tests/test_automations.py flexmeasures/api/v3_0/tests/test_automations_api.py flexmeasures/ui/tests/test_asset_crud.py

Further Improvements

Per #2288: schedules and reports as automations, CRUD for automations in the UI (POST/PATCH/DELETE endpoints).

• update pr description after fixes commits pushed

Closes #2288

🤖 Generated with Claude Code

https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	cron-descriptor@​2.1.0

View full report

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #33929564 | 📁 Comparing 318cd91 against latest (60cfda8)

  🔍 Preview build  

111 files changed · + 19 added · ± 92 modified

+ Added

• _autodoc/COUPLING.html
• _autodoc/COUPLING_COEFFICIENT.html
• _autodoc/GROUP.html
• _autodoc/INFLEXIBLE_CONSUMPTION.html
• _autodoc/INFLEXIBLE_PRODUCTION.html
• _autodoc/OPERATION_MODES.html
• _autosummary/flexmeasures.api.common.rate_limiting.html
• _autosummary/flexmeasures.data.models.automations.html
• _autosummary/flexmeasures.data.models.planning.devices.html
• _autosummary/flexmeasures.data.models.planning.highspy_optimization.html
• and 9 more...

± Modified

• changelog.html
• configuration.html
• genindex.html
• getting-started.html
• http-routingtable.html
• index.html
• py-modindex.html
• _autodoc/COMMITMENTS.html
• _autodoc/INFLEXIBLE_DEVICE_SENSORS.html
• _autodoc/SOC_MAXIMA.html
• and 82 more...

[BelhsanHmida]

I checked the current behavior and considered the trade-offs.

1. Execution semantics

The current runner evaluates cron expressions in the global FLEXMEASURES_TIMEZONE. A skipped DST hour is missed, while a repeated hour can run twice. There is no downtime catch-up.

I recommend storing an IANA timezone per automation, defaulted at creation to FLEXMEASURES_TIMEZONE. For forecasts, I would run once per local wall-clock occurrence and coalesce missed occurrences into one catch-up run rather than replaying every stale forecast after downtime. Other automation types may later need configurable skip, run-once, or replay policies.

This requires persistent scheduling state, so I suggest handling it in a follow-up issue.

2. Failure semantics

Retrying immediately is unsafe because an exception may occur after some jobs have already been queued. Until runs and jobs have durable occurrence IDs and idempotency protection, the current at-most-once guard is the safer milestone-one behavior.

I suggest a follow-up issue for durable automation-run records, partial-failure states and safe retries.

3. Cross-asset targets

The current CLI only warns about a target sensor on another asset. I recommend rejecting output sensors that do not belong to the automation asset or one of its descendants. Arbitrary cross-organisation output targets should not be allowed.

Regressors are inputs and could remain cross-asset when readable, but output sensors need the appropriate write boundary. Run statistics should primarily belong to the automation asset; individual jobs can remain visible under their target sensor subject to normal permissions.

I think this validation belongs in the current PR because it protects a permission boundary.

4. Generator lifecycle

I verified that beliefs and annotations reference their DataSource with restrictive foreign keys: PostgreSQL rejects deleting a DataSource while those records reference it.

Automations currently differ: generator_id uses ON DELETE SET NULL, which can leave an active automation without a generator and make every future run fail.

I recommend matching the existing belief/annotation policy: make generator_id required and reject DataSource deletion while an automation references it. The user should delete or repoint the automation first. I think this also belongs in the current PR.

Proposed scope: address target authorization and generator deletion integrity here; create follow-ups for timezone/catch-up and safe retry semantics. Does that split look right?

[Flix6x]

Proposed scope: address target authorization and generator deletion integrity here; create follow-ups for timezone/catch-up and safe retry semantics. Does that split look right?

Yes, that looks right. For 1 and 2, please open the issues and cross-reference them. For 3 and 4, remarks below.

3. Cross-asset targets Cross-account targets

The current CLI only warns about a target sensor on another asset. I recommend rejecting output sensors that do not belong to the automation asset or one of its descendants. Arbitrary cross-organisation output targets should not be allowed.

Regressors are inputs and could remain cross-asset when readable, but output sensors need the appropriate write boundary. Run statistics should primarily belong to the automation asset; individual jobs can remain visible under their target sensor subject to normal permissions.

I think this validation belongs in the current PR because it protects a permission boundary.

I seems reasonable to mandate that automations can only register beliefs on sensors that belong to the asset that the automation is registered on, or to an asset further down the tree (e.g. to a child or grandchild asset).

But note that tree structure and ownership are separate concepts! And it is ownership that is my primary concern. Tree structure is topology only, it grants nothing.

Here's an example to reason about:

flowchart TD
    classDef ownerless fill:#495057,stroke:#212529,color:#fff,stroke-dasharray: 5 3
    classDef foundation fill:#2d6a4f,stroke:#14342b,color:#fff
    classDef participant fill:#7f4f24,stroke:#3f2712,color:#fff
    classDef sensor fill:#f8f9fa,stroke:#adb5bd,color:#212529,stroke-dasharray: 3 3

    subgraph PUBLIC["🌍 No owner — readable by every logged-in user"]
        BZ["Bidding zone<br/><i>NL</i>"]
        BZ -.-> S1(["day-ahead price"]):::sensor
        BZ -.-> S2(["imbalance price"]):::sensor
    end

    subgraph FND["🤝 Account: Energy Community Foundation"]
        GRID["Grid connection point<br/><i>collective connection</i>"]
        PV["Shared solar park"]
        BAT["Community battery"]
        GRID --> PV
        GRID --> BAT
        GRID -.-> S3(["net exchange"]):::sensor
        BAT -.-> S4(["SoC / power"]):::sensor
    end

    subgraph P1["🏠 Account: Participant A"]
        HA["Household A"]
        EVA["EV charger"]
        HPA["Heat pump"]
        HA --> EVA
        HA --> HPA
        EVA -.-> S5(["power"]):::sensor
    end

    subgraph P2["🏠 Account: Participant B"]
        HB["Household B"]
        PVB["Rooftop PV"]
    end

    BZ ==>|"child asset,<br/>now owned"| GRID
    GRID ==>|"child asset,<br/>other account"| HA
    GRID ==>|"child asset,<br/>other account"| HB

    class BZ ownerless
    class GRID,PV,BAT foundation
    class HA,EVA,HPA,HB,PVB participant

Loading

Here are two permissions scenarios:

Community is a consultancy to its participants

flowchart TD
    classDef acct fill:#3d348b,stroke:#1d1846,color:#fff
    classDef asset fill:#f8f9fa,stroke:#adb5bd,color:#212529
    classDef ownerless fill:#495057,stroke:#212529,color:#fff,stroke-dasharray: 5 3

    subgraph ACCOUNTS["Account graph — where authority lives"]
        FA["Foundation account<br/><i>role: consultant</i>"]:::acct
        AA["Participant A account"]:::acct
        AB["Participant B account"]:::acct
        AA -->|"consultancy_account_id"| FA
        AB -->|"consultancy_account_id"| FA
    end

    subgraph ASSETS["Asset tree — where topology lives"]
        BZ2["Bidding zone<br/><i>no owner</i>"]:::ownerless
        GRID2["Community assets"]:::asset
        HA2["Household A"]:::asset
        HB2["Household B"]:::asset
        BZ2 --> GRID2
        GRID2 --> HA2
        GRID2 --> HB2
    end

    FA ==>|"CRUD (owner)"| GRID2
    FA ==>|"read + update + create-children<br/>via consultancy"| HA2
    FA ==>|"read + update + create-children<br/>via consultancy"| HB2
    AA ==>|"CRUD (owner)"| HA2
    AB ==>|"CRUD (owner)"| HB2
    AA -.->|"❌ no read — not the owner,<br/>not a consultant"| GRID2
    AA -.->|"❌ no read"| HB2
    AA -->|"read (ownerless)"| BZ2
    AB -->|"read (ownerless)"| BZ2
    FA -->|"read (ownerless)"| BZ2

Loading

Fully sovereign participants

flowchart TD
    classDef acct fill:#3d348b,stroke:#1d1846,color:#fff
    classDef asset fill:#f8f9fa,stroke:#adb5bd,color:#212529
    classDef ownerless fill:#495057,stroke:#212529,color:#fff,stroke-dasharray: 5 3

    subgraph ACCOUNTS["Account graph — no consultancy edges at all"]
        FA["Foundation account"]:::acct
        AA["Participant A account"]:::acct
        AB["Participant B account"]:::acct
    end

    subgraph ASSETS["Asset tree — topology only, grants nothing"]
        BZ2["Bidding zone<br/><i>no owner</i>"]:::ownerless
        GRID2["Community assets"]:::asset
        HA2["Household A"]:::asset
        HB2["Household B"]:::asset
        BZ2 --> GRID2
        GRID2 -->|"cross-account edge —<br/>site admin / CLI only"| HA2
        GRID2 -->|"cross-account edge —<br/>site admin / CLI only"| HB2
    end

    FA ==>|"CRUD (owner)"| GRID2
    AA ==>|"CRUD (owner)"| HA2
    AB ==>|"CRUD (owner)"| HB2

    FA -.->|"❌ no read, despite being the tree parent"| HA2
    FA -.->|"❌ no read"| HB2
    AA -.->|"❌ no read"| GRID2
    AA -.->|"❌ no read"| HB2

    FA -->|"read"| BZ2
    AA -->|"read"| BZ2
    AB -->|"read"| BZ2

Loading

4. Generator lifecycle

I recommend matching the existing belief/annotation policy: make generator_id required and reject DataSource deletion while an automation references it. The user should delete or repoint the automation first. I think this also belongs in the current PR.

I agree.

[BelhsanHmida]

Thanks for the detailed response, I pushed the changes addressing the agreed current-PR scope.

Follow-up issues

I created separate follow-up issues for:

• per-automation timezones, DST behavior and downtime catch-up: Add per-automation timezones and catch-up semantics #2392
• durable automation-run records, partial failures and safe retries: Add durable automation-run records and safe retry semantics #2393

Until safe retries have durable occurrence IDs and idempotency protection, the runner retains its at-most-once guard after a failed attempt.

Output target scope

The pushed changes now:

• allow output sensors on the automation asset, its children or its grandchildren;
• reject ancestor and unrelated output sensors;
• validate the effective sensor-to-save, when supplied;
• repeat the validation immediately before every unattended execution; and
• prevent an automation from running if its target asset has since been moved outside the subtree.

One ownership question remains. Manual testing confirmed that an automation owned by organisation A can currently target a descendant sensor owned by unrelated organisation B, even when A has no consultancy relationship with B. The topology check passes because the target is a descendant, but topology itself grants no permissions.

For unattended execution, should authority come from:

1. the automation asset’s owning organisation and its current consultancy relationships; or
2. the individual creator’s identity and that user’s current permissions?

I recommend the first option. The automation belongs to the asset/organisation and should not depend on whether its original creator remains employed or keeps the same role. Current consultancy relationships should be checked again on every run, so revoking access immediately prevents subsequent writes.

Does that match the intended permissions model? Once confirmed, I will add the corresponding service-level permission check and cross-organisation regression test.

Generator lifecycle

The pushed changes also make generator handling consistent with beliefs and annotations:

• generator_id is required;
• PostgreSQL rejects deleting a referenced DataSource;
• the automation must first be deleted or repointed; and
• the generator can be deleted after its referencing automations are removed.

[Flix6x]

@BelhsanHmida — Felix went through this PR and we picked up his review points, so here is what changed since you last saw it. Could you review these commits, and then take the PR over again?

What we contributed (on top of your branch, after merging it in):

• Data generators now report their sensors. DataGenerator got input_sensors / output_sensors properties (implemented for Forecaster: the sensor to forecast plus its regressors, and the sensor to save to), mirrored on Automation. This is what the UI links use, and it is also the hook for permission checks (see CRUD for automations in the API and UI #2294).
• Automation details link to the sensors it feeds. The details modal now shows "Reads from" and "Writes to", linking to /sensors/<id>?source=<generator id>, and the sensor page pre-selects that data source in its statistics panel. GET /assets/<id>/automations/<automation_id> returns input_sensors and output_sensors for this.
• The sensor page lists the automations feeding it, permission-filtered, linking to the automations page of their asset. To keep that cheap, only automations on the sensor's asset or an ancestor are considered — which is where an automation writing to it must live, per your validate_forecast_output_scope.
• A data source can be inspected from the sensor page: an info button next to the source selector opens a modal with the full record, including the attributes where data generators store their configuration. Backed by a new GET /api/v3_0/sources/<id>.
• The automations listing is sortable (on the ISO timestamp, the activation status and the cron string, rather than on the rendered text), newest first.
• --cron now defaults to daily ("0 0 * * *"), so flexmeasures add automation no longer requires it.
• flexmeasures jobs run-automations no longer logs "Starting Train-Predict Pipeline" per automation: run() logged that even when it only queues the cycles, which the workers then announce themselves.

One bug fix worth a closer look: the --source guard compared against the assembled config, which always contains the schema defaults of the list-valued options — so any use of --source was rejected. It now detects the conflicting options from click's parameter sources, so --source on its own works again while explicitly given configuration options still abort, and the error names them.

Note on the timezone review point: it is already covered by #2396 (which also adds the scheduling cursor, catch-up and DST handling), so we deliberately left it out here to avoid a second, conflicting migration.

Heads-up for #2396: it will hit small conflicts in asset_automations.html and data/services/automations.py when it merges this base.

🤖 Generated with Claude Code