Fix: CEM work session - #188
Draft
Flix6x wants to merge 186 commits into
Draft
Conversation
…, consolidating all dependencies into pyproject.toml and a single uv.lock file. This simplifies and speeds up the development setup greatly. Changes: - Switch build backend from setuptools to hatchling - Remove setup.cfg, tox.ini, .isort.cfg and setup.py in favour of .flake8 and pyproject.toml - Add Poethepoet tasks - Upgrade main Python version (CI/CD, .python-version, etc.) to 3.12 - Deprecate Python 3.9 - Add .python-version for consistent Python version management - Create separate Dockerfile for flexmeasures-client - Replace pip-tools with uv in all CI/CD workflows - Remove ci/run_mypy.sh in favour of Poethepoet task - Update documentation Signed-off-by: Stijn van Houwelingen <teadrinkingprogrammer@github.io>
Signed-off-by: Stijn van Houwelingen <teadrinkingprogrammer@github.io>
Signed-off-by: Stijn van Houwelingen <teadrinkingprogrammer@github.io>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
This reverts commit d63afe5.
…connection shuts down Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
This reverts commit 4799585.
Signed-off-by: F.N. Claessen <claessen@seita.nl>
… is missing Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
This reverts commit 5328de9.
Signed-off-by: F.N. Claessen <claessen@seita.nl>
…) and make system_description_id optional Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
This reverts commit 374e44a.
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
…andlers accordingly Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
…, so scheduling jobs run in parallel instead of serializing through a single worker
…stalling the timestep Observed a pandas 'cannot reindex on an axis with duplicate labels' error from a scheduling job during a multi-day, frequently-re-triggered (implementation_horizon_hours=3) run; re-running the exact same job manually succeeded immediately, confirming it's transient. Without a retry, trigger_schedule's fire-and-forget task just logs and drops the exception, so the RM never receives an instruction for that timestep and polls forever. Retries up to 3 times on a ValueError containing 'Scheduling job failed' before giving up for real.
The base override now runs without access to the private ems repo (needed for single-house simulations by collaborators without that repo); community runs add ../ems/docker-compose.community.yml as a third compose file.
Builds the new flex-model "operation-modes" field from the FRBC system description's operation-mode element power ranges (only when they actually restrict the range), so FlexMeasures schedules only power values the device can run at. Requires flexmeasures feat/2113-operation-mode-power-bands.
A plain disconnect (RMs reconnect at every replan boundary) previously left the producer and watchdog tasks and the CEM itself running forever; the old session lived on as a zombie able to keep posting SoC and triggering schedules for an asset the RM no longer implements.
At startup, the CEM's first trigger_schedule can beat the controller's initial battery-SoC post (a separate client writing to the flex-model's state-of-charge sensor). FlexMeasures then rejects the trigger with a 422 "No recent state-of-charge value found", and since this whole call runs as a fire-and-forget task, that single failure permanently stalled the house at its barrier. Treat it as transient alongside "Scheduling job failed" (5 attempts, 2s backoff) - it resolves itself once the SoC post lands moments later. Heavier stacks lose this race more often.
…ted time The CEM now creates a dedicated 'measured-power' sensor (kW, consumption-positive) per site, separate from the StorageScheduler 'power' schedule sensor, and maps ELECTRIC.POWER.L1 measurements to it (replacing a hardcoded fallback sensor id). Measurements are binned and posted at their own SIMULATED timestamp instead of wall-clock time, and posted directly per incoming value (the RM sends one aggregated value per interval, so the old 5-minute wall-clock buffered-averaging would have suppressed nearly every post in a co-simulation). Values are posted in Watts (S2 PowerMeasurement convention) and converted to the kW sensor by FlexMeasures.
…'t crash CEM.close() calls close() on every registered control-type handler when a websocket tears down, but only FRBCTunes implemented it - FRBCSimple (and any other handler) raised AttributeError, which killed the CEM's request handler and left it unable to serve the next RM connection (houses then hang forever waiting for a schedule). Add a default no-op close() on the ControlTypeHandler base class.
Images built from FlexMeasures' uv-based Dockerfile put a pip-less /app/.venv first on PATH while bare `pip` is the system one under /usr/local, so packages installed at container start were invisible to the runtime interpreter (ModuleNotFoundError: aiohttp) on any freshly built image - notably Samuel's Windows setups - while long-cached single-Python images kept working. Use `python3 -m pip` throughout (with an ensurepip bootstrap for pip-less venvs) and exec the server with that same python3. Also set the generic SETUPTOOLS_SCM_PRETEND_VERSION alongside the client-specific one for the git-less editable install. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LQFkeGxDwErydUV5ZmHmki
An S2 PowerRange runs from start_of_range (lower bound) to end_of_range (upper bound), but the capacity derivation introduced in 9f20ab4 (March 2026) read them the other way around: production-capacity became the smallest upper bound across operation modes (e.g. 2.5 kW for a consumption-only heat pump) and consumption-capacity the largest lower bound. The scheduler consequently discharged the heat pump's thermal storage as if it were an electric battery, and capped its consumption below the device's real maximum. Derive the overall [min, max] power range instead, with production only where ranges extend below zero. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LQFkeGxDwErydUV5ZmHmki
… realizations Belief time (defect 4b): PowerMeasurements posted by the CEM now carry a simulated prior (the binned interval's end), so realized apartment power is stamped at simulation time instead of wall-clock 2026, and the UI horizon view no longer shows data "recorded in 2026". Timezone (defect 2): naive S2 measurement timestamps are now localized as Europe/Amsterdam (was UTC), matching the community orchestrator, RM and controller which all localize the same naive sim times as Europe/Amsterdam. Live apartment realizations (defect 4a): besides the dedicated measured-power sensor, the CEM now also mirrors the realized apartment aggregate onto the apartment's flex-context "aggregate-power" sensor, resolved lazily from the asset's flex_context (it is attached by the community runner only after the CEM connects, so it appears from step 1). Realized posts stay distinguishable from the StorageScheduler schedule data on that same sensor by their own source and simulated belief time. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LQFkeGxDwErydUV5ZmHmki
The RM sends one PowerMeasurement per 15-minute interval (stamped at interval start), but the posting path binned and stamped priors with the 5-minute _minimum_measurement_period, yielding belief times 10 minutes before each event's end - an ex-ante horizon on measurements. Bin and stamp at the measured-power sensor's 15-minute event resolution instead, so each realized value is believed exactly when its interval elapses (zero horizon). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LQFkeGxDwErydUV5ZmHmki
An explicit production-capacity of 0 kW gets softened by FlexMeasures' relax-constraints defaults (100 EUR/kW device breach price), so under a tight community capacity the scheduler chose to 'produce' 1.764 kW from a heat pump rather than pay the 10000 EUR/kW site breach. Setting is_strictly_non_positive on the FRBC power sensor keeps production hard-bounded at zero regardless of relaxation settings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LQFkeGxDwErydUV5ZmHmki
Later polling steps slept for polling_interval * 2**step seconds (minutes), dozing far past the moment a schedule became available - and taking just as long to conclude it never would. Cap each sleep at 10s so both success and failure are detected promptly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: F.N. Claessen <claessen@seita.nl>
With backoff sleeps capped at 10s, 10 polling steps amounted to only ~90s of patience - less than a time-limited (120s) scheduling job - so clients gave up on schedules that were about to arrive. 30 steps puts the ceiling above polling_timeout (200s), which is the intended budget. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: F.N. Claessen <claessen@seita.nl>
The scalar soc-min/soc-max are hard bounds server-side; only the soc-minima/maxima profiles are softened by relax-soc-constraints. The FRBC fill range is a comfort band and the realized state can drift outside it (community steering keeping the heat pump off drains the buffer below the comfort floor), which made every subsequent day's schedule hard-infeasible. Clamp the hard band around the current state and leave comfort steering to the soft profiles. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: F.N. Claessen <claessen@seita.nl>
Under multi-house load a scheduling job can queue behind a full wave of solver-time-limited (120s) jobs, so 200s of polling gave up on schedules that were one wave away. 360s covers two waves plus overhead; step count raised to keep the per-sleep cap (10s) from binding first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: F.N. Claessen <claessen@seita.nl>
Opaque orchestration metadata passed alongside the domain payload (a sibling of flex-model/flex-context, mirroring the S2 wrapper convention). Passed through verbatim to servers that accept it. Signed-off-by: F.N. Claessen <claessen@seita.nl>
Per S2, an RM only sends FRBC.* after accepting SelectControlType, so such a message proves activation. A dropped ReceptionStatus previously wedged the CEM in NO_SELECTION, bouncing every FRBC.SystemDescription with TEMPORARY_ERROR while the RM retried forever (in-vivo hang, 2026-07-26). With regression test. Signed-off-by: F.N. Claessen <claessen@seita.nl>
…tion batches, sent-instruction registry Latest-request-wins generation counters kill backlogged handler tasks (observed storm: 96 -> 2,225 instructions/burst, ack starvation, RM retry loops, 800 s control misses); content-hash dedupe of re-sent system descriptions; instruction batches carry only state CHANGES (the RM holds state, fm-s2 applies the same filter); sent-instruction registry with InstructionStatusUpdate tracking and opt-in RevokeObject on batch replacement (send_revocations, off by default for RMs without revoke handling). Six new tests. Signed-off-by: F.N. Claessen <claessen@seita.nl>
Large thermal fill levels sit far from zero; the default zero-including axis flattened all variation into a sliver. Signed-off-by: F.N. Claessen <claessen@seita.nl>
The trigger retry loop treats 'Scheduling job failed' as transient, but a deterministic server-side failure (e.g. a UniqueViolation on saving the schedule) re-fails identically on every attempt; re-triggering it five times with fresh jobs only multiplies failed jobs across the worker pool. Skip retries when the failure message carries a UniqueViolation/duplicate-key signature. Signed-off-by: F.N. Claessen <claessen@seita.nl>
The RM forwards realized power as one S2 PowerMeasurement per 15-min interval, serially; awaiting each FM post inline serialized ~200 HTTP round-trips per house per simulated day on the co-simulation critical path. Posts now run as tracked asyncio tasks with identical payloads (same event_start, prior, unit and values), and CEM.flush_measurement_posts() provides the read barrier: FRBC trigger_schedule awaits it, so FlexMeasures schedulers (and the community compliance check they feed) still see the complete realized series, exactly as the old serial awaits guaranteed. Post failures are swallowed with a debug log as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
The RM re-sends the same realized-power series with every retrigger round's flexinput, so the CEM re-posted ~200 identical values per house per round. Track content keys (sensor, event start, value) and skip posts already made or in flight; a failed post releases its key so a later re-send retries it. Changed values (which realized data should never produce) still post. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
The server silently substitutes the fallback scheduler's coarse charging policy when the real problem is infeasible, and the fallback saves no state-of-charge stream - which broke follow-the-schedule batteries a day later (soc-at-start resolution found nothing) and starved houses in co-simulation. Inspect scheduler_info on the returned schedule and log an ERROR naming the window, so a fallback result is never mistaken for a healthy schedule. The fallback scheduler is slated for removal in FlexMeasures v1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
The CEM object is rebuilt for every RM websocket (re)connection - in co-simulation that is every replan round - so the instance-level dedupe keys and pending-post set reset each round: the dedupe never held (a profiled 1-day run re-posted each house's realized series ~3x, ~13k sensor-data calls), and a fresh instance's flush barrier could not await the previous instance's still-in-flight posts. Hoist both sets to process scope; one CEM server process serves one apartment, so that is their correct lifetime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
configure_site runs on every RM (re)connection - in co-simulation that is every replan round - and its full-catalog get_assets scan cost ~5-7s per call against a large database, about half the API time of a profiled co-simulation day. Find the site asset via an id/name listing and fetch only the match, falling back to the full scan on servers without the fields parameter (< 0.31). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
Even the lean id/name listing costs seconds against a large database, and configure_site runs on every RM (re)connection - every replan round in co-simulation. Remember the resolved site-asset id at process scope (one CEM server process per apartment) and fetch it directly on reconnections, invalidating on a miss. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
The RM's declared fill-level range used to be sent to FlexMeasures as hard soc-min/soc-max bounds. A hard bound at the declared range made physically-reachable states infeasible: free-floating above "full", or a time-capped incumbent left below "empty" for the next day, which is a deterministically infeasible scheduling job (the sweep-9 run-killer class). The comfort steering machinery is already soft: the fill-level target profile is posted onto soc-minima/-maxima sensors, which the scheduler enforces as breach-priced StockCommitments under relax-soc-constraints. So make the declared range the behavioral band on the soft path, and demote the hard bounds to wide safety rails: - trigger_schedule: soc-min = 0 and soc-max = 1.5x the declared range top (still widened to include the current state as a last resort). The rails only exist to keep the LP bounded (e.g. negative-price hours combined with a soc-value-at-end incentive). - send_fill_level_target_profile: clip the posted soc-minima/-maxima into the declared range (night setbacks clip up to the range bottom; ceilings clip down to the range top). Where a target band lies entirely outside the declared range, both bounds collapse to the midpoint of the clipped pair (keeping minima <= maxima pointwise) and a warning is logged. The clip lives in clip_fill_level_target_profile (FRBC/utils.py), unit-tested in tests/s2/test_frbc_soft_range.py along with the flex-model rails. FM-side needs no change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVTVfL4fQ9QSqbLXmAGD Signed-off-by: F.N. Claessen <claessen@seita.nl>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #187.
FRBC.FillLevelTargetProfileandFRBC.UsageForecastto theFRBCSimplehandler.SimpleFRBChandler to get to the point thatFRBCInstructionsare sent back.FRBC.LeakageBehaviourto theFRBCSimplehandler.