From 2694a6c732a109a11c0d5e957f992f9ef739596f Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 5 Aug 2026 10:33:20 +0200 Subject: [PATCH 1/8] Update the device scheduler's (MI)LP formulation in the docs The formulation had drifted from the model the scheduler actually builds. It now covers sub-commitments (deviation variables indexed by commitment group rather than by time step), stock groups and the recursive stock balance, per-commodity EMS constraint groups, device-scoped and stock commitments, hard flow coupling, balance groups for internal commodity nodes, and power bands (S2 operation modes), plus the binary variables that make this a mixed-integer program. Also drops the perfect/left/right efficiency variants, which the scheduler no longer models (it always uses the linear treatment; the other variants live on in apply_stock_changes_and_losses). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UKx19DmqHiTBS7iEeFTijs Signed-off-by: F.N. Claessen --- documentation/concepts/device_scheduler.rst | 321 ++++++++++++++------ 1 file changed, 223 insertions(+), 98 deletions(-) diff --git a/documentation/concepts/device_scheduler.rst b/documentation/concepts/device_scheduler.rst index 289f40cfc8..66eff20d8f 100644 --- a/documentation/concepts/device_scheduler.rst +++ b/documentation/concepts/device_scheduler.rst @@ -1,18 +1,25 @@ .. _storage_device_scheduler: -Storage device scheduler: Linear model -======================================= +Device scheduler: mixed-integer linear model +============================================= Introduction -------------- -This generic storage device scheduler is able to handle a site with multiple devices, with various types of constraints on the site level and on the device level, +This generic device scheduler is able to handle a site with multiple devices, with various types of constraints on the site level and on the device level, and with multiple market commitments on the site level. A typical example is a house with many devices. The commitments are assumed to be with regard to the flow of energy to the device (positive for consumption, negative for production). In practice, this generic scheduler is used in the **StorageScheduler** to schedule a storage device. - + The solver minimizes the costs of deviating from the commitments. For a more detailed explanation of commitments in FlexMeasures, see :ref:`commitments`. +The model is a *mixed-integer* linear program: binary variables model the sign of a device's power, the direction of a commitment deviation (only when the cost curve is non-convex), and the choice of operation mode for devices with power bands. +Without any of these, the model reduces to a plain linear program. + +.. note:: + The model below is built by ``flexmeasures.data.models.planning.linear_optimization.device_scheduler`` using Pyomo. + A second backend (``highspy_optimization``) builds the same model directly with the HiGHS Python API, which is much faster to construct. + Both are fed by the same input preparation step (``scheduling_problem.prepare_scheduling_problem``), and the symbols below refer to its output. Notation @@ -20,119 +27,133 @@ Notation Indexes ^^^^^^^^ -================================ ================================================ ============================================================================================================== -Symbol Variable in the Code Description -================================ ================================================ ============================================================================================================== -:math:`c` c Commitments, for example, day-ahead or intra-day market commitments. -:math:`d` d Devices, for example, a battery or a load. -:math:`j` j 0-indexed time dimension. -================================ ================================================ ============================================================================================================== +========= ==================== =============================================================================================================== +Symbol Variable in the Code Description +========= ==================== =============================================================================================================== +:math:`c` c Sub-commitments: one per commitment group and per deviation direction (see `Commitments and sub-commitments`_). +:math:`d` d Devices, for example, a battery or a load. +:math:`j` j 0-indexed time dimension. +:math:`g` cg, cjg Device groups within a device-scoped commitment :math:`c`. +:math:`s` sg Stock groups: sets of devices that share one stock (e.g. one state-of-charge sensor). +:math:`e` eg EMS constraint groups: sets of devices sharing one site-level capacity constraint (one per commodity). +:math:`b` db Power bands (S2 operation modes) of a banded device :math:`d`. +:math:`k` coupling_group_range Coupling groups: sets of devices whose flows are hard-coupled in fixed proportions. +:math:`n` balance_group_range Balance groups: internal commodity nodes (e.g. a heat or steam network) whose flows must net to zero. +========= ==================== =============================================================================================================== .. note:: - The time index :math:`j` has two interpretations: a time period or an instantaneous moment at the end of time period :math:`j`. + The time index :math:`j` has two interpretations: a time period or an instantaneous moment at the end of time period :math:`j`. For example, :math:`j` in flow constraints correspond to time periods, whereas :math:`j` used in a stock constraint refers to the end of time period :math:`j`. Parameters ^^^^^^^^^^ -================================ ================================================ ============================================================================================================== -Symbol Variable in the Code Description -================================ ================================================ ============================================================================================================== -:math:`Price_{up}(c,j)` up_price Price of incurring an upwards deviations in commitment :math:`c` during time period :math:`j`. -:math:`Price_{down}(c,j)` down_price Price of incurring a downwards deviations in commitment :math:`c` during time period :math:`j`. -:math:`\eta_{up}(d,j)` device_derivative_up_efficiency Upwards conversion efficiency. -:math:`\eta_{down}(d,j)` device_derivative_down_efficiency Downwards conversion efficiency. -:math:`Stock_{min}(d,j)` device_min Minimum quantity for the stock of device :math:`d` at the end of time period :math:`j`. -:math:`Stock_{max}(d,j)` device_max Maximum quantity for the stock of device :math:`d` at the end of time period :math:`j`. -:math:`\epsilon(d,j)` efficiencies Stock energy losses. -:math:`P_{max}(d,j)` device_derivative_max Maximum flow of device :math:`d` during time period :math:`j`. -:math:`P_{min}(d,j)` device_derivative_min Minimum flow of device :math:`d` during time period :math:`j`. -:math:`P^{ems}_{min}(j)` ems_derivative_min Minimum flow of the site's grid connection point during time period :math:`j`. -:math:`P^{ems}_{max}(j)` ems_derivative_max Maximum flow of the site's grid connection point during time period :math:`j`. -:math:`Commitment(c,j)` commitment_quantity Commitment c (at site level) over time step :math:`j`. -:math:`M` M Large constant number, upper bound of :math:`Power_{up}(d,j)` and :math:`|Power_{down}(d,j)|`. -:math:`D(d,j)` stock_delta Explicit energy gain or loss of device :math:`d` during time period :math:`j`. -================================ ================================================ ============================================================================================================== +========================================== ================================= ======================================================================================================================================== +Symbol Variable in the Code Description +========================================== ================================= ======================================================================================================================================== +:math:`Price_{up}(c)` up_price Price of incurring an upwards deviation in sub-commitment :math:`c`. +:math:`Price_{down}(c)` down_price Price of incurring a downwards deviation in sub-commitment :math:`c`. +:math:`Commitment(c,j)` commitment_quantity Committed quantity of sub-commitment :math:`c` for time period :math:`j` (a flow, or a stock for stock commitments). +:math:`\eta_{up}(d,j)` device_derivative_up_efficiency Upwards conversion efficiency (stock increase : flow in). +:math:`\eta_{down}(d,j)` device_derivative_down_efficiency Downwards conversion efficiency (flow out : stock decrease). +:math:`Stock_{min}(d,j)` device_min Minimum quantity for the stock of device :math:`d` at the end of time period :math:`j`. +:math:`Stock_{max}(d,j)` device_max Maximum quantity for the stock of device :math:`d` at the end of time period :math:`j`. +:math:`Stock_0(d)` initial_stock Initial stock of device :math:`d`, shared by all devices in its stock group. +:math:`\epsilon(d,j)` device_efficiency Storage efficiency (stock losses), shared by all devices in a stock group. +:math:`P_{max}(d,j)` device_derivative_max Maximum flow of device :math:`d` during time period :math:`j`. +:math:`P_{min}(d,j)` device_derivative_min Minimum flow of device :math:`d` during time period :math:`j`. +:math:`P^{ems}_{min}(e,j)` ems_derivative_min Minimum aggregated flow of EMS constraint group :math:`e` during time period :math:`j`. +:math:`P^{ems}_{max}(e,j)` ems_derivative_max Maximum aggregated flow of EMS constraint group :math:`e` during time period :math:`j`. +:math:`D(d,j)` stock_delta Explicit stock gain or loss of device :math:`d` during time period :math:`j`. +:math:`\gamma(k,d)` coupling coefficient Fixed proportion of device :math:`d` within coupling group :math:`k`. Positive for inputs (consuming), negative for outputs (producing). +:math:`B_{min}(d,b)`, :math:`B_{max}(d,b)` band_lookup Lower and upper flow bound of power band :math:`b` of device :math:`d`. +:math:`M_d` Md Big-M bounding device power: the largest absolute device flow limit (at least 1 MW). +:math:`M_c` Mc Big-M bounding commitment deviations: the summed absolute device flow limits (at least 1 MW). +========================================== ================================= ======================================================================================================================================== Variables ^^^^^^^^^ -================================ ================================================ ============================================================================================================== -Symbol Variable in the Code Description -================================ ================================================ ============================================================================================================== -:math:`\Delta_{up}(c,j)` commitment_upwards_deviation Upwards deviation from the power commitment :math:`c` of the site during time period :math:`j`. -:math:`\Delta_{down}(c,j)` commitment_downwards_deviation Downwards deviation from the power commitment :math:`c` of the site during time period :math:`j`. -:math:`\Delta Stock(d,j)` n/a Change of stock of device :math:`d` at the end of time period :math:`j`. -:math:`P_{up}(d,j)` device_power_up Upwards power of device :math:`d` during time period :math:`j`. -:math:`P_{down}(d,j)` device_power_down Downwards power of device :math:`d` during time period :math:`j`. -:math:`P^{ems}(j)` ems_power Aggregated power of all the devices during time period :math:`j`. -:math:`\sigma(d,j)` device_power_sign Upwards power activation if :math:`\sigma(d,j)=1`, downwards power activation otherwise. -================================ ================================================ ============================================================================================================== +======================== ============================== ===================================================================================================================== +Symbol Variable in the Code Description +======================== ============================== ===================================================================================================================== +:math:`\Delta_{up}(c)` commitment_upwards_deviation Upwards deviation from sub-commitment :math:`c` (:math:`\geq 0`). One variable per sub-commitment, not per time step. +:math:`\Delta_{down}(c)` commitment_downwards_deviation Downwards deviation from sub-commitment :math:`c` (:math:`\leq 0`). +:math:`\sigma_c(c)` commitment_sign Binary. Upwards deviation allowed if :math:`\sigma_c(c)=1`, downwards deviation otherwise. +:math:`P_{up}(d,j)` device_power_up Upwards (consuming) power of device :math:`d` during time period :math:`j` (:math:`\geq 0`). +:math:`P_{down}(d,j)` device_power_down Downwards (producing) power of device :math:`d` during time period :math:`j` (:math:`\leq 0`). +:math:`P^{ems}(d,j)` ems_power Net flow of device :math:`d` during time period :math:`j`. +:math:`\sigma(d,j)` device_power_sign Binary. Upwards power activation if :math:`\sigma(d,j)=1`, downwards power activation otherwise. +:math:`Stock(s,j)` group_stock Stock of stock group :math:`s` at the end of time period :math:`j`. +:math:`\alpha(k,j)` coupling_alpha Common normalised flow level of coupling group :math:`k` during time period :math:`j`. +:math:`y(d,b,j)` device_band Binary. Device :math:`d` operates in power band :math:`b` during time period :math:`j`. +======================== ============================== ===================================================================================================================== + +Commitments and sub-commitments +-------------------------------- + +A commitment may declare a ``group`` per time step, which defines the set of time steps within which deviations are accounted for together +(for example: only the highest breach per calendar month is penalised). +Before the model is built, every commitment is split into *sub-commitments*: one per group, and — where a group spans multiple time steps — one per deviation direction. +The index :math:`c` therefore runs over sub-commitments, and each carries a single pair of deviation prices. + +This is why the deviation variables :math:`\Delta_{up}(c)` and :math:`\Delta_{down}(c)` are *not* indexed by time: +one deviation is paid for per sub-commitment, however many time steps it spans. +For the common case of a per-time-step commitment (such as an energy tariff), each time step forms its own group, and the two coincide. + +After solving, the costs of the sub-commitments are aggregated back to the original commitments (``model.commitment_costs``) and per commodity (``model.commodity_costs``). Cost function -------------- -The cost function quantifies the total cost of upwards and downwards deviations from the different commitments. +The cost function quantifies the total cost of upwards and downwards deviations from the different sub-commitments. -.. math:: +.. math:: :name: cost_function - \min [\sum_{c,j} \Delta_{up}(c,j) \cdot Price_{up}(c,j) + \Delta_{down}(c,j) \cdot Price_{down}(c,j)] + \min \sum_{c} [\Delta_{up}(c) \cdot Price_{up}(c) + \Delta_{down}(c) \cdot Price_{down}(c)] + +Note that :math:`\Delta_{down}(c) \leq 0`, so a positive downwards deviation price yields a negative contribution, and vice versa. +This mirrors the sign convention described under :ref:`commitments`. State dynamics --------------- -To simplify the description of the model, the auxiliary variable :math:`\Delta Stock(d,j)` is introduced in the documentation. It represents the -change of :math:`Stock(d,j)`, taking into account conversion efficiencies but not considering the storage losses. +Stock is tracked per *stock group* :math:`s`: a set of devices that share one stock, such as several converters filling one buffer. +Devices that are not part of a declared stock group form a stock group of their own. +Because the stock is a property of the group, all its devices must declare the same storage efficiency and the same initial stock. + +To simplify the description of the model, the auxiliary variable :math:`\Delta Stock(s,j)` is introduced in the documentation. It represents the +change of :math:`Stock(s,j)`, taking into account conversion efficiencies but not considering the storage losses. .. math:: :name: stock - \Delta Stock(d,j) = \frac{P_{down}(d,j)}{\eta_{down}(d,j) } + P_{up}(d,j) \cdot \eta_{up}(d,j) + D(d,j) - - -.. math:: - :name: device_bounds - - Stock_{min}(d,j) \leq Stock(d,j) - Stock(d,-1)\leq Stock_{max}(d,j) + \Delta Stock(s,j) = \sum_{d \in s} \left[ \frac{P_{down}(d,j)}{\eta_{down}(d,j)} + P_{up}(d,j) \cdot \eta_{up}(d,j) + D(d,j) \right] +The stock is then defined recursively, rather than as a running sum over all preceding time steps, which keeps the number of nonzeros in the model linear (rather than quadratic) in the scheduling horizon: -Perfect efficiency -^^^^^^^^^^^^^^^^^^^ - -.. math:: - :name: efficiency_e1 - - Stock(d, j) = Stock(d, j-1) + \Delta Stock(d,j) - -Left efficiency -^^^^^^^^^^^^^^^^^ -First apply the stock change, then apply the losses (i.e. the stock changes on the left side of the time interval in which the losses apply) - - -.. math:: - :name: efficiency_left - - Stock(d, j) = (Stock(d, j-1) + \Delta Stock(d,j)) \cdot \epsilon(d,j) +.. math:: + :name: group_stock_balance + Stock(s, j) = a(s,j) \cdot Stock(s, j-1) + b(s,j) \cdot \Delta Stock(s,j) -Right efficiency -^^^^^^^^^^^^^^^^^ -First apply the losses, then apply the stock change (i.e. the stock changes on the right side of the time interval in which the losses apply) +with :math:`Stock(s, -1) = Stock_0(s)`, and with the loss coefficients -.. math:: - :name: efficiency_right - - Stock(d, j) = Stock(d, j-1) \cdot \epsilon(d,j) + \Delta Stock(d,j) +.. math:: + :name: loss_coefficients -Linear efficiency -^^^^^^^^^^^^^^^^^ -Assume the change happens at a constant rate, leading to a linear stock change, and exponential decay, within the current interval + (a(s,j), b(s,j)) = \begin{cases} + (1, 1) & \text{if } \epsilon(d,j) = 1 \\ + \left(\epsilon(d,j), \frac{\epsilon(d,j) - 1}{\log(\epsilon(d,j))}\right) & \text{otherwise} + \end{cases} -.. math:: - :name: efficiency_linear +for any device :math:`d \in s` (they all share the group's storage efficiency). - Stock(d, j) = Stock(d, j-1) \cdot \epsilon(d,j) + \Delta Stock(d,j) \cdot \frac{\epsilon(d,j) - 1}{log(\epsilon(d,j))} +.. note:: + This is the *linear* treatment of storage losses: the stock is assumed to change at a constant rate, while losses decay exponentially, within each time step. + The scheduler models losses this way exclusively. The alternative treatments (perfect, left and right) still exist in + ``flexmeasures.utils.calculations.apply_stock_changes_and_losses``, which reconstructs a stock series from a known power series. Constraints -------------- @@ -140,18 +161,25 @@ Constraints Device bounds ^^^^^^^^^^^^^ -.. math:: +Stock bounds are expressed relative to the device's initial stock, and refer to the stock of the group the device belongs to: + +.. math:: + :name: device_bounds + + Stock_{min}(d,j) \leq Stock(s(d), j) - Stock_0(d) \leq Stock_{max}(d,j) + +.. math:: :name: device_derivative_bounds P_{min}(d,j) \leq P_{up}(d,j) + P_{down}(d,j)\leq P_{max}(d,j) -.. math:: +.. math:: :name: device_down_derivative_bounds min(P_{min}(d,j),0) \leq P_{down}(d,j)\leq 0 -.. math:: +.. math:: :name: device_up_derivative_bounds 0 \leq P_{up}(d,j)\leq max(P_{max}(d,j),0) @@ -162,35 +190,132 @@ Upwards/Downwards activation selection Avoid simultaneous upwards and downwards activation during the same time period. -.. math:: +.. math:: :name: device_up_derivative_sign - P_{up}(d,j) \leq M \cdot \sigma(d,j) + P_{up}(d,j) \leq M_d \cdot \sigma(d,j) -.. math:: +.. math:: :name: device_down_derivative_sign - -P_{down}(d,j) \leq M \cdot (1-\sigma(d,j)) + -P_{down}(d,j) \leq M_d \cdot (1-\sigma(d,j)) + +The same trick prevents a sub-commitment from deviating in both directions at once: + +.. math:: + :name: commitment_up_derivative_sign + + \Delta_{up}(c) \leq M_c \cdot \sigma_c(c) + +.. math:: + :name: commitment_down_derivative_sign + + -\Delta_{down}(c) \leq M_c \cdot (1-\sigma_c(c)) +These two constraints (and their binary variables) are only added when the summed deviation prices do not describe a convex cost curve. +For a convex cost curve, deviating in both directions is never optimal anyway, and leaving the binaries out keeps the problem a pure LP. Grid constraints ^^^^^^^^^^^^^^^^^ -.. math:: +.. math:: :name: device_derivative_equalities P^{ems}(d,j) = P_{up}(d,j) + P_{down}(d,j) -.. math:: +Site-level capacity is enforced per EMS constraint group :math:`e`, over the devices :math:`E(e)` it covers. +The StorageScheduler uses one group per commodity, so each commodity gets its own site-level capacity constraint. +A single group covering all devices is the default (and the historical behaviour). + +.. math:: :name: ems_derivative_bounds - P^{ems}_{min}(j) \leq \sum_d P^{ems}(d,j) \leq P^{ems}_{max}(j) + P^{ems}_{min}(e,j) \leq \sum_{d \in E(e)} P^{ems}(d,j) \leq P^{ems}_{max}(e,j) + +EMS-level commitments +^^^^^^^^^^^^^^^^^^^^^^ + +Commitments that do not name a device apply to the site as a whole. +Writing :math:`\mathcal{D}(c)` for the devices such a commitment covers — all devices, or, if the commitment names a commodity, the devices of that commodity — and + +.. math:: + :name: ems_flow_commitment_deviation + + \Xi(c,j) = Commitment(c,j) + \Delta_{down}(c) + \Delta_{up}(c) - \sum_{d \in \mathcal{D}(c)} P^{ems}(d,j) + +the constraint is bounded on the side(s) for which the sub-commitment carries a deviation price: + +================================ ========================= ===================================================================== +Sub-commitment prices Constraint Meaning +================================ ========================= ===================================================================== +Both prices given :math:`0 \leq \Xi \leq 0` The commitment is met exactly, or paid for in both directions. +Only an upwards deviation price :math:`\Xi \geq 0` Flow above the committed quantity is a breach; staying below is free. +Only a downwards deviation price :math:`\Xi \leq 0` Flow below the committed quantity is a breach; staying above is free. +================================ ========================= ===================================================================== + +Device-scoped commitments +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A commitment may instead name devices, optionally organised in device groups :math:`g` (the ``device_group`` column; by default each device forms its own group). +It is then bound once per device group, against the aggregate of that group's devices :math:`G(c,g)`. +For a flow commitment, the aggregate is a flow: + +.. math:: + :name: grouped_flow_commitment_deviation + + \Xi(c,j,g) = Commitment(c,j) + \Delta_{down}(c) + \Delta_{up}(c) - \sum_{d \in G(c,g)} P^{ems}(d,j) + +and for a stock commitment, it is a stock change since the start of the schedule: + +.. math:: + :name: grouped_stock_commitment_deviation -Power coupling constraints -^^^^^^^^^^^^^^^^^^^^^^^^^^^ + \Xi(c,j,g) = Commitment(c,j) + \Delta_{down}(c) + \Delta_{up}(c) - \sum_{d \in G(c,g)} [Stock(s(d), j) - Stock_0(d)] -.. math:: - :name: ems_flow_commitment_equalities +Both are bounded exactly as in the table above. +A stock commitment that names a stock group couples to that group as a whole, through a single representative device, so a shared stock is not counted more than once. + +Hard flow coupling +^^^^^^^^^^^^^^^^^^^ + +Devices in a coupling group :math:`k` are forced to operate in fixed proportion to one another — +for example a CHP unit whose gas input, heat output and power output move together. +A free variable :math:`\alpha(k,j)` represents the group's common normalised flow level: + +.. math:: + :name: flow_coupling + + P^{ems}(d,j) = \gamma(k,d) \cdot \alpha(k,j) \qquad \forall d \in k + +Balance groups (internal commodity nodes) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +An internal commodity node — a heat or steam network without its own grid connection — stores nothing itself: +everything produced into the node must be consumed from it within the same time step. + +.. math:: + :name: node_balance + + \sum_{d \in N(n)} P^{ems}(d,j) = 0 + +Derivative efficiencies and stock deltas describe each device's own stock-side conversion, and do not enter this commodity-side balance. +To give a node storage, include a storage device in the group: its flow absorbs the imbalance, while its stock is bounded by its own device constraints. + +Power bands (S2 operation modes) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A device may declare a list of signed power bands, and must then operate within exactly one of them at every time step. +Which band it runs in is a free binary decision: + +.. math:: + :name: device_band_choice + + \sum_{b} y(d,b,j) = 1 + +.. math:: + :name: device_band_power_bounds - \sum_d P^{ems}(d,j) = \sum_c Commitment(c,j) + \Delta_{up}(c,j) + \Delta_{down}(c,j) + \sum_{b} y(d,b,j) \cdot B_{min}(d,b) \leq P_{up}(d,j) + P_{down}(d,j) \leq \sum_{b} y(d,b,j) \cdot B_{max}(d,b) +Because exactly one band binary is 1, each sum selects the bound of the chosen band. +Bands are what make a device's feasible operating region non-convex (for instance, "off, or between 40% and 100%"), at the cost of one binary variable per device per band per time step. From 31fb06e77a46dc3414e8aa42d209c271d1e03b21 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 5 Aug 2026 11:06:17 +0200 Subject: [PATCH 2/8] Add changelog entry for the updated scheduler formulation docs Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UKx19DmqHiTBS7iEeFTijs Signed-off-by: F.N. Claessen --- documentation/changelog.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/documentation/changelog.rst b/documentation/changelog.rst index a734982fe4..fa99c2d795 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -56,6 +56,7 @@ Infrastructure / Support * Speed up scheduling jobs by building the scheduling problem directly with the HiGHS Python API (``highspy``), bypassing Pyomo's model construction and solution-ingestion overhead (roughly a second for a single-device job, and several seconds for multi-device jobs); this direct backend is the new default for the ``FLEXMEASURES_LP_SOLVER`` setting (``"highspy"``), while any Pyomo-based solver (e.g. the previous default ``"appsi_highs"``, or ``"cbc"``) remains available as before [see `PR #2364 `_] * Price fields in the flex-context (including nested commitment prices, which are now also held to the flex-context's shared currency) are selected for currency validation by field type (``PriceField``) instead of by name suffix [see `PR #2311 `_] * Document ``SECURITY_TWO_FACTOR`` and related 2FA configuration settings [see `PR #2340 `_] +* Bring the documented ``device_scheduler`` formulation up to date with the model the scheduler builds, covering sub-commitments, stock groups, per-commodity EMS constraint groups, device-scoped and stock commitments, flow coupling, balance groups and power bands, plus the binary variables that make it a mixed-integer program [see `PR #2394 `_] * Warn on startup when ``TRUSTED_HOSTS`` is unset, as that lets clients poison the URLs FlexMeasures generates, such as password reset links; the setting can now also be given as a comma-separated environment variable, and the ``development`` environment trusts loopback hosts by default (so reaching a development server by its LAN address or through a tunnel now means listing that host) [see `PR #2389 `_] * ``flexmeasures db upgrade`` now runs ``VACUUM ANALYZE`` after upgrading by default, so Postgres has fresh planner statistics right after a migration; opt out with ``--no-vacuum`` [see `PR #2333 `_] * Upgraded dependencies [see `PR #1485 `_, `PR #2215 `_, `PR #2243 `_, `PR #2348 `_ and `PR #2388 `_] From cf463eef90116a287f87ea475d47423fef361202 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 5 Aug 2026 12:01:18 +0200 Subject: [PATCH 3/8] Address review: define the device sets, fix Stock_0's index, qualify module paths Adds a "Sets and mappings" table introducing s(d), S(s), E(e), D(c), G(c,g) and N(n), which the constraints sum over but which were only inferable from context. The stock recursion's initial condition now uses Stock_0(d), as the parameters table defines it, rather than an undefined per-group Stock_0(s). Also spells out the HiGHS backend and problem-preparation entrypoints in full. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UKx19DmqHiTBS7iEeFTijs Signed-off-by: F.N. Claessen --- documentation/concepts/device_scheduler.rst | 24 ++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/documentation/concepts/device_scheduler.rst b/documentation/concepts/device_scheduler.rst index 66eff20d8f..5205c43f3f 100644 --- a/documentation/concepts/device_scheduler.rst +++ b/documentation/concepts/device_scheduler.rst @@ -18,8 +18,10 @@ Without any of these, the model reduces to a plain linear program. .. note:: The model below is built by ``flexmeasures.data.models.planning.linear_optimization.device_scheduler`` using Pyomo. - A second backend (``highspy_optimization``) builds the same model directly with the HiGHS Python API, which is much faster to construct. - Both are fed by the same input preparation step (``scheduling_problem.prepare_scheduling_problem``), and the symbols below refer to its output. + A second backend, ``flexmeasures.data.models.planning.highspy_optimization.device_scheduler_highspy``, + builds the same model directly with the HiGHS Python API, which is much faster to construct. + Both are fed by the same input preparation step, ``flexmeasures.data.models.planning.scheduling_problem.prepare_scheduling_problem``, + and the symbols below refer to its output. Notation @@ -45,6 +47,22 @@ Symbol Variable in the Code Description The time index :math:`j` has two interpretations: a time period or an instantaneous moment at the end of time period :math:`j`. For example, :math:`j` in flow constraints correspond to time periods, whereas :math:`j` used in a stock constraint refers to the end of time period :math:`j`. +Sets and mappings +^^^^^^^^^^^^^^^^^^ + +The constraints below sum over the devices that a group covers. These sets name those memberships. + +====================== ============================ ============================================================================================================== +Symbol Variable in the Code Description +====================== ============================ ============================================================================================================== +:math:`s(d)` device_to_group The stock group that device :math:`d` belongs to (its primary one, if it participates in several). +:math:`S(s)` group_to_devices The devices that share stock group :math:`s`. Written as :math:`d \in s` below. +:math:`E(e)` ems_constraint_device_groups The devices covered by EMS constraint group :math:`e`. +:math:`\mathcal{D}(c)` commodity_devices The devices covered by an EMS-level commitment :math:`c`: all devices, or those of the commitment's commodity. +:math:`G(c,g)` device_group_lookup The devices in device group :math:`g` of a device-scoped commitment :math:`c`. +:math:`N(n)` balance_group_specs The devices attached to internal commodity node :math:`n`. +====================== ============================ ============================================================================================================== + Parameters ^^^^^^^^^^ ========================================== ================================= ======================================================================================================================================== @@ -138,7 +156,7 @@ The stock is then defined recursively, rather than as a running sum over all pre Stock(s, j) = a(s,j) \cdot Stock(s, j-1) + b(s,j) \cdot \Delta Stock(s,j) -with :math:`Stock(s, -1) = Stock_0(s)`, and with the loss coefficients +with :math:`Stock(s, -1) = Stock_0(d)` for any device :math:`d \in s` (the group's devices share one initial stock), and with the loss coefficients .. math:: :name: loss_coefficients From ecb7ff362b0a4c7b95af4ae1c1a88d0e49542297 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 5 Aug 2026 12:08:07 +0200 Subject: [PATCH 4/8] Address review: stock bounds are relative, and Pyomo always declares the sign binary The parameters table described device_min/device_max as absolute stock bounds, while both the constraint below it and the implementation bound the stock change since the start of the schedule. Also corrects the claim that the commitment-sign binaries are only added for a non-convex cost curve: only the constraints are gated in the Pyomo backend, which declares the variable either way. The direct HiGHS backend does skip the variable. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UKx19DmqHiTBS7iEeFTijs Signed-off-by: F.N. Claessen --- documentation/concepts/device_scheduler.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/documentation/concepts/device_scheduler.rst b/documentation/concepts/device_scheduler.rst index 5205c43f3f..bb6ea63b36 100644 --- a/documentation/concepts/device_scheduler.rst +++ b/documentation/concepts/device_scheduler.rst @@ -73,8 +73,8 @@ Symbol Variable in the Code D :math:`Commitment(c,j)` commitment_quantity Committed quantity of sub-commitment :math:`c` for time period :math:`j` (a flow, or a stock for stock commitments). :math:`\eta_{up}(d,j)` device_derivative_up_efficiency Upwards conversion efficiency (stock increase : flow in). :math:`\eta_{down}(d,j)` device_derivative_down_efficiency Downwards conversion efficiency (flow out : stock decrease). -:math:`Stock_{min}(d,j)` device_min Minimum quantity for the stock of device :math:`d` at the end of time period :math:`j`. -:math:`Stock_{max}(d,j)` device_max Maximum quantity for the stock of device :math:`d` at the end of time period :math:`j`. +:math:`Stock_{min}(d,j)` device_min Minimum stock of device :math:`d` at the end of time period :math:`j`, relative to its initial stock. +:math:`Stock_{max}(d,j)` device_max Maximum stock of device :math:`d` at the end of time period :math:`j`, relative to its initial stock. :math:`Stock_0(d)` initial_stock Initial stock of device :math:`d`, shared by all devices in its stock group. :math:`\epsilon(d,j)` device_efficiency Storage efficiency (stock losses), shared by all devices in a stock group. :math:`P_{max}(d,j)` device_derivative_max Maximum flow of device :math:`d` during time period :math:`j`. @@ -230,8 +230,9 @@ The same trick prevents a sub-commitment from deviating in both directions at on -\Delta_{down}(c) \leq M_c \cdot (1-\sigma_c(c)) -These two constraints (and their binary variables) are only added when the summed deviation prices do not describe a convex cost curve. -For a convex cost curve, deviating in both directions is never optimal anyway, and leaving the binaries out keeps the problem a pure LP. +These two constraints are only added when the summed deviation prices do not describe a convex cost curve. +For a convex cost curve, deviating in both directions is never optimal anyway, so dropping them leaves the problem a pure LP. +The direct HiGHS backend then omits :math:`\sigma_c(c)` as well, while the Pyomo backend still declares the variable, leaving it unreferenced. Grid constraints ^^^^^^^^^^^^^^^^^ From 994c908700634de7e919292024f5700032136f26 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 5 Aug 2026 12:13:13 +0200 Subject: [PATCH 5/8] Address review: a single-time-step commitment group is not split by direction The index table claimed sub-commitments are always split per deviation direction, while the split only happens for groups spanning several time steps, as the section below already explained. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UKx19DmqHiTBS7iEeFTijs Signed-off-by: F.N. Claessen --- documentation/concepts/device_scheduler.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/documentation/concepts/device_scheduler.rst b/documentation/concepts/device_scheduler.rst index bb6ea63b36..312a55be69 100644 --- a/documentation/concepts/device_scheduler.rst +++ b/documentation/concepts/device_scheduler.rst @@ -29,10 +29,10 @@ Notation Indexes ^^^^^^^^ -========= ==================== =============================================================================================================== +========= ==================== =========================================================================================================================================================== Symbol Variable in the Code Description -========= ==================== =============================================================================================================== -:math:`c` c Sub-commitments: one per commitment group and per deviation direction (see `Commitments and sub-commitments`_). +========= ==================== =========================================================================================================================================================== +:math:`c` c Sub-commitments: one per commitment group, and one per deviation direction where a group spans several time steps (see `Commitments and sub-commitments`_). :math:`d` d Devices, for example, a battery or a load. :math:`j` j 0-indexed time dimension. :math:`g` cg, cjg Device groups within a device-scoped commitment :math:`c`. @@ -41,7 +41,7 @@ Symbol Variable in the Code Description :math:`b` db Power bands (S2 operation modes) of a banded device :math:`d`. :math:`k` coupling_group_range Coupling groups: sets of devices whose flows are hard-coupled in fixed proportions. :math:`n` balance_group_range Balance groups: internal commodity nodes (e.g. a heat or steam network) whose flows must net to zero. -========= ==================== =============================================================================================================== +========= ==================== =========================================================================================================================================================== .. note:: The time index :math:`j` has two interpretations: a time period or an instantaneous moment at the end of time period :math:`j`. From 27e8aadfb66ba2d8df19a933d1918515c191ba0c Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 5 Aug 2026 12:19:26 +0200 Subject: [PATCH 6/8] Address review: name coupling_device_specs, and list the auxiliary stock change Points the coupling coefficient at the prepared-problem field that actually holds it, and puts the documentation-only Delta Stock(s,j) symbol back in the variables table, so the notation section covers everything the equations below use. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UKx19DmqHiTBS7iEeFTijs Signed-off-by: F.N. Claessen --- documentation/concepts/device_scheduler.rst | 31 +++++++++++---------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/documentation/concepts/device_scheduler.rst b/documentation/concepts/device_scheduler.rst index 312a55be69..0f67b1db62 100644 --- a/documentation/concepts/device_scheduler.rst +++ b/documentation/concepts/device_scheduler.rst @@ -82,7 +82,7 @@ Symbol Variable in the Code D :math:`P^{ems}_{min}(e,j)` ems_derivative_min Minimum aggregated flow of EMS constraint group :math:`e` during time period :math:`j`. :math:`P^{ems}_{max}(e,j)` ems_derivative_max Maximum aggregated flow of EMS constraint group :math:`e` during time period :math:`j`. :math:`D(d,j)` stock_delta Explicit stock gain or loss of device :math:`d` during time period :math:`j`. -:math:`\gamma(k,d)` coupling coefficient Fixed proportion of device :math:`d` within coupling group :math:`k`. Positive for inputs (consuming), negative for outputs (producing). +:math:`\gamma(k,d)` coupling_device_specs Fixed proportion of device :math:`d` within coupling group :math:`k`. Positive for inputs (consuming), negative for outputs (producing). :math:`B_{min}(d,b)`, :math:`B_{max}(d,b)` band_lookup Lower and upper flow bound of power band :math:`b` of device :math:`d`. :math:`M_d` Md Big-M bounding device power: the largest absolute device flow limit (at least 1 MW). :math:`M_c` Mc Big-M bounding commitment deviations: the summed absolute device flow limits (at least 1 MW). @@ -91,20 +91,21 @@ Symbol Variable in the Code D Variables ^^^^^^^^^ -======================== ============================== ===================================================================================================================== -Symbol Variable in the Code Description -======================== ============================== ===================================================================================================================== -:math:`\Delta_{up}(c)` commitment_upwards_deviation Upwards deviation from sub-commitment :math:`c` (:math:`\geq 0`). One variable per sub-commitment, not per time step. -:math:`\Delta_{down}(c)` commitment_downwards_deviation Downwards deviation from sub-commitment :math:`c` (:math:`\leq 0`). -:math:`\sigma_c(c)` commitment_sign Binary. Upwards deviation allowed if :math:`\sigma_c(c)=1`, downwards deviation otherwise. -:math:`P_{up}(d,j)` device_power_up Upwards (consuming) power of device :math:`d` during time period :math:`j` (:math:`\geq 0`). -:math:`P_{down}(d,j)` device_power_down Downwards (producing) power of device :math:`d` during time period :math:`j` (:math:`\leq 0`). -:math:`P^{ems}(d,j)` ems_power Net flow of device :math:`d` during time period :math:`j`. -:math:`\sigma(d,j)` device_power_sign Binary. Upwards power activation if :math:`\sigma(d,j)=1`, downwards power activation otherwise. -:math:`Stock(s,j)` group_stock Stock of stock group :math:`s` at the end of time period :math:`j`. -:math:`\alpha(k,j)` coupling_alpha Common normalised flow level of coupling group :math:`k` during time period :math:`j`. -:math:`y(d,b,j)` device_band Binary. Device :math:`d` operates in power band :math:`b` during time period :math:`j`. -======================== ============================== ===================================================================================================================== +========================= ============================== ===================================================================================================================== +Symbol Variable in the Code Description +========================= ============================== ===================================================================================================================== +:math:`\Delta_{up}(c)` commitment_upwards_deviation Upwards deviation from sub-commitment :math:`c` (:math:`\geq 0`). One variable per sub-commitment, not per time step. +:math:`\Delta_{down}(c)` commitment_downwards_deviation Downwards deviation from sub-commitment :math:`c` (:math:`\leq 0`). +:math:`\sigma_c(c)` commitment_sign Binary. Upwards deviation allowed if :math:`\sigma_c(c)=1`, downwards deviation otherwise. +:math:`P_{up}(d,j)` device_power_up Upwards (consuming) power of device :math:`d` during time period :math:`j` (:math:`\geq 0`). +:math:`P_{down}(d,j)` device_power_down Downwards (producing) power of device :math:`d` during time period :math:`j` (:math:`\leq 0`). +:math:`P^{ems}(d,j)` ems_power Net flow of device :math:`d` during time period :math:`j`. +:math:`\sigma(d,j)` device_power_sign Binary. Upwards power activation if :math:`\sigma(d,j)=1`, downwards power activation otherwise. +:math:`Stock(s,j)` group_stock Stock of stock group :math:`s` at the end of time period :math:`j`. +:math:`\Delta Stock(s,j)` n/a Auxiliary symbol used below: the stock change of stock group :math:`s` during time period :math:`j`, before losses. +:math:`\alpha(k,j)` coupling_alpha Common normalised flow level of coupling group :math:`k` during time period :math:`j`. +:math:`y(d,b,j)` device_band Binary. Device :math:`d` operates in power band :math:`b` during time period :math:`j`. +========================= ============================== ===================================================================================================================== Commitments and sub-commitments -------------------------------- From c96af4b40ae38cf5fc8e4fb6081852df50cf9705 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 5 Aug 2026 13:05:57 +0200 Subject: [PATCH 7/8] Link the module references to the code docs, showing only the short name Replaces the fully qualified literals with :func:/:mod: roles, which resolve into the _autosummary pages (down to the function anchor) while rendering as just device_scheduler(), highspy_optimization, and so on, so the paragraphs no longer wrap around long unbreakable literals. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UKx19DmqHiTBS7iEeFTijs Signed-off-by: F.N. Claessen --- documentation/concepts/device_scheduler.rst | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/documentation/concepts/device_scheduler.rst b/documentation/concepts/device_scheduler.rst index 0f67b1db62..a5a1f69f4b 100644 --- a/documentation/concepts/device_scheduler.rst +++ b/documentation/concepts/device_scheduler.rst @@ -17,11 +17,9 @@ The model is a *mixed-integer* linear program: binary variables model the sign o Without any of these, the model reduces to a plain linear program. .. note:: - The model below is built by ``flexmeasures.data.models.planning.linear_optimization.device_scheduler`` using Pyomo. - A second backend, ``flexmeasures.data.models.planning.highspy_optimization.device_scheduler_highspy``, - builds the same model directly with the HiGHS Python API, which is much faster to construct. - Both are fed by the same input preparation step, ``flexmeasures.data.models.planning.scheduling_problem.prepare_scheduling_problem``, - and the symbols below refer to its output. + The model below is built by :func:`~flexmeasures.data.models.planning.linear_optimization.device_scheduler`, using Pyomo. + A second backend, :mod:`~flexmeasures.data.models.planning.highspy_optimization`, builds the same model directly with the HiGHS Python API, which is much faster to construct. + Both are fed by the same input preparation step, :func:`~flexmeasures.data.models.planning.scheduling_problem.prepare_scheduling_problem`, and the symbols below refer to its output. Notation @@ -172,7 +170,7 @@ for any device :math:`d \in s` (they all share the group's storage efficiency). .. note:: This is the *linear* treatment of storage losses: the stock is assumed to change at a constant rate, while losses decay exponentially, within each time step. The scheduler models losses this way exclusively. The alternative treatments (perfect, left and right) still exist in - ``flexmeasures.utils.calculations.apply_stock_changes_and_losses``, which reconstructs a stock series from a known power series. + :func:`~flexmeasures.utils.calculations.apply_stock_changes_and_losses`, which reconstructs a stock series from a known power series. Constraints -------------- From fbfa1a870c5d34ff963f5b46e7f51bfb39a5029d Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Fri, 7 Aug 2026 15:18:18 +0200 Subject: [PATCH 8/8] Address review: sign binaries are unconditional, and other accuracy fixes - The device sign binaries are part of every model with a device, so the model does not reduce to a plain LP without the conditional binary families. - The StorageScheduler also adds one EMS constraint group per device group with a power capacity, not only one per commodity. - apply_stock_changes_and_losses offers only the left and right alternatives; the perfect treatment is the lossless case of any of them. - Mc sums the absolute flow limits over devices and time steps. - Only flow commitments bind at the EMS level, and a commodity's device set is collected from device-naming commitments. Co-Authored-By: Claude Fable 5 Signed-off-by: F.N. Claessen --- documentation/concepts/device_scheduler.rst | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/documentation/concepts/device_scheduler.rst b/documentation/concepts/device_scheduler.rst index a5a1f69f4b..04c4118999 100644 --- a/documentation/concepts/device_scheduler.rst +++ b/documentation/concepts/device_scheduler.rst @@ -14,7 +14,8 @@ The solver minimizes the costs of deviating from the commitments. For a more detailed explanation of commitments in FlexMeasures, see :ref:`commitments`. The model is a *mixed-integer* linear program: binary variables model the sign of a device's power, the direction of a commitment deviation (only when the cost curve is non-convex), and the choice of operation mode for devices with power bands. -Without any of these, the model reduces to a plain linear program. +The sign binaries are part of every model with at least one device; +the other two families are only added when a non-convex cost curve or a banded device calls for them. .. note:: The model below is built by :func:`~flexmeasures.data.models.planning.linear_optimization.device_scheduler`, using Pyomo. @@ -35,7 +36,7 @@ Symbol Variable in the Code Description :math:`j` j 0-indexed time dimension. :math:`g` cg, cjg Device groups within a device-scoped commitment :math:`c`. :math:`s` sg Stock groups: sets of devices that share one stock (e.g. one state-of-charge sensor). -:math:`e` eg EMS constraint groups: sets of devices sharing one site-level capacity constraint (one per commodity). +:math:`e` eg EMS constraint groups: sets of devices sharing one site-level capacity constraint. :math:`b` db Power bands (S2 operation modes) of a banded device :math:`d`. :math:`k` coupling_group_range Coupling groups: sets of devices whose flows are hard-coupled in fixed proportions. :math:`n` balance_group_range Balance groups: internal commodity nodes (e.g. a heat or steam network) whose flows must net to zero. @@ -83,7 +84,7 @@ Symbol Variable in the Code D :math:`\gamma(k,d)` coupling_device_specs Fixed proportion of device :math:`d` within coupling group :math:`k`. Positive for inputs (consuming), negative for outputs (producing). :math:`B_{min}(d,b)`, :math:`B_{max}(d,b)` band_lookup Lower and upper flow bound of power band :math:`b` of device :math:`d`. :math:`M_d` Md Big-M bounding device power: the largest absolute device flow limit (at least 1 MW). -:math:`M_c` Mc Big-M bounding commitment deviations: the summed absolute device flow limits (at least 1 MW). +:math:`M_c` Mc Big-M bounding commitment deviations: the absolute device flow limits, summed over devices and time steps (at least 1 MW). ========================================== ================================= ======================================================================================================================================== @@ -169,8 +170,9 @@ for any device :math:`d \in s` (they all share the group's storage efficiency). .. note:: This is the *linear* treatment of storage losses: the stock is assumed to change at a constant rate, while losses decay exponentially, within each time step. - The scheduler models losses this way exclusively. The alternative treatments (perfect, left and right) still exist in + The scheduler models losses this way exclusively. The alternative treatments (left and right) still exist in :func:`~flexmeasures.utils.calculations.apply_stock_changes_and_losses`, which reconstructs a stock series from a known power series. + The perfect treatment is simply the lossless case :math:`\epsilon(d,j) = 1`, covered above. Constraints -------------- @@ -242,7 +244,8 @@ Grid constraints P^{ems}(d,j) = P_{up}(d,j) + P_{down}(d,j) Site-level capacity is enforced per EMS constraint group :math:`e`, over the devices :math:`E(e)` it covers. -The StorageScheduler uses one group per commodity, so each commodity gets its own site-level capacity constraint. +The StorageScheduler uses one group per commodity, so each commodity gets its own site-level capacity constraint, +plus one group per device group that declares its own power capacity, enforcing that group's hard bound. A single group covering all devices is the default (and the historical behaviour). .. math:: @@ -253,7 +256,8 @@ A single group covering all devices is the default (and the historical behaviour EMS-level commitments ^^^^^^^^^^^^^^^^^^^^^^ -Commitments that do not name a device apply to the site as a whole. +Flow commitments that do not name a device apply to the site as a whole +(a stock commitment only ever binds through the devices or stock group it names; see the next section). Writing :math:`\mathcal{D}(c)` for the devices such a commitment covers — all devices, or, if the commitment names a commodity, the devices of that commodity — and .. math:: @@ -271,6 +275,11 @@ Only an upwards deviation price :math:`\Xi \geq 0` Flow above the comm Only a downwards deviation price :math:`\Xi \leq 0` Flow below the committed quantity is a breach; staying above is free. ================================ ========================= ===================================================================== +A commodity's device set is not declared directly: +it is collected from the commitments that name both that commodity and devices. +A commodity that no commitment maps to devices this way has an empty :math:`\mathcal{D}(c)`, +and the constraint is skipped (leaving the commitment unenforced). + Device-scoped commitments ^^^^^^^^^^^^^^^^^^^^^^^^^^