Repository navigation
HiPO after some time restarts with IPX #3339
Replies: 8 comments 3 replies
|
Most of the behaviour of the HiGHS solvers is undocumented, because it would only be appreciable by a tiny minority of users, so the investment of time is not worthwhile. For example, the switch to IPX is done for good reasons, determined by expert understanding of interior point methods. @filikat can comment on the wisdom of this being configurable by users. |
So, the total "real" solution time after switching from HiPO to IPX after 1950s is about 20000s? |
|
HiPO switches to IPX because it decided that it cannot continue any more. This can happen based on various technical reasons that are not worth documenting, but that are related to numerical precision issues. It is not configurable because the only alternative for HiPO would be to stop there and return an error. The fact that HiPO is not able to continue can and will be improved in future releases, but for now there is not much that can be done. Notice however that numerical issues are particularly sensitive to problem scaling. |
|
Interestingly, #3424 was also an yearly power system model, but a naive look at coefficients did not point to any issue. Let me know if I can help with example instances. |
Another instance of the #3424 pattern, on a PyPSA model: one week of a yearly DC-OPF dispatch model (NW Europe, hourly), 2.70M rows, 1.66M columns, 4.9M nonzeros (363k x 1.25M after presolve). HiGHS at HiPO stops on five steps below 0.05 while the gap is still falling and the direction stays accurate (
20 and 100 take the identical path (same 157 iterations and objective), so on this model the longest run of short steps is between 10 and 20. On our Linux server, 10 of the 53 weeks of a heavy year end with HiPO "No progress" and no basis; I have instrumented only this one. I can't post the MPS publicly; I can check whether we can share it privately if that helps with the stagnation test. Last iterations of the stock run (log_dev_level=1)
Alphas are printed to two decimals, so the 0.05 entries are below the threshold. Then "Solution may be optimal, perform kktCheck / Check failed / Restarting with IPX"; IPX warns "singular matrix" while building its starting basis and stops after 5 iterations at gap 0.10. The corrector Side note for anyone reproducing: with |
|
Thanks for sharing these results @MaykThewessen. |
Happy to. I'll send the original MPS (zstd-compressed, If it helps, I can add one or two of the other weeks that end in HiPO "No progress" on our Linux server. Then you have more than one instance of the stagnation pattern to test against. see file here: hipo_3339.tar.zip |
|
The file that you shared here already contains an mps file. Is that the file that you mentioned before? |
Yes, that's the one. The tar contains:
The MPS was re-exported from the same model dump as those runs, so it is not the byte-identical file, but the dimensions match exactly. Run with |
|
Thanks! So is there anything that needs to be shared privately? |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
For some of my solver runs, HIGHS gives up after a while and restarts with IPX. There seems to be no predetermined number of iterations or time before it does that, but seems to occur approximately when the "gap" reduces by less than 5% (I'm not a technical enough in solvers to understand what "gap" is so it's just my observation). For context, this is a 1-year pyPSA dispatch model with half-hour resolution and linearized unit commitment.
When it restarts with IPX, the multi-threaded optimization is lost (IPX runs on a single thread) and the iterations are actually much slower than HiPO previously was (Ignore the large time gap between factorization 36 and 37 - my computer was hibernated).
I would much prefer this "restarting with IPX" behavior to be adjustable. For example, I would have preferred for HiPO to try many more iterations (as long as the "gap" is still getting smaller) through an adjustable solver option.
Currently this behavior doesn't seem to be documented. So I just wanted to confirm if this occurs with others, whether this is intended, and under what conditions it will change to IPX. Once I have a better understanding of this behavior, I would then like to set up a feature request to make this IPX-restart configurable.
All reactions