Version Incompatability

[Tom-Finke]

Downgrade tests with OpenFHE v0.1.14 currently fail, see #100.
This is because we force minimum version but do not restrict maximum versions and openfhe introduces breaking changes in v1.x -> v1.y minor version updates.

Implementation idea: add weak dependency¹ for openfhe-julia.jll and openfhe.jll, both with a required minimum and maximum version. Every release of openfhe-julia / openfhe would then entail a new release of OpenFHE.jl / SecureArithmetic.jl with the new maximum version number of upstream packages, either as a patch patch or minor (in the future major) version update, depending on if the upstream changes are breaking.

Footnotes

1. https://pkgdocs.julialang.org/v1/creating-packages/#Weak-dependencies ↩

[Tom-Finke]

This can (currently) not be reproduced in the downgrade tests for OpenFHE.jl, because the current OpenFHE.jl v0.2 needs at least openfhe_julia_jll v0.6, which is compatible with the currently most recent version of OpenFHE_jl.jl, v1.5.1.

If a new minor version of OpenFHE_jl.jll was released which was not backwards compatible, then I would expect downgrade tests in OpenFHE_jl.jll to fail as well, because it will use openfhe_julia_jll v0.6 with a new incompatible version of OpenFHE_jll.jl, say v1.6.

The binary builder docs specifically mention this problem by giving an example:

CFL releases version 1.1.1 and version 1.1.2 that are incompatible. A real world example is Boost (which breaks the ABI in every single release because they embed the full version number in the soname of libraries). If you have a typical permissive semver-style compat section in a package that depends on CFL_jll, then your package will break whenever CFL_jll gets a new release. To solve this issue you have to use "hyphen style" compat bounds like "0.9.0 - 1.1.2". This leads to a separate problem: you need to change the compat bound every time there is a new CFL_jll release: this is the least bad option though – it causes more annoyance for developers but it ensures users never end up with broken installs. And bots like CompatHelper can mostly automate that issue.

https://docs.binarybuilder.org/stable/FAQ/#Now-that-I-have-a-published-and-versioned-jll-package,-what-compat-bounds-do-I-put-in-its-dependents?-What-if-the-upstream-does-not-follow-SemVer?

I would therefore suggest to not use weak dependencies in OpenFHE.jl, but use the hyphen style compatability bounds¹ directly in openfhe_julia. This also makes sense in my opinion, because every release of OpenFHE_jll means we have to manually check if openfhe_julia is still compatible anyways. If it is compatible, we can simply adjust the upper compat bound and make a patch release. If it is incompatible, update the lower and upper compat bound and make a minor (or major release when arriving at v1.x). Thus, openfhe_julia is our "compatability layer", which enforces semantic versioning around OpenFHE_jll.jl. For OpenFHE.jl, we can use default CompatHelper / Dependabot to keep versions up to date, because OpenFHE.jl only depends on openfhe_julia, not OpenFHE_jll, and openfhe_julia follows semantic versioning. The same then applies to SecureArithmetic.jl, which in the future can use any compatible OpenFHE version following semantic versioning, which in turn can use any compatible openfhe_julia version following semantic versioning, which enforces a compatible OpenFHE_jll.jl version by manual hyphen compat bounds.

Footnotes

1. For an example of the hyphen syntax in action in an Yggdrasil package see https://github.com/JuliaPackaging/Yggdrasil/blob/7bbb59b999752223da267f8e1b637e8dc923488d/C/Compose/build_tarballs.jl#L69. ↩

[Tom-Finke]

@sloede what do you think?

[sloede]

Haha, I think you lost me here. Let's discuss this tomorrow!

[Tom-Finke]

Deps are set

• https://github.com/JuliaPackaging/Yggdrasil/blob/master/O/openfhe_julia/build_tarballs.jl#L14
• https://github.com/hpsc-lab/OpenFHE.jl/blob/main/Project.toml#L18
• https://github.com/hpsc-lab/SecureArithmetic.jl/blob/main/Project.toml#L10

[Tom-Finke]

Problem with setting the upper bounds in Yggdrasil: we would also change the patch number. But that would mean having to change the patch number in openfhe-julia's VERSION file as well, without an actual code change.

Yggdrasil offers way of triggering a new build without changing the version number.

Open question: have breaking changes in openfhe-development happened in patch versions? If not, we could use the tilde modifier to fix openfhe-development to 1.5.x instead of explicit hyphen syntax.

If this option works: document it in openfhe-julia and / or OpenFHE.jl

[Tom-Finke]

Openfhe claims they follow semantic versioning, stating that

Major release number must be incremented when the OpenFHE user API changes, requiring user code rewrite.¹

But in going from v1.4.1 to v1.5.0 they removed parameters from functions², which broke our wrapper. Nowhere do they mention that this is a breaking change to API. This has happened with v1.3.1, where they did add a notice that the API has changed to the release notes upon request, see openfheorg/openfhe-development#1046.

Conclusion: We cannot rely on API changes only happening with minor or major versions, could also happen in new patch release.

Footnotes

1. https://openfhe-development.readthedocs.io/en/latest/sphinx_rsts/misc/governance.html#release-numbering ↩

2. https://github.com/openfheorg/openfhe-development/pull/1088 ↩

[Tom-Finke]

Regarding changing compat bounds without a new release of openfhe-julia:

The Yggdrasil CONTRIBUTING docs mentions that if one wants to modify compat bounds

you first must correct the problem in the General registry. This is due to a limitation of Pkg which can't handle different compat bounds for the same package within releases which only differ by the build number¹

and to

make sure a pull request is not changing julia compat bounds without changing the version number of the package²

Footnotes

1. https://github.com/JuliaPackaging/Yggdrasil/blob/master/CONTRIBUTING.md#changing-compat-bounds-of-a-package ↩

2. https://github.com/JuliaPackaging/Yggdrasil/blob/master/CONTRIBUTING.md#information-for-maintainers ↩

[Tom-Finke]

@sloede decided: we will restrict openfhe-development to a concrete range and release a new openfhe-julia patch with every openfhe-development release

[Tom-Finke]

We had some feedback on our PR JuliaPackaging/Yggdrasil#13850.
One suggestion is to manage the version compatability in the downstream package, i.e. in OpenFHE.jl in our case, in order to avoid having to update the JLL.
They pin the two JLLs Singular.jl depends on to one exact version in Singular.jl.

If openfhe_julia is not intended to ever be used as a standalone package, I think this approach is okay, trading guaranteed compatibility further upstream for possibly less package management overhead.

[sloede]

I'd be OK with moving the compatibility handling completely to OpenFHE.jl.

[Tom-Finke]

Implementing with hpsc-lab/OpenFHE.jl#123.