Add hector adapter

[znicholls]

The motivation

Adds hector the set of models that can be used by OpenSCM-Runner

The proposed solution

Add openscm_runner.adapaters.Hector. @kdorheim, @claudiatebaldi, and @noahprime will lead implementation.

This issue will act as the overarching issue for tracking this implementation. We can split out further issues as needed.

[znicholls]

Copying first set of discussions from email

 Should the SCM executable be compiled and ready to run without dependencies? Hector has a dependency on a C++ library and we are trying to figure out the best way to handle that. 

It's completely up to you (@openclimatedata worked out how to do the C distribution for pyhector so you could copy/use that https://github.com/openclimatedata/pyhector). We basically have 2 patterns in OpenSCM:

• specify as pypi dependency, everything just works: that's what FaIR and ciceroscmpy do (they're pure python)
• tell the user where to get the dependencies (pre-compiled or not), how to tell openscm-runner where to find them, how to set runtime environment etc. then run: that's what MAGICC does
  CICREO-SCM just put the binary in the openscm-runner package, but we don't want to do that anymore (see Remove CICERO-SCM binary #49)

On a somewhat related note, we’ve been struggling to get the Cicero running. Do you know if there is an extra dependency on our end that we would need to install?

CICEROSCMPY should just work. CICEROSCM requires the binary, but that's included in the package. The binary only works on linux systems. If you're on linux and it's still not working, probably best to make an issue and tag maritsandstad (she is generally very responsive).

 Do modeling groups submit their own parameters for the 5/95th percentiles? Or does OpenSCM runner calculate them based on an ensemble of runs?

Neither really. OpenSCM-Runner just does the running (take a bunch of different model configs, run the model, return the results from each run). Then users can process them however they want. We tend to use process_over from scmdata but you could use whatever (https://scmdata.readthedocs.io/en/latest/notebooks/summary-statistics.html#process-over).

 The examples in the lab notebooks look like scenario emissions start in 2015, what is being used for the historical portion of the run?

Good question.
So, the ideal pattern is: user specifies emissions for the entire run, the model only runs over the period for which the emissions are provided.
However, the real-world pattern (i.e. supporting IPCC emissions databases is): user specifies emissions for the future only, the model then has to add its own historical emissions onto these in order to create a full timeseries it can run with. At the moment, each model has its own module for grabbing historical emissions (e.g. for FaIR, it is happening in here:

openscm-runner/src/openscm_runner/adapters/fair_adapter/_scmdf_to_emissions.py

Line 199 in 261bd17

for species in EMISSIONS_SPECIES_UNITS_CONTEXT["species"]:

). In theory, this means they could all be using different historical emissions. In practice, we did check this before AR6 but it's obviously something we want to improve at some point. I don't think the solution is particularly hard, so if you want to tackle that problem please feel free.

 What units does OpenSCM runner expect the emission units in? Are they the same units as the RCMIP emission files?

Any: the idea is that the adapters deal with conversion from emissions' input units (whatever they may be) into the emissions required by the model. We generally use ScmRun's unit conversion for this (https://scmdata.readthedocs.io/en/latest/notebooks/scmrun.html#unit-conversion) but you are free to use whatever unit conversion set up you want before the data gets passed to Hector.

[rgieseke]

Cheers, great to see plans for OpenSCM extension!

I had tried a (long) while ago (unsuccessfully) to get Pyhector running in OpenSCM-Runner (#24 (comment)).

As for Pyhector, there is still some work needed to update Pyhector with Hector 3 (openclimatedata/pyhector#63).
Using Pyhector might have faster runs than dealing with writing emissions files and calling a compiled executable.
It has been mainly developed and tested on Linux (and Mac) but Windows compilation and usage has also been reported (just not been streamlined yet).

[maritsandstad]

Just to answer the ciceroscm part. I did add a Windows executable at some point, but whether that still works, I don't know. I'm not particularly interested in updating that though...

[kdorheim]

Re ciceroscm - @maritsandstad & @znicholls good news I was able to get cicero running with the open scm-runner, there were additional dependencies we needed install first. Sorry about that!

Re pyhector - @znicholls & @rgieseke that is a good point about pyhector unfortunately I don't think we have the resources to contribute to the pyhector to V3 update and integrate it here. we will probably follow MAGICC's set or something, I plan on talking to our open source software mastermind Chris Vernon next week.

@znicholls Thanks for your answers! They were really helpful!

[rgieseke]

Re pyhector - @znicholls & @rgieseke that is a good point about pyhector unfortunately I don't think we have the resources to contribute to the pyhector to V3 update and integrate it here. we will probably follow MAGICC's set or something, I plan on talking to our open source software mastermind Chris Vernon next week.

FWIW, the Pyhector v3 update is almost done. A few minor things to clean up and check, then I'll make a new release.

[kdorheim]

@rgieseke that is awesome!

[noahprime]

Hello, opening this conversation back up as we have an adapter for Hector that's working by calling a built version of Hector akin to the cicero solution. I understand packaging binaries isn't the solution we're looking for, but I'm lacking experience here on what to do.

tell the user where to get the dependencies (pre-compiled or not), how to tell openscm-runner where to find them, how to set runtime environment etc. then run: that's what MAGICC does

^ Is this the path I should be exploring? Users could download and build Hector independently and somehow direct openscm-runner to find that build?

Here's the fork I've been working off of and how the Hector executable gets called:

https://github.com/noahprime/openscm-runner/blob/c36933ad9baf6b9ef14e63530387951c2c8de015/src/openscm_runner/adapters/hector_adapter/hector_wrapper.py#L93-L97

[znicholls]

Is this the path I should be exploring? Users could download and build Hector independently and somehow direct openscm-runner to find that build?

This is what I would do (put pre-built binaries somewhere and point users to them) if the pyhector option is a non-starter for you.