Guidelines on how to contribute

[jplehr]

Hi everyone. We would like to start contributing to HeCBench with fixes and potentially more, if there is an appetite for it.

We have a few questions on how to contribute best:

• Should we file issues and then PRs to address those issues?
• Should we just open PRs with fixes, or should we open issues and then PRs?
• Is there a policy around AI-assisted contributions (human-reviewed before PRs)?

Thanks for your answers.

[zjin-lcf]

Hello @jplehr,

I appreciate your descriptions of an issue as I don't fully understand the implementation of every program. However, a PR with fixes may include descriptions of an issue as well. If AI creates a PR or a few PRs, they may address issues for one benchmark. There are issues that may exist in many programs. For example, "sm_60" in Makefiles is not supported by a recent CUDA compiler, then a PR (sm_60 -> sm_75) for all benchmarks is better and simpler. Thank you for your questions and contributions!

[keitaTN]

This is good suggestion. We will discuss this with the leads (me, Aaron Young and Jeff Vetter) team of this project.
There are action items.

1. I will reach out Olga Pearce at LLNL who is leading Benchmark Software Ecosystem across DOE. She is working across multiple benchmark software projects.
2. I am leaning toward opening issues/bugs, and then PR for update/fix indicating which issues being addressed.
3. We need to address AI contibutions as RIKEN is preparing for this. Again, I will talk to Olga for recommendations.

[keitaTN]

Internally, we are leaning toward to establish criteria for adding new benchmark code, and some repository structure changes.

[Geekdude]

Hi all, here are my thoughts:

1. We need to establish a contributing policy, either in the README.md or in a CONTRIBUTING.md file. This statement should also have statements about our expectation around AI use.
2. As far as contribution procedure, if it's a new benchmark or feature, I'm good with a PR only without an issue as there is no issue it is addressing. If instead the PR is addressing a bug or issue with an existing benchmark, then creating the issue first, then addressing it with the PR is my preference.
3. Regarding AI, this is a fast-moving area in the community, but my current position is that the quality of the code must be ensured by a human, I do not want fully agentic contributions without a human review, and a human must take ownership/responsibility of the code. If AI is used, I would like the committer to include the Assisted-by tag in the commit with information about the module used. For example Assisted-by: Claude Opus 4.5, these kind of tags are easy to add if the Agent is creating the commits. Just tell them to use Assisted-by instead of Co-authored-by which they seem to default to. See Assisted-by: How open source projects are drawing the line on AI contributions for a recent article I read on this.

[keitaTN]

We had a monthly meeting today. We are going to post a draft for the rule of PR and commit. Issues and PRs can be submitted by anybody. Any PR must be corresponding to a fork/branch. Each PR needs approvals from 2 reviewers before merging. We will implement this very soon once the formal draft is ready (will be posted soon).

@zjin-lcf I want you to be aware of this change. Please let me know if you have any questions.

[jplehr]

Thank you for all the remarks and thoughts.
We are sorting out a few things on our end as well. Once done, I'll check again for a contribution guide.

I agree with the human-ownership of the code that is submitted, as it is ultimately taking reviewer bandwidth.

[spophale]

Here are my thoughts, I agree with the discussion regarding AI/coding agent assisted codes.

1. Identify the primary maintainer(s) and active contributors for each key area/programming model.

2. Steps for contribution:
   a. Create an Issue - this maintains the history/discussion and may be helpful for future contributors
   b. Contributions should be made from a forked repo with branch name that reflects the Issue#
   c. All discussions should be on the Issue, including proposed solution
   d. The Pull Request (PR) created for solving the issue must be linked to the issue
   e. Approvals for new features/benchmarks should be discussed in the monthly meeting, while corrections can be merged without discussion in the monthly call with appropriate approvals.
   f. Require at-least two approvals for a PR before merging to the main repository.
   g. Merged PR should close the corresponding Issue

[Geekdude]

1. Identify the primary maintainer and active contributors for each key area/programming model.

I agree with @spophale and recommend that we add a CODEOWNERS file to specify the primary maintainer/active contributor for each key area/programming model/benchmark. See https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners for information about the CODEOWNERS file. The code owner will then need to approve any merge request that changes their area.

I also agree with the steps for contribution. They seem to follow best practices.