pip.parse vendored support

[gfrankliu]

We are currently using WORKSPACE and pip_parse_vendored following the examples in https://github.com/bazelbuild/rules_python/tree/main/examples/pip_parse_vendored

While migrating to MODULE.bazel, there are no similar pip.parse_vendored example. In particular, I am looking for the equivalent of https://github.com/bazelbuild/rules_python/blob/main/examples/pip_parse_vendored/BUILD.bazel#L24-L34

Looking at pip_parse generated @pip_deps_to_be_vendored//:requirements.bzl and pip.parse generated :requirements.bzl , I see the latter misses _config = { ... } and _packages = [ ... ]
Is that a bug?

[gfrankliu]

Here is a quick test example to show the differences in the requirements.bzl

Using WORKSPACE and pip_parse:

# WORKSPACE
python_register_toolchains(
    name = "python_interpreter",
    ignore_root_user_error = True,
    python_version = "3.10.11",
)
load("@python_interpreter//:defs.bzl", "interpreter")
pip_parse(
    name = "test_lib",
    enable_implicit_namespace_pkgs = True,
    python_interpreter_target = interpreter,
    requirements_lock = "//:requirements.txt",
)

# BUILD
genrule(
    name = "test_bzl",
    srcs = ["@test_lib//:requirements.bzl"],
    outs = ["test.bzl"],
    cmd = "cp $< $@",
)

Using the new MODULE.bazel and pip.parse:

# MODULE.bazel
python.toolchain(
    ignore_root_user_error = True,
    is_default = True,
    python_version = "3.10.11",
)
use_repo(python, "python_3_10_11")
pip.parse(
    hub_name = "test_lib",
    enable_implicit_namespace_pkgs = True,
    python_version = "3.10.11",
    requirements_lock = "//:requirements.txt",
)

# BUILD
genrule(
    name = "test_bzl",
    srcs = ["@test_lib//:requirements.bzl"],
    outs = ["test.bzl"],
    cmd = "cp $< $@",
)

You can see the new test.bzl misses:

load("@rules_python//python/pip_install:pip_repository.bzl", "whl_library")

_packages =
_config = 
def entry_point(pkg, script = None):
def _get_annotation(requirement):
def install_deps(**whl_library_kwargs):

Is this a bug in the new pip.parse?

[aignas]

bzlmod has lockfile mechanism and its own vendoring code. Does it not satisfy your needs?

There are no plans to add vendoring in the same way it was done in WORKSPACE.

[gfrankliu]

We have some internal custom bazel rules that

• uses _config (specifically extra_pip_args from _config) from the old requirement.bzl generated by pip_parse. The new pip.parse generated requirement.bzl misses the whole _config.
• use _packages from the old requirement.bzl generated by pip_parse. The new pip.parse generated requirement.bzl misses _packages

The python modules directory under runfiles also look different, eg: module httplib2 looks like app.runfiles/third_party_lib_httplib2/site-packages/ when using WORKSPACE/pip_parse, but looks like app.runfiles/rules_python~~pip~third_party_lib_310_httplib2/site-packages/ when using bzmod/pip.parse

Is it possible to configure pip.parse to behave more like pip_parse to make the rules_python WORKSPACE to bzlmod migration easier?

[aignas]

It seems that you may be depending on implementation details within rules_python and unfortunately some things just are not possible in bzlmod:

• Having extra_pip_args might be possible, but in general they can be different for each target platform, so supporting that might be tough. What is the use case here?
• What is the use case for _packages?
• The layout is not controlled by us and is imposed by bzlmod itself, so unfortunately that won't be possible.

You can go the other way though to first make pip_parse closer to what pip.parse is. We tried to make it possible to do everything that you can do through the API exposed by pip_parse (i.e. the attributes and the publicly available constants in requirements.bzl) to be possible in pip.parse bzlmod extension.

Since there is nothing to do here, I will migrate this to a discussion.