Create run with TaskId for lazy task

[popojk]

Goal

While creating runs with remote task(lazy task), we should create run request with TaskID instead of TaskSpec. Thus, the user only needs Action_ACTION_CREATE_FLYTE_EXECUTIONS to do such an operation.

p.s. While executing ephemeral runs(create runs with TaskSpec), the user must process Action_ACTION_REGISTER_FLYTE_INVENTORY permission because the BE will create TaskSpec in the DB.

Tests

Add log to see if we are creating run with TaskID or TaskSpec:

Test with this script to run remote task:

import flyte

if __name__ == "__main__":
    flyte.init_from_config()
    task = flyte.remote.Task.get("named_fanout.fanout", auto_version="latest")
    result = flyte.run(task, n1=2, n2=3, n3=4)
    print(f"Result: {result}")

The log shows we create run with TaskID

Execute an ephemeral run, the log show we create run with TaskSpec:

Test if task override works by overriding the task Short_name:

import flyte

if __name__ == "__main__":
    flyte.init_from_config()
    task = flyte.remote.Task.get("named_fanout.fanout", auto_version="latest").override(short_name="overridden_my_task")
    result = flyte.run(task, n1=2, n2=3, n3=4)
    print(f"Result: {result}")

The name is overridden:

Rollout plan

The backend permission check changes is till processing. It is ok to merge this PR first as current BE authz logic will not block create run request with TaskID.

[pingsutw]

does it work with task override?

[popojk]

Thanks for noticing. Fixed in c0ab563. I also tested it and put the result in PR description.

[popojk]

Add this var in TaskDetails to flag if the fetched TaskDetail is overridden. If it is overridden, we should still send full TaskSpec to the backend while creating runs.

[kumare3]

so does flyte run remote-task not work already?

[popojk]

so does flyte run remote-task not work already?

@kumare3 Right now the flyte run remote-task will send the full TaskSpec to the BE while creating runs. However, we could just send TaskID to the BE as the task is already deployed, and that is the point of this PR. We made RBAC changes in the BE so ephemeral runs(create runs with TaskSpec) will require Deploy_Inventory permission because the BE will create TaskSpec data in the DB for ephemeral runs.