Skip to content

fix: detect publishing mode from the platform for target= pipelines (#250) - #538

Draft
riyaz-ali-mohammad wants to merge 1 commit into
databrickslabs:v0.2.0from
riyaz-ali-mohammad:issue_250
Draft

riyaz-ali-mohammad wants to merge 1 commit into
databrickslabs:v0.2.0from
riyaz-ali-mohammad:issue_250

Conversation

@riyaz-ali-mohammad

Copy link
Copy Markdown

Fixes #250.

Supersedes the closed main-targeted #535. Raised against v0.2.0 per the release flow (issue_branch → release_branch → main).

Problem

When a pipeline is configured with target= (legacy publishing-mode spec) but the platform reports DEFAULT_PUBLISHING_MODE, bronze tables land in the pipeline's default schema instead of the schema set in onboarding (bronze_database_dev).

Root cause: DataflowPipeline._is_dpm_enabled inferred the publishing mode only from pipelines.schema. That Spark conf is not set for target= pipelines, so sdp-meta fell back to bare table names. Lakeflow then created the tables in the pipeline's default schema.

Fix

  • Read effective_publishing_mode from the pipelines API via WorkspaceClient().pipelines.get(pipeline_id=...), using the runtime pipelines.id.
  • Cache successful results, API failures, and unrecognised modes per pipeline id, so the control plane is contacted at most once per pipeline per process (addresses review feedback on fix: detect publishing mode from the platform for target= pipelines (#250) #535).
  • Fall back to the existing pipelines.schema check when the id is missing or the cached platform result is None. Detection never fails a pipeline on its own.
  • Update the legacy-mode log message, which previously claimed pipelines.schema is unset as the cause.

Tests

  • New PlatformPublishingModeDetectionTests (9 tests): platform default wins when pipelines.schema is unset (regression for Bronze Tables Not Created in Onboarding-Specified Schema #250), platform legacy wins over the conf, API failure falls back to conf, API failure without conf is legacy, unrecognised mode falls back, no pipeline id skips the API, and caching of successes, failures, and unrecognised modes.
  • LegacyPublishingModeTests + new tests: 28 passed locally.
  • flake8 clean on the touched files.

End-to-end verification (from the earlier #535 work)

Same serverless UC pipeline (target= configured, pipelines.schema unset, platform reports DEFAULT_PUBLISHING_MODE), same onboarding spec, new table customers_e2e onboarded to bronze schema sdpmeta_issue250_bronze. No monkeypatching; only the wheel changed.

Run has_api_detection API mode sdp-meta dpm_enabled customers_e2e in bronze schema customers_e2e in pl_default
Unpatched wheel 0.1.1 false n/a false (bare names) missing 3 rows (bug)
Patched wheel true DEFAULT_PUBLISHING_MODE true (qualified names) 3 rows leftover from baseline

Draft / next steps

Per Ravi's feedback on #535:

  1. Cache failed and unrecognized API results (done in this PR).
  2. Waiting for #534 to land.
  3. Rebase onto the updated v0.2.0 after Add pluggable Lakeflow and DQX quality engines to v0.2.0 #534 merges, preserve v0.2.0 changelog content, then mark ready for review.

This PR is a draft until step 3.

…atabrickslabs#250)

Bronze tables were created in the pipeline's default schema instead of the
onboarding-specified schema when the pipeline was configured with `target=`.

`DataflowPipeline._is_dpm_enabled` only inspected `pipelines.schema`, which
is unset for `target=` pipelines even when the platform reports default
publishing mode. sdp-meta then used bare table names, and Lakeflow placed
the tables in the pipeline's default schema.

Read `effective_publishing_mode` from the pipelines API using the runtime
`pipelines.id`, cache successes, API failures, and unrecognised modes per
pipeline, and fall back to the `pipelines.schema` check when the platform
result is unavailable.

Tests: PlatformPublishingModeDetectionTests cover the platform-default and
platform-legacy cases, API failure fallback, unknown mode, missing pipeline
id, and caching of successes, failures, and unrecognised modes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant