Purpose-Based Data Model: OSS models, schemas, and migration#7635
Draft
Purpose-Based Data Model: OSS models, schemas, and migration#7635
Conversation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Clarify FidesBase inheritance pattern, add cascade/delete behavior, fix response schema conventions, document facade coercion, add comprehensive index specifications, and resolve GET-by-ID ambiguity. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dataset-level purposes stored as ARRAY(String) column on ctl_datasets. Collection/field/sub-field purposes remain as soft refs in JSON. All dataset purpose references are fides_key strings validated on write. No join table needed for dataset-to-purpose relationships. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
18-task plan across 4 chunks covering models/migration (fides OSS), services, API routes (fidesplus), and dataset integration. Reviewed and validated against the design spec. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DataPurpose goes in src/fides/api/models/data_purpose.py instead of sql_models.py. Imports FidesBase from sql_models without circular dependency since sql_models doesn't import back. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Model tests use the standard db session fixture and don't need the postgres marker. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces the DataPurpose SQLAlchemy model as a centrally-governed, reusable declaration of why data is processed. Follows the same pattern as DataCategory/DataUse (Base + FidesBase, fides_key uniqueness enforced in create override). Registers the model in base.py for Alembic discovery. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolves mypy errors for string-based relationship references. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add bidirectional relationships between DataPurpose, DataConsumer, DataProducer, System, and their join tables. Include migration for all purpose-model tables (data_purpose, data_consumer, data_producer, system_purpose, data_consumer_purpose, data_producer_member). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 2 Skipped Deployments
|
…onsumer_purpose table The migration had a truncated line (sa on its own) and was missing: - data_purpose columns (data_use, data_subject, data_categories, etc.) - data_consumer_purpose join table creation - data_purpose indexes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticket ENG-2756
Description Of Changes
Add the foundational data model for purpose-based access control (PBAC) to Fides OSS. This introduces new SQLAlchemy models, Pydantic schemas, and an Alembic migration for tracking data purposes, data consumers, data producers, and their relationships to systems and datasets.
The purpose-based data model enables organizations to define why data is collected/processed (purposes), who consumes it (consumers), and who produces it (producers), with many-to-many relationships linking purposes to systems, consumers, and datasets.
Code Changes
src/fides/api/models/data_purpose.py- New DataPurpose model with taxonomy fields (data_use, data_subject, legal_basis, retention, etc.)src/fides/api/models/system_purpose.py- SystemPurpose join table linking systems to purposes with assigned_by trackingsrc/fides/api/models/data_consumer.py- DataConsumer model with contact info, egress/ingress, and DataConsumerPurpose join tablesrc/fides/api/models/data_producer.py- DataProducer model with DataProducerMember join table for user assignmentssrc/fides/api/models/sql_models.py- Added system_purposes relationship to System modelsrc/fides/api/db/base.py- Registered new models for Alembic discoverysrc/fides/api/schemas/data_purpose.py- Create/Update/Response Pydantic schemas for DataPurposesrc/fides/api/schemas/data_consumer.py- Create/Update/Response schemas for DataConsumer with purpose assignmentsrc/fides/api/schemas/data_producer.py- Create/Update/Response schemas for DataProducer with member managementsrc/fides/api/alembic/migrations/versions/xx_2026_03_12_...- Migration creating all PBAC tables with indexes and foreign keysSteps to Confirm
nox -s "pytest(ops-unit)" -- tests/ops/models/test_data_purpose.py tests/ops/models/test_system_purpose.py tests/ops/models/test_data_consumer.py tests/ops/models/test_data_producer.py tests/ops/models/test_dataset_purposes.py -vnox -s check_migrationsnox -s static_checksPre-Merge Checklist
CHANGELOG.mdupdatedmaindowngrade()migration is correct and works