Skip to content

feat: add data_purposes to Dataset, DatasetCollection, DatasetField #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
galvana merged 4 commits into from
Mar 25, 2026
+29 −19
Merged

feat: add data_purposes to Dataset, DatasetCollection, DatasetField #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
71172a3
feat: add data_purposes field to Dataset, DatasetCollection, and Data…
Mar 16, 2026
22bc6b7
style: reformat with black (CI formatter)
Mar 16, 2026
71b5a4e
fix: resolve mypy errors from pydantic dependency drift
Mar 23, 2026
9ad251d
fix: properly fix mypy errors instead of suppressing with type: ignore
Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@ where = ["src"]
[tool.mypy]
check_untyped_defs = true
disallow_untyped_defs = true
disallow_any_explicit = true
files = ["src"]
no_implicit_reexport = true
plugins = ["pydantic.mypy"]
Expand Down
47 changes: 29 additions & 18 deletions src/fideslang/models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"""
Contains all of the Fides resources modeled as Pydantic models.
"""

from __future__ import annotations

from datetime import datetime
Expand All @@ -19,7 +20,6 @@
HttpUrl,
PositiveInt,
SerializeAsAny,
ValidationInfo,
field_validator,
model_validator,
)
Expand Down Expand Up @@ -75,11 +75,11 @@ class MaskingStrategyOverride(BaseModel):
strategy: MaskingStrategies


class FieldMaskingStrategyOverride(BaseModel): # type: ignore[misc]
class FieldMaskingStrategyOverride(BaseModel):
"""Overrides field-level masking strategies."""

strategy: str
configuration: Optional[Dict[str, Any]] = {} # type: ignore[misc]
configuration: Optional[Dict[str, Any]] = {}


class FidesModel(BaseModel):
Expand Down Expand Up @@ -365,6 +365,10 @@ class MyDatasetField(DatasetFieldBase):
default=None,
description="Arrays of Data Categories, identified by `fides_key`, that applies to this field.",
)
data_purposes: Optional[List[FidesKey]] = Field(
default=None,
description="Array of Data Purpose resources, identified by `fides_key`, that apply to this field.",
)
Comment on lines +368 to +371
Copy link
Copy Markdown

@adamsachs adamsachs Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this deliberately an Optional[List] that defaults to None rather than a List[] that defaults to []?

i realize this follows the data_categories convention above, but i just want to make sure we're not perpetuating a bad design choice. do we imagine that None will signify something different than an empty list ([]) in this case? perhaps the flexibility is good for a model as foundational as this one, because there may be many different applications, and it can be hard to anticipate whether some application may want to distinguish None from [] in the future, even if we don't have that use case now.

i can conceive of different meanings for None vs [] (e.g. None = 'hasn't yet been reviewed/annotated for data_purposes'; [] = 'reviewed and determined no data_purposes apply') so i think this choice is justifiable. but just wanted to raise this for thought/discussion/confirmation!

Copy link
Copy Markdown

@adamsachs adamsachs Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK and now i see the PR note about backward compatibility. that seems like enough justification 👍



class EdgeDirection(str, Enum):
Expand Down Expand Up @@ -481,7 +485,6 @@ def valid_meta(cls, meta_values: Optional[FidesMeta]) -> Optional[FidesMeta]:
@model_validator(mode="after")
def validate_object_fields(
self,
_: ValidationInfo,
) -> DatasetField:
"""Two validation checks for object fields:
- If there are sub-fields specified, type should be either empty or 'object'
Expand Down Expand Up @@ -555,14 +558,18 @@ class DatasetCollection(FidesopsMetaBackwardsCompat):
default=None,
description="Array of Data Category resources identified by `fides_key`, that apply to all fields in the collection.",
)
data_purposes: Optional[List[FidesKey]] = Field(
default=None,
description="Array of Data Purpose resources, identified by `fides_key`, that apply to all fields in the collection.",
)
fields: List[DatasetField] = Field(
description="An array of objects that describe the collection's fields.",
)

fides_meta: Optional[CollectionMeta] = None

_sort_fields: classmethod = field_validator("fields")(sort_list_objects_by_name) # type: ignore[assignment]
_unique_items_in_list: classmethod = field_validator("fields")(unique_items_in_list) # type: ignore[assignment]
_sort_fields: classmethod = field_validator("fields")(sort_list_objects_by_name)
_unique_items_in_list: classmethod = field_validator("fields")(unique_items_in_list)


class ContactDetails(BaseModel):
Expand Down Expand Up @@ -618,17 +625,21 @@ class Dataset(FidesModel, FidesopsMetaBackwardsCompat):
default=None,
description="Array of Data Category resources identified by `fides_key`, that apply to all collections in the Dataset.",
)
data_purposes: Optional[List[FidesKey]] = Field(
default=None,
description="Array of Data Purpose resources, identified by `fides_key`, that apply to all collections in the Dataset.",
)
fides_meta: Optional[DatasetMetadata] = Field(
description=DatasetMetadata.__doc__, default=None
)
collections: List[DatasetCollection] = Field(
description="An array of objects that describe the Dataset's collections.",
)

_sort_collections: classmethod = field_validator("collections")( # type: ignore[assignment]
_sort_collections: classmethod = field_validator("collections")(
sort_list_objects_by_name
)
_unique_items_in_list: classmethod = field_validator("collections")( # type: ignore[assignment]
_unique_items_in_list: classmethod = field_validator("collections")(
unique_items_in_list
)

Expand Down Expand Up @@ -807,10 +818,10 @@ class Policy(FidesModel):
description=PolicyRule.__doc__,
)

_sort_rules: classmethod = field_validator("rules")(sort_list_objects_by_name) # type: ignore[assignment]
_sort_rules: classmethod = field_validator("rules")(sort_list_objects_by_name)


def validate_deprecated_cookies(values: Dict[str, Any] | Any) -> None: # type: ignore[misc]
def validate_deprecated_cookies(values: Dict[str, Any] | Any) -> None:
"""
Shared function to validate that the `cookies` field is deprecated and warn that it should not be used.
"""
Expand Down Expand Up @@ -903,7 +914,7 @@ class PrivacyDeclaration(BaseModel):

@model_validator(mode="before")
@classmethod
def validate_cookies(cls, values: Dict[str, Any] | Any) -> Dict[str, Any]: # type: ignore[misc]
def validate_cookies(cls, values: Dict[str, Any] | Any) -> Dict[str, Any]:
"""
Validate that the `cookies` field is deprecated and warn that it should not be used.
"""
Expand Down Expand Up @@ -1126,14 +1137,14 @@ class System(FidesModel):

@model_validator(mode="before")
@classmethod
def validate_cookies(cls, values: Dict[str, Any] | Any) -> Dict[str, Any]: # type: ignore[misc]
def validate_cookies(cls, values: Dict[str, Any] | Any) -> Dict[str, Any]:
"""
Validate that the `cookies` field is deprecated and warn that it should not be used.
"""
validate_deprecated_cookies(values)
return values

_sort_privacy_declarations: classmethod = field_validator("privacy_declarations")( # type: ignore[assignment]
_sort_privacy_declarations: classmethod = field_validator("privacy_declarations")(
sort_list_objects_by_name
)

Expand Down Expand Up @@ -1180,11 +1191,11 @@ class Taxonomy(BaseModel):
"""

data_category: List[DataCategory] = Field(default_factory=list)
data_subject: Optional[List[DataSubject]] = Field(default_factory=list)
data_use: Optional[List[DataUse]] = Field(default_factory=list)
data_subject: List[DataSubject] = Field(default_factory=list)
data_use: List[DataUse] = Field(default_factory=list)

dataset: Optional[List[Dataset]] = Field(default_factory=list)
system: Optional[List[System]] = Field(default_factory=list)
policy: Optional[List[Policy]] = Field(default_factory=list)
dataset: List[Dataset] = Field(default_factory=list)
system: List[System] = Field(default_factory=list)
policy: List[Policy] = Field(default_factory=list)

organization: List[Organization] = Field(default_factory=list)
Loading