diff --git a/PYEGERIA_ISSUES.md b/PYEGERIA_ISSUES.md index 23522a71..22e2dafa 100644 --- a/PYEGERIA_ISSUES.md +++ b/PYEGERIA_ISSUES.md @@ -309,7 +309,27 @@ Tests: `tests/micro-tests/test_issue_160_template_placeholders.py` (5 cases; all **Live check 2026-10-21 14:52Z** (Egeria `main` 05a5e982, erinoverview, after a peer round; the fixed code on this branch): `create_folder_element_from_template("/tmp/pyeg_rt_folder_20261021T145254Z", ..., version="1.0")` created FileFolder `e9a45c7f` with `deployedImplementationType` "File System Directory" and `versionIdentifier` "1.0". Its anchored Endpoint `8f003544` had **`networkAddress` = `/tmp/pyeg_rt_folder_20261021T145254Z`**. The folder, Connection `18ef2ee9` and Endpoint had no `~{...}~` left in their own properties. A metadata-expert delete (`deleteMethod: PURGE`, `cascadedDelete: true`) removed all three; each was verified gone by GUID. A first run (`a3475d3e`) confirmed the folder fields but its script missed the Endpoint. Exact `networkAddress` searches (forLineage) afterwards found no leftover for that run either. The survey itself was not run. -**Not fixed: needs the owner's decision.** These are API additions or behaviour choices, not one-line key fixes. +**Follow-up fix 2026-10-21 (owner: "fix them all in a new pr")** on branch `fix/issue-160-remaining-placeholders`, built on #474. All 13 hard-coded builders now wrap their values in `_placeholder_values`, which sends `""` for any placeholder left `None`, so no `~{name}~` literal is stored. `body_slimmer` drops only `None`. A folder created without a file system therefore gets qualified name `FileFolder:::/path`. +- **Postgres server and database:** new optional `secrets_store_path_name` / `secrets_collection_name`, sent as `secretsStorePathName` / `secretsCollectionName`, plus `version` (and `resource_name` on the server, defaulting to the server name). `db_user` / `db_pwd` are now optional, deprecated and no longer sent; passing either raises a `DeprecationWarning`. +- **Kafka server:** `resourceName` (defaults to the server name) and `versionIdentifier`. +- **Unity Catalog:** + - UC server: `resourceName`, defaulting to the server name; + - UC catalog: `ucServerQualifiedName`; + - UC table and function: `rootSchemaTypeQualifiedName`; + - UC volume: `directoryPathName` and `localStorageLocation`. + Each is a new optional parameter on the async method and its sync wrapper. +- **Item 4 needed no change:** "Keystore File" is a real template (`DataAssetTemplateDefinition.KEYSTORE_FILE_TEMPLATE`), and live, both "YAML File Secrets Collection" and "Keystore File" resolve to template `fbcfcc0c`. + +Tests: `tests/micro-tests/test_issue_160_remaining_placeholders.py`, 11 cases, all failing on #474's code. The full micro-test suite (1,420) passes in a clean python:3.13-slim container. **Follow-up not in this fix:** Dr.Egeria's `Postgres Server Element` / `Postgres Database Element` commands still pass `Database User Id` / `Database Password`. Those are now ignored with a warning; their compact specs need `Secrets Store Path Name` / `Secrets Collection Name` attributes. + +**Follow-up live check 2026-10-21 15:16Z** (Egeria `main` 05a5e982, erinoverview, after a peer round; #475 code): +- **(a)** `create_postgres_server_element_from_template("pyeg_rt_pg_…", "pyeg-rt.invalid", "5432", secrets_store_path_name=…, secrets_collection_name=…)`, with no `db_user` / `db_pwd`, created SoftwareServer `b6b2ebff`: `resourceName` = the server name, `versionIdentifier` and `description` = `""`, and no `~{...}~` in its own properties. Its Connection `55686913` and Endpoint `59014065` (`jdbc:postgresql://pyeg-rt.invalid:5432/postgres`) had no literals either. +- The server had **no CatalogTarget link** (its relationships were sourcedFromTemplate, serverEndpoints, capabilities, connections), so the JDBC integration connector never saw it. +- I did not find where Egeria stored the secrets-store values. They were not in the Connection's own properties, and no literal was left. Unconfirmed. +- **(b)** `create_folder_element_from_template("/tmp/pyeg_rt_folder2_…", …, None)`, with no file system, version or description: qualifiedName `FileFolder:::/tmp/pyeg_rt_folder2_…`, `resourceName` `:/tmp/…`, `versionIdentifier` / `description` = `""`, `deployedImplementationType` "File System Directory", and Endpoint `networkAddress` = the path. No literals on the folder, Connection or Endpoint. +- Both were purged (metadata-expert delete, `PURGE`, `cascadedDelete`), and all six GUIDs were verified gone. Name, host, stamp and secrets-file searches afterwards found nothing, which covers the server's capabilities too. + +**Originally listed as not fixed** (kept for the record): 1. **Postgres server and database credentials:** both templates now take `secretsStorePathName` and `secretsCollectionName` (credentials come from a secrets store). pyegeria never sends them, so the connection keeps literals. pyegeria still sends `databaseUserId` / `databasePassword`, which the templates no longer list (`312c172639`, `56fa898e43`), so those are dead. Proposed: add optional `secrets_store_path_name` / `secrets_collection_name` parameters, and deprecate `db_user` / `db_pwd`. 2. **Optional arguments left unset:** `versionIdentifier`, `description`, `fileSystemName` and the like are dropped when `None`, so they stay `~{...}~` (RE's `versionIdentifier` case). Options: send `""` for unset optional placeholders, or document that callers must pass them. 3. **Unsent template placeholders in the Unity Catalog and Kafka builders:** diff --git a/pyegeria/omvs/automated_curation.py b/pyegeria/omvs/automated_curation.py index 3bc02107..578b680d 100644 --- a/pyegeria/omvs/automated_curation.py +++ b/pyegeria/omvs/automated_curation.py @@ -7,6 +7,7 @@ """ import asyncio +import warnings from datetime import datetime from httpx import Response @@ -673,6 +674,12 @@ def create_elem_from_template(self, body: Optional[dict | TemplateRequestBody] = ) return response + @staticmethod + def _placeholder_values(values: dict) -> dict: + """ISSUE-160: Egeria stores an unsent template placeholder as the literal ~{name}~, and body_slimmer + drops None values, so send "" for any placeholder the caller left unset.""" + return {k: ("" if v is None else v) for k, v in values.items()} + async def _async_create_secrets_store_element_from_template( self, file_path_name: str, @@ -731,7 +738,7 @@ async def _async_create_secrets_store_element_from_template( { "templateGUID": "130d819e-e17d-46bf-bfea-d09d862e341f", "isOwnAnchor": true, - "placeholderPropertyValues": { + "placeholderPropertyValues": self._placeholder_values({ "fileSystemName": "", "filePathName": "secrets/mySecrets.omsecrets", "fileName": "mySecrets.omsecrets", @@ -740,7 +747,7 @@ async def _async_create_secrets_store_element_from_template( "fileExtension": "omsecrets", "fileEncoding": "YAML", "versionIdentifier": "V1.0" - } + }) } ``` """ @@ -854,6 +861,8 @@ async def _async_create_kafka_server_element_from_template( host_name: str, port: str, description: Optional[str] = None, + resource_name: Optional[str] = None, + version: Optional[str] = None, ) -> str: """Create a Kafka server element from a template. Async version. @@ -870,6 +879,10 @@ async def _async_create_kafka_server_element_from_template( description: str, opt A description of the Kafka server. + resource_name : str, optional + Technical identifier for the element (the template's resourceName). Defaults to the element's name. + version : str, optional + Version identifier for the element. @@ -883,12 +896,14 @@ async def _async_create_kafka_server_element_from_template( "class": "TemplateRequestBody", "templateGUID": template_guid, "isOwnAnchor": True, - "placeholderPropertyValues": { + "placeholderPropertyValues": self._placeholder_values({ "serverName": kafka_server, "hostIdentifier": host_name, "portNumber": port, "description": description, - }, + "resourceName": resource_name or kafka_server, + "versionIdentifier": version, + }), } body_s = body_slimmer(body) return await self._async_create_elem_from_template(body_s) @@ -899,6 +914,8 @@ def create_kafka_server_element_from_template( host_name: str, port: str, description: Optional[str] = None, + resource_name: Optional[str] = None, + version: Optional[str] = None, ) -> str: """Create a Kafka server element from a template. @@ -915,6 +932,10 @@ def create_kafka_server_element_from_template( description: str, opt A description of the Kafka server. + resource_name : str, optional + Technical identifier for the element (the template's resourceName). Defaults to the element's name. + version : str, optional + Version identifier for the element. @@ -926,7 +947,11 @@ def create_kafka_server_element_from_template( loop = asyncio.get_event_loop() response = loop.run_until_complete( self._async_create_kafka_server_element_from_template( - kafka_server, host_name, port, description + kafka_server, host_name, port, description, + + resource_name=resource_name, + + version=version, ) ) return response @@ -974,7 +999,7 @@ async def _async_create_csv_data_file_element_from_template( "class": "TemplateRequestBody", "templateGUID": template_guid, "isOwnAnchor": True, - "placeholderPropertyValues": { + "placeholderPropertyValues": self._placeholder_values({ "fileName": file_name, "fileType": file_type, "filePathName": file_path_name, @@ -986,7 +1011,7 @@ async def _async_create_csv_data_file_element_from_template( "fileExtension": file_extension, "fileSystemName": file_system_name, "description": description - }, + }), } body_s = body_slimmer(body) return await self._async_create_elem_from_template(body_s) @@ -1085,7 +1110,7 @@ async def _async_get_create_csv_data_file_element_from_template( "templateGUID": template_guid, "isOwnAnchor": True, "allowRetrieve": True, - "placeholderPropertyValues": { + "placeholderPropertyValues": self._placeholder_values({ "fileName": file_name, "fileType": file_type, "filePathName": file_path_name, @@ -1097,7 +1122,7 @@ async def _async_get_create_csv_data_file_element_from_template( "fileExtension": file_extension, "fileSystemName": file_system_name, "description": description - }, + }), **kwargs, } body_s = body_slimmer(body) @@ -1165,9 +1190,13 @@ async def _async_create_postgres_server_element_from_template( postgres_server: str, host_name: str, port: str, - db_user: str, - db_pwd: str, + db_user: Optional[str] = None, + db_pwd: Optional[str] = None, description: Optional[str] = None, + secrets_store_path_name: Optional[str] = None, + secrets_collection_name: Optional[str] = None, + version: Optional[str] = None, + resource_name: Optional[str] = None, ) -> str: """Create a Postgres server element from a template. Async version. @@ -1190,6 +1219,16 @@ async def _async_create_postgres_server_element_from_template( description: str, opt A description of the element. + secrets_store_path_name : str, optional + Path name of the secrets store file holding the connection's credentials (secretsStorePathName). + secrets_collection_name : str, optional + Name of the secrets collection in that store (secretsCollectionName). + version : str, optional + Version identifier for the element. + resource_name : str, optional + Technical identifier for the element (the template's resourceName). Defaults to the element's name. + db_user, db_pwd : str, optional + Deprecated and ignored (ISSUE-160): the templates now take credentials from a secrets store. @@ -1198,20 +1237,28 @@ async def _async_create_postgres_server_element_from_template( str The GUID of the Postgres server element. """ + if db_user is not None or db_pwd is not None: + # ISSUE-160: Egeria's Postgres templates take credentials from a secrets store + # (secretsStorePathName / secretsCollectionName); databaseUserId / databasePassword + # are no longer template placeholders, so these values are not sent. + warnings.warn("db_user and db_pwd are deprecated and ignored; pass secrets_store_path_name " + "and secrets_collection_name instead (ISSUE-160)", DeprecationWarning, stacklevel=2) template_guid = await self._async_get_template_guid_for_technology_type("PostgreSQL Server") body = { "class": "TemplateRequestBody", "templateGUID": template_guid, "isOwnAnchor": True, - "placeholderPropertyValues": { + "placeholderPropertyValues": self._placeholder_values({ "serverName": postgres_server, "hostIdentifier": host_name, "portNumber": port, - "databaseUserId": db_user, "description": description, - "databasePassword": db_pwd, - }, + "secretsStorePathName": secrets_store_path_name, + "secretsCollectionName": secrets_collection_name, + "resourceName": resource_name or postgres_server, + "versionIdentifier": version, + }), } body_s = body_slimmer(body) return await self._async_create_elem_from_template(body_s) @@ -1221,9 +1268,13 @@ def create_postgres_server_element_from_template( postgres_server: str, host_name: str, port: str, - db_user: str, - db_pwd: str, + db_user: Optional[str] = None, + db_pwd: Optional[str] = None, description: Optional[str] = None, + secrets_store_path_name: Optional[str] = None, + secrets_collection_name: Optional[str] = None, + version: Optional[str] = None, + resource_name: Optional[str] = None, ) -> str: """Create a Postgres server element from a template. @@ -1248,6 +1299,16 @@ def create_postgres_server_element_from_template( db_pwd: str User password to connect to the database + secrets_store_path_name : str, optional + Path name of the secrets store file holding the connection's credentials (secretsStorePathName). + secrets_collection_name : str, optional + Name of the secrets collection in that store (secretsCollectionName). + version : str, optional + Version identifier for the element. + resource_name : str, optional + Technical identifier for the element (the template's resourceName). Defaults to the element's name. + db_user, db_pwd : str, optional + Deprecated and ignored (ISSUE-160): the templates now take credentials from a secrets store. Returns ------- @@ -1257,7 +1318,15 @@ def create_postgres_server_element_from_template( loop = asyncio.get_event_loop() response = loop.run_until_complete( self._async_create_postgres_server_element_from_template( - postgres_server, host_name, port, db_user, db_pwd, description + postgres_server, host_name, port, db_user, db_pwd, description, + + secrets_store_path_name=secrets_store_path_name, + + secrets_collection_name=secrets_collection_name, + + version=version, + + resource_name=resource_name, ) ) return response @@ -1268,9 +1337,12 @@ async def _async_create_postgres_database_element_from_template( server_name: str, host_identifier: str, port: str, - db_user: str, - db_pwd: str, + db_user: Optional[str] = None, + db_pwd: Optional[str] = None, description: Optional[str] = None, + secrets_store_path_name: Optional[str] = None, + secrets_collection_name: Optional[str] = None, + version: Optional[str] = None, ) -> str: """Create a Postgres database element from a template. Async version. @@ -1290,29 +1362,44 @@ async def _async_create_postgres_database_element_from_template( User password to connect to the database description: str, opt A description of the element. + secrets_store_path_name : str, optional + Path name of the secrets store file holding the connection's credentials (secretsStorePathName). + secrets_collection_name : str, optional + Name of the secrets collection in that store (secretsCollectionName). + version : str, optional + Version identifier for the element. + db_user, db_pwd : str, optional + Deprecated and ignored (ISSUE-160): the templates now take credentials from a secrets store. Returns ------- str The GUID of the Postgres database element. """ + if db_user is not None or db_pwd is not None: + # ISSUE-160: Egeria's Postgres templates take credentials from a secrets store + # (secretsStorePathName / secretsCollectionName); databaseUserId / databasePassword + # are no longer template placeholders, so these values are not sent. + warnings.warn("db_user and db_pwd are deprecated and ignored; pass secrets_store_path_name " + "and secrets_collection_name instead (ISSUE-160)", DeprecationWarning, stacklevel=2) template_guid = await self._async_get_template_guid_for_technology_type("PostgreSQL Relational Database") body = { "class": "TemplateRequestBody", "templateGUID": template_guid, "isOwnAnchor": True, - "placeholderPropertyValues": { + "placeholderPropertyValues": self._placeholder_values({ "databaseName": postgres_database, "serverName": server_name, "hostIdentifier": host_identifier, "portNumber": port, - "databaseUserId": db_user, "description": description, + "secretsStorePathName": secrets_store_path_name, + "secretsCollectionName": secrets_collection_name, + "versionIdentifier": version, # ISSUE-160: the database template's description placeholder is databaseDescription. "databaseDescription": description, - "databasePassword": db_pwd, - }, + }), } body_s = body_slimmer(body) response = await self._async_create_elem_from_template(body_s) @@ -1324,9 +1411,12 @@ def create_postgres_database_element_from_template( server_name: str, host_identifier: str, port: str, - db_user: str, - db_pwd: str, + db_user: Optional[str] = None, + db_pwd: Optional[str] = None, description: Optional[str] = None, + secrets_store_path_name: Optional[str] = None, + secrets_collection_name: Optional[str] = None, + version: Optional[str] = None, ) -> str: """Create a Postgres database element from a template. Async version. @@ -1346,6 +1436,14 @@ def create_postgres_database_element_from_template( User password to connect to the database description: str, opt A description of the element. + secrets_store_path_name : str, optional + Path name of the secrets store file holding the connection's credentials (secretsStorePathName). + secrets_collection_name : str, optional + Name of the secrets collection in that store (secretsCollectionName). + version : str, optional + Version identifier for the element. + db_user, db_pwd : str, optional + Deprecated and ignored (ISSUE-160): the templates now take credentials from a secrets store. Returns ------- @@ -1362,6 +1460,12 @@ def create_postgres_database_element_from_template( db_user, db_pwd, description, + + secrets_store_path_name=secrets_store_path_name, + + secrets_collection_name=secrets_collection_name, + + version=version, ) ) return response @@ -1417,7 +1521,7 @@ async def _async_create_folder_element_from_template( "templateGUID": template_guid, "isOwnAnchor": is_own_anchor, "initialClassifications": initial_classifications, - "placeholderPropertyValues": { + "placeholderPropertyValues": self._placeholder_values({ "directoryPathName": path_name, # ISSUE-160: the template's endpoint address is ~{directoryAddress}~. "directoryAddress": path_name, @@ -1426,7 +1530,7 @@ async def _async_create_folder_element_from_template( "versionIdentifier": version, "fileSystemName": file_system, "description": description, - }, + }), } body_s = body_slimmer(body) response = await self._async_create_elem_from_template(body_s) @@ -1490,6 +1594,7 @@ async def _async_create_uc_server_element_from_template( port: str, description: Optional[str] = None, version: Optional[str] = None, + resource_name: Optional[str] = None, ) -> str: """Create a Unity Catalog Server element from a template. Async version. @@ -1509,6 +1614,8 @@ async def _async_create_uc_server_element_from_template( version: str, opt version of the element - typically of the form x.y.z + resource_name : str, optional + Technical identifier for the element (the template's resourceName). Defaults to the element's name. @@ -1523,13 +1630,14 @@ async def _async_create_uc_server_element_from_template( "class": "TemplateRequestBody", "templateGUID": template_guid, "isOwnAnchor": True, - "placeholderPropertyValues": { + "placeholderPropertyValues": self._placeholder_values({ "serverName": server_name, "hostURL": host_url, "versionIdentifier": version, "portNumber": port, "description": description, - }, + "resourceName": resource_name or server_name, + }), } body_s = body_slimmer(body) response = await self._async_create_elem_from_template(body_s) @@ -1542,6 +1650,7 @@ def create_uc_server_element_from_template( port: str, description: Optional[str] = None, version: Optional[str] = None, + resource_name: Optional[str] = None, ) -> str: """Create a Unity Catalog Server element from a template. Async version. @@ -1561,6 +1670,8 @@ def create_uc_server_element_from_template( version: str, opt version of the element - typically of the form x.y.z + resource_name : str, optional + Technical identifier for the element (the template's resourceName). Defaults to the element's name. @@ -1572,7 +1683,9 @@ def create_uc_server_element_from_template( loop = asyncio.get_event_loop() response = loop.run_until_complete( self._async_create_uc_server_element_from_template( - server_name, host_url, port, description, version + server_name, host_url, port, description, version, + + resource_name=resource_name, ) ) return response @@ -1583,6 +1696,7 @@ async def _async_create_uc_catalog_element_from_template( network_address: str, description: Optional[str] = None, version: Optional[str] = None, + uc_server_qualified_name: Optional[str] = None, ) -> str: """Create a Unity Catalog Catalog element from a template. Async version. @@ -1599,6 +1713,8 @@ async def _async_create_uc_catalog_element_from_template( version: str, opt version of the element - typically of the form x.y.z + uc_server_qualified_name : str, optional + Qualified name of the Unity Catalog server element (ucServerQualifiedName). @@ -1613,12 +1729,13 @@ async def _async_create_uc_catalog_element_from_template( "class": "TemplateRequestBody", "templateGUID": template_guid, "isOwnAnchor": True, - "placeholderPropertyValues": { + "placeholderPropertyValues": self._placeholder_values({ "ucCatalogName": uc_catalog, "serverNetworkAddress": network_address, "versionIdentifier": version, "description": description, - }, + "ucServerQualifiedName": uc_server_qualified_name, + }), } body_s = body_slimmer(body) response = await self._async_create_elem_from_template(body_s) @@ -1630,6 +1747,7 @@ def create_uc_catalog_element_from_template( network_address: str, description: Optional[str] = None, version: Optional[str] = None, + uc_server_qualified_name: Optional[str] = None, ) -> str: """Create a Unity Catalog Catalog element from a template. @@ -1646,6 +1764,8 @@ def create_uc_catalog_element_from_template( version: str, opt version of the element - typically of the form x.y.z + uc_server_qualified_name : str, optional + Qualified name of the Unity Catalog server element (ucServerQualifiedName). @@ -1657,7 +1777,9 @@ def create_uc_catalog_element_from_template( loop = asyncio.get_event_loop() response = loop.run_until_complete( self._async_create_uc_catalog_element_from_template( - uc_catalog, network_address, description, version + uc_catalog, network_address, description, version, + + uc_server_qualified_name=uc_server_qualified_name, ) ) return response @@ -1702,13 +1824,13 @@ async def _async_create_uc_schema_element_from_template( "class": "TemplateRequestBody", "templateGUID": template_guid, "isOwnAnchor": True, - "placeholderPropertyValues": { + "placeholderPropertyValues": self._placeholder_values({ "ucCatalogName": uc_catalog, "ucSchemaName": uc_schema, "serverNetworkAddress": network_address, "versionIdentifier": version, "description": description, - }, + }), } body_s = body_slimmer(body) response = await self._async_create_elem_from_template(body_s) @@ -1767,6 +1889,7 @@ async def _async_create_uc_table_element_from_template( network_address: str, description: Optional[str] = None, version: Optional[str] = None, + root_schema_type_qualified_name: Optional[str] = None, ) -> str: """Create a Unity Catalog table element from a template. Async version. @@ -1795,6 +1918,8 @@ async def _async_create_uc_table_element_from_template( version: str, opt version of the element - typically of the form x.y.z + root_schema_type_qualified_name : str, optional + Qualified name of the root schema type (rootSchemaTypeQualifiedName). @@ -1809,7 +1934,7 @@ async def _async_create_uc_table_element_from_template( "class": "TemplateRequestBody", "templateGUID": template_guid, "isOwnAnchor": True, - "placeholderPropertyValues": { + "placeholderPropertyValues": self._placeholder_values({ "ucCatalogName": uc_catalog, "ucSchemaName": uc_schema, "ucTableName": uc_table, @@ -1819,7 +1944,8 @@ async def _async_create_uc_table_element_from_template( "serverNetworkAddress": network_address, "versionIdentifier": version, "description": description, - }, + "rootSchemaTypeQualifiedName": root_schema_type_qualified_name, + }), } body_s = body_slimmer(body) response = await self._async_create_elem_from_template(body_s) @@ -1836,6 +1962,7 @@ def create_uc_table_element_from_template( network_address: str, description: Optional[str] = None, version: Optional[str] = None, + root_schema_type_qualified_name: Optional[str] = None, ) -> str: """Create a Unity Catalog table element from a template. @@ -1864,6 +1991,8 @@ def create_uc_table_element_from_template( version: str, opt version of the element - typically of the form x.y.z + root_schema_type_qualified_name : str, optional + Qualified name of the root schema type (rootSchemaTypeQualifiedName). @@ -1884,6 +2013,8 @@ def create_uc_table_element_from_template( network_address, description, version, + + root_schema_type_qualified_name=root_schema_type_qualified_name, ) ) return response @@ -1896,6 +2027,7 @@ async def _async_create_uc_function_element_from_template( network_address: str, description: Optional[str] = None, version: Optional[str] = None, + root_schema_type_qualified_name: Optional[str] = None, ) -> str: """Create a Unity Catalog function element from a template. Async version. @@ -1918,6 +2050,8 @@ async def _async_create_uc_function_element_from_template( version: str, opt version of the element - typically of the form x.y.z + root_schema_type_qualified_name : str, optional + Qualified name of the root schema type (rootSchemaTypeQualifiedName). @@ -1932,14 +2066,15 @@ async def _async_create_uc_function_element_from_template( "class": "TemplateRequestBody", "templateGUID": template_guid, "isOwnAnchor": True, - "placeholderPropertyValues": { + "placeholderPropertyValues": self._placeholder_values({ "ucCatalogName": uc_catalog, "ucSchemaName": uc_schema, "ucFunctionName": uc_function, "serverNetworkAddress": network_address, "versionIdentifier": version, "description": description, - }, + "rootSchemaTypeQualifiedName": root_schema_type_qualified_name, + }), } body_s = body_slimmer(body) response = await self._async_create_elem_from_template(body_s) @@ -1953,6 +2088,7 @@ def create_uc_function_element_from_template( network_address: str, description: Optional[str] = None, version: Optional[str] = None, + root_schema_type_qualified_name: Optional[str] = None, ) -> str: """Create a Unity Catalog function element from a template. @@ -1975,6 +2111,8 @@ def create_uc_function_element_from_template( version: str, opt version of the element - typically of the form x.y.z + root_schema_type_qualified_name : str, optional + Qualified name of the root schema type (rootSchemaTypeQualifiedName). @@ -1992,6 +2130,8 @@ def create_uc_function_element_from_template( network_address, description, version, + + root_schema_type_qualified_name=root_schema_type_qualified_name, ) ) return response @@ -2006,6 +2146,8 @@ async def _async_create_uc_volume_element_from_template( network_address: str, description: Optional[str] = None, version: Optional[str] = None, + directory_path_name: Optional[str] = None, + local_storage_location: Optional[str] = None, ) -> str: """Create a Unity Catalog volume element from a template. Async version. @@ -2033,6 +2175,10 @@ async def _async_create_uc_volume_element_from_template( version: str, opt version of the element - typically of the form x.y.z + directory_path_name : str, optional + Path name of the volume's directory (directoryPathName). + local_storage_location : str, optional + Local storage location of the volume (localStorageLocation). @@ -2047,7 +2193,7 @@ async def _async_create_uc_volume_element_from_template( "class": "TemplateRequestBody", "templateGUID": template_guid, "isOwnAnchor": True, - "placeholderPropertyValues": { + "placeholderPropertyValues": self._placeholder_values({ "ucCatalogName": uc_catalog, "ucSchemaName": uc_schema, "ucVolumeName": uc_volume, @@ -2056,7 +2202,9 @@ async def _async_create_uc_volume_element_from_template( "serverNetworkAddress": network_address, "versionIdentifier": version, "description": description, - }, + "directoryPathName": directory_path_name, + "localStorageLocation": local_storage_location, + }), } body_s = body_slimmer(body) response = await self._async_create_elem_from_template(body_s) @@ -2072,6 +2220,8 @@ def create_uc_volume_element_from_template( network_address: str, description: Optional[str] = None, version: Optional[str] = None, + directory_path_name: Optional[str] = None, + local_storage_location: Optional[str] = None, ) -> str: """Create a Unity Catalog volume element from a template. Async version. @@ -2100,6 +2250,10 @@ def create_uc_volume_element_from_template( version: str, opt version of the element - typically of the form x.y.z + directory_path_name : str, optional + Path name of the volume's directory (directoryPathName). + local_storage_location : str, optional + Local storage location of the volume (localStorageLocation). @@ -2119,6 +2273,10 @@ def create_uc_volume_element_from_template( network_address, description, version, + + directory_path_name=directory_path_name, + + local_storage_location=local_storage_location, ) ) return response diff --git a/tests/micro-tests/test_issue_160_remaining_placeholders.py b/tests/micro-tests/test_issue_160_remaining_placeholders.py new file mode 100644 index 00000000..3a7fdc02 --- /dev/null +++ b/tests/micro-tests/test_issue_160_remaining_placeholders.py @@ -0,0 +1,158 @@ +""" +ISSUE-160 follow-up: the remaining template-placeholder gaps. +- Postgres server/database templates take credentials from a secrets store + (secretsStorePathName / secretsCollectionName); databaseUserId and + databasePassword are no longer placeholders, so db_user/db_pwd are deprecated. +- An unset optional argument must not leave a literal ~{name}~: it is sent as "". +- Unity Catalog and Kafka builders send the placeholders their templates declare. +No live server: the template lookup and the create call are captured. +""" +import warnings + +import pytest + +from pyegeria.omvs.automated_curation import AutomatedCuration + + +def _client(captured: dict, monkeypatch) -> AutomatedCuration: + monkeypatch.setattr(AutomatedCuration, "check_connection", lambda self: "test") + client = AutomatedCuration("view-server", "https://localhost:1", "user", "pwd") + + async def fake_template_guid(type_name, **kwargs): + captured["technology_type"] = type_name + return "template-guid" + + async def fake_create(body=None): + captured["body"] = body + return "new-guid" + + client._async_get_template_guid_for_technology_type = fake_template_guid + client._async_create_elem_from_template = fake_create + return client + + +def _values(captured): + return captured["body"]["placeholderPropertyValues"] + + +@pytest.mark.asyncio +async def test_postgres_server_sends_secrets_and_resource_name(monkeypatch): + captured = {} + client = _client(captured, monkeypatch) + await client._async_create_postgres_server_element_from_template( + "pg1", "db.example", "5432", secrets_store_path_name="secrets/pg.omsecrets", + secrets_collection_name="pg1-creds", version="16") + + values = _values(captured) + assert values["secretsStorePathName"] == "secrets/pg.omsecrets" + assert values["secretsCollectionName"] == "pg1-creds" + assert values["resourceName"] == "pg1" + assert values["versionIdentifier"] == "16" + assert "databaseUserId" not in values and "databasePassword" not in values + + +@pytest.mark.asyncio +async def test_postgres_db_user_and_password_are_deprecated_and_not_sent(monkeypatch): + captured = {} + client = _client(captured, monkeypatch) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + await client._async_create_postgres_database_element_from_template( + "sales", "pg1", "db.example", "5432", "user", "secret", description="Sales") + + values = _values(captured) + assert any(issubclass(w.category, DeprecationWarning) for w in caught) + assert "databaseUserId" not in values and "databasePassword" not in values + assert "secret" not in values.values() + assert values["databaseDescription"] == "Sales" + assert values["secretsStorePathName"] == "" and values["secretsCollectionName"] == "" + + +@pytest.mark.asyncio +async def test_unset_optional_placeholders_are_sent_as_empty_strings(monkeypatch): + captured = {} + client = _client(captured, monkeypatch) + await client._async_create_folder_element_from_template("/data/landing", "landing", None) + + values = _values(captured) + for key in ("description", "versionIdentifier", "fileSystemName"): + assert values[key] == "", key + + +@pytest.mark.asyncio +async def test_csv_unset_file_system_and_description_are_empty_strings(monkeypatch): + captured = {} + client = _client(captured, monkeypatch) + await client._async_create_csv_data_file_element_from_template("a.csv", "CSV Data File", "/data/a.csv", None) + + values = _values(captured) + assert values["fileSystemName"] == "" + assert values["description"] == "" + assert values["versionIdentifier"] == "" + + +@pytest.mark.asyncio +async def test_kafka_sends_resource_name_and_version(monkeypatch): + captured = {} + client = _client(captured, monkeypatch) + await client._async_create_kafka_server_element_from_template("kafka1", "kafka.example", "9092") + + values = _values(captured) + assert values["resourceName"] == "kafka1" + assert values["versionIdentifier"] == "" + + +@pytest.mark.asyncio +async def test_uc_server_sends_resource_name(monkeypatch): + captured = {} + client = _client(captured, monkeypatch) + await client._async_create_uc_server_element_from_template("uc1", "http://uc.example", "8080", + resource_name="uc-main") + assert _values(captured)["resourceName"] == "uc-main" + + +@pytest.mark.asyncio +async def test_uc_catalog_sends_server_qualified_name(monkeypatch): + captured = {} + client = _client(captured, monkeypatch) + await client._async_create_uc_catalog_element_from_template( + "cat1", "http://uc.example:8080", uc_server_qualified_name="SoftwareServer::uc1") + assert _values(captured)["ucServerQualifiedName"] == "SoftwareServer::uc1" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("method, args", [ + ("_async_create_uc_table_element_from_template", + ("cat1", "sch1", "t1", "MANAGED", "s3://b/t1", "DELTA", "http://uc.example:8080")), + ("_async_create_uc_function_element_from_template", + ("cat1", "sch1", "f1", "http://uc.example:8080")), +]) +async def test_uc_table_and_function_send_root_schema_type(method, args, monkeypatch): + captured = {} + client = _client(captured, monkeypatch) + await getattr(client, method)(*args, root_schema_type_qualified_name="RootSchemaType::x") + assert _values(captured)["rootSchemaTypeQualifiedName"] == "RootSchemaType::x" + + +@pytest.mark.asyncio +async def test_uc_volume_sends_directory_and_local_storage(monkeypatch): + captured = {} + client = _client(captured, monkeypatch) + await client._async_create_uc_volume_element_from_template( + "cat1", "sch1", "v1", "MANAGED", "s3://b/v1", "http://uc.example:8080", + directory_path_name="/volumes/v1", local_storage_location="/mnt/v1") + + values = _values(captured) + assert values["directoryPathName"] == "/volumes/v1" + assert values["localStorageLocation"] == "/mnt/v1" + + +def test_sync_wrappers_pass_new_arguments_through(monkeypatch): + captured = {} + client = _client(captured, monkeypatch) + client.create_postgres_server_element_from_template( + "pg1", "db.example", "5432", secrets_store_path_name="s.omsecrets", secrets_collection_name="c") + assert _values(captured)["secretsStorePathName"] == "s.omsecrets" + client.create_uc_volume_element_from_template( + "cat1", "sch1", "v1", "MANAGED", "s3://b/v1", "http://uc.example:8080", local_storage_location="/mnt/v1") + assert _values(captured)["localStorageLocation"] == "/mnt/v1"