Terminology around datasets, files, packages, etc.

[alexrichey]

When @damonmcc and I were working on metadata for our datasets (in the context of the Socrata automations) we arrived at the following:

dataset_package: the full collection of files for a product release, basically the dataset (see below) + attachments
dataset: an output of the table, with a specific file format (ignore for now that an fgdb might have multiple tables/layers)
attachments: pretty self-explanatory

Here's an example:

dataset_package:
  datasets:
  - name: primary_shapefile
    type: shapefile
    filename: shapefile.zip
  attachments:
  - nycmih_metadata.pdf

But I don't think this is quite right - I don't think the shapefile output of Pluto is a really a "dataset." What is it though?

Also happy to get input on other names. This also makes me think we should start a glossary in our repo. cc @fvankrieken @damonmcc @sf-dcp

[fvankrieken]

I'm not quite sure what else you'd call it. For the distinction, let's use a simplified cpdb as an example

dataset_package:
  datasets:
  - name: cpdb_projects
    type: csv
  - name: cpdb_commitments
    type: csv
  - name: cpdb_projects_mapped
    type: shapefile

For these, cpdb_projects and cpdb_commitments are definitively different datasets, so maybe we'd be okay with calling this segment datasets. However, the two projects outputs represent the same underlying data just in a different format, so maybe we need a different label?

OR is it that you feel that either way, "dataset" is potentially an ill-fitting choice of word

[alexrichey]

No, that gets at it. Suppose we've got a version of Pluto built, and we export a csv and a shapefile - those aren't really different datasets.

But cpdb_projects and cpdb_commitments are. So maybe cpdb is the product, and product can contain multiple datasets?

[alexrichey]

Maybe something like dataset_artifact? dataset_output? dataset_artifact?

Had thought dataset_file, but I think this could potentially include the table in postgres as well. but I wonder if this concept we're shooting for also includes the table in postgres.

[damonmcc]

I was gonna say dataset_files since we're talking about single files with a single format. artifact and output feel too general for this last-mile stuff that we probably want to be more on the plain-language side of things

of course attachments are files too, but those are listed in a separate section and are conceptually a subset of all files

so maybe

dataset_package:
  dataset_files:
  - name: cpdb_projects
    type: csv
  - name: cpdb_commitments
    type: csv
  - name: cpdb_projects_mapped
    type: shapefile

  attachments:
  - cpdb_metadata.pdf

[alexrichey]

Yeah, I agree. What we're packaging is the file version of a dataset (as opposed to a database table, which we can't package) so I think dataset_file is good.

[sf-dcp]

I like dataset_files better because dataset can be one or multiple files, while dataset_file is always one item. In terms of figuring out the "primary" file, I would just add a free form description attribute

[damonmcc]

a tricky thing about tables too is that we very often have more than one single table at the end of a build that's used to generate files we distribute. the shorter the list of "final" tables the better though

[alexrichey]

can you give me a specific use-case? Want to make sure it fits the conception from 👆