Add scripts to automate the generation of databases based on downloads from source.
The scripts would automate the process of downloading the source data, decompressing it (where relevant), importing it and saving the database.
This would ideally come in the form of terraform scripts which boot up one cloud instance per data source, allowing everything to be processed in parallel.
Additionally, we need to consider:
- Where the build artefacts (the database and the logs) are stored
- If we want to automatically merge all the sources into an 'everything.db' or leave that to the end user.
Add scripts to automate the generation of databases based on downloads from source.
The scripts would automate the process of downloading the source data, decompressing it (where relevant), importing it and saving the database.
This would ideally come in the form of terraform scripts which boot up one cloud instance per data source, allowing everything to be processed in parallel.
Additionally, we need to consider: