As discussed on the reading club, we should run https://github.com/google/sentencepiece on our identifiers dataset and produce a nice compact ASDF in Modelforge to embed any identifier.
That model can be further integrated somewhere near our ID splitter.
As discussed on the reading club, we should run https://github.com/google/sentencepiece on our identifiers dataset and produce a nice compact ASDF in Modelforge to embed any identifier.
That model can be further integrated somewhere near our ID splitter.