Restructure input_pipeline and migrate to src/maxtext

[aireenmei]

Description

Restructure the input pipeline folder as follows:
under src/maxtext:
input_pipeline

• packing
• • prefill_packing.py
• • sequence_packing.py
• tokenizer.py
• multihost_dataloading.py
• distillation_data_processing.py (prev _distillation_data_processing.py)
• grain_data_processing.py (_grain_data_processing.py)
• grain_tokenizer.py (_grain_tokenizer.py)
• hf_data_processing.py (_hf_data_processing.py)
• input_pipeline_utils.py (_input_pipeline_utils.py)
• tfds_data_processing.py (_tfds_data_processing.py)
• tfds_data_processing_c4_mlperf.py (_tfds_data_processing_c4_mlperf.py)
• input_pipeline_interface.py
• synthetic_data_processing.py

Makes corresponding changes in imports

Tests

CI test

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

[codecov]

Codecov Report

❌ Patch coverage is 77.77778% with 12 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...rc/maxtext/input_pipeline/grain_data_processing.py	62.50%	6 Missing ⚠️
src/maxtext/input_pipeline/hf_data_processing.py	61.53%	5 Missing ⚠️
src/MaxText/rl/train_rl.py	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

🤖 Hi @hengtaoguo, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[hengtaoguo]

Browsed through all the files and looks like a solid restructure and relocation. Are there any edge cases which could not be covered by the CI? Thanks!

[github-actions]

📋 Review Summary

This pull request effectively restructures the input pipeline by migrating it to src/maxtext and organizing it into submodules. The changes are consistent and well-executed across the entire codebase, including examples, tests, and tools.

🔍 General Feedback

• The refactoring significantly improves the project structure and modularity of the input pipeline.
• All import paths have been updated correctly to reflect the new structure.
• Renaming files by removing the leading underscore is a good cleanup practice that enhances clarity.

[aireenmei]

Browsed through all the files and looks like a solid restructure and relocation. Are there any edge cases which could not be covered by the CI? Thanks!

As this is mainly restructure and migration, nothing new is added, the existing CI test should be sufficient to make sure no import errors.