Skip to content

Conversation

@AvinashYerra
Copy link

@AvinashYerra AvinashYerra commented Jan 3, 2026

This PR implements NDJSON (newline-delimited JSON) reader and writer support for Polars, following the same pattern as the existing JSON reader/writer implementation.
coming from #1197

Changes

  • Added PolarsNDJSONReader class in hamilton/plugins/polars_post_1_0_0_extensions.py

  • Added PolarsNDJSONWriter class in hamilton/plugins/polars_post_1_0_0_extensions.py

  • Supports writing NDJSON files using data.write_ndjson()

  • Registered both classes in register_data_loaders() function to make them available through the materialization system

  • Added the following tests:

    • test_polars_ndjson() in tests/plugins/test_polars_extensions.py for DataFrame support
    • test_polars_ndjson() in tests/plugins/test_polars_lazyframe_extensions.py for LazyFrame support
  • Added example in examples/polars/materialization/my_script.py

How I tested this

Ran the specific NDJSON tests:

  • pytest tests/plugins/test_polars_extensions.py::test_polars_ndjson -v
  • pytest tests/plugins/test_polars_lazyframe_extensions.py::test_polars_ndjson -v
    Both tests passed

Notes

Checklist

  • [Y] PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • [Y] Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.

@skrawcz
Copy link
Contributor

skrawcz commented Jan 5, 2026

@AvinashYerra would you mind rebasing against the latest hamilton main? I should have merged a fix for a bunch of unit tests.

@AvinashYerra AvinashYerra force-pushed the feat/add-ndjson-support branch from 5dc8d8a to a6ea097 Compare January 5, 2026 18:22
@AvinashYerra
Copy link
Author

Done @skrawcz, rebased to latest main. Thanks!

@skrawcz
Copy link
Contributor

skrawcz commented Jan 7, 2026

ugh sorry -- fix here #1437 -- if you wanted to apply it to this PR that would also work...

@skrawcz
Copy link
Contributor

skrawcz commented Jan 7, 2026

One more rebase please :)

@AvinashYerra
Copy link
Author

Hi @skrawcz , just for confirmation. you want me to rebase to the latest main right?

@AvinashYerra
Copy link
Author

Hi @skrawcz , just for confirmation. you want me to rebase to the latest main right?

it is saying everything is up to date
Screenshot 2026-01-07 230432

@skrawcz
Copy link
Contributor

skrawcz commented Jan 8, 2026

Strange. You can see that I merged #1437 ? So it's on main. Do you have that commit?

@AvinashYerra AvinashYerra force-pushed the feat/add-ndjson-support branch from a6ea097 to 4342d84 Compare January 8, 2026 15:27
@AvinashYerra
Copy link
Author

@skrawcz, It is done now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants