Feature: Non Public Object Storage - #499
Conversation
Neirpyc
changed the title
|
Fantastic! I just deployed spliit on our server and I was missing the possibility to use with our LocalAI instance and I thought, that it would be nice to use private S3 buckets as we use Garage as our S3 bucket provider which is a little bit complicated to configure to provide files over the web (different domains for the S3 endpoint and the web endpoint). Now there is your PR which solves those two problems. The PR works like a charm (together with LocalAI and Garage). Please include this in the next release, @scastiel. |
|
|
||
| const categories = await getCategories() | ||
|
|
||
| const resolvedUrl = await resolveDocumentToPresignedUrl(id) |
There was a problem hiding this comment.
OpenAI also supports sending images via base64 encoded urls: https://developers.openai.com/api/docs/guides/images-vision?format=base64-encoded
This would make this way easier as we never need a third-party (OpenAI) to download the image.
There was a problem hiding this comment.
It’s definitely possible. That said, it would make requests heavier due to the base64 encoding overhead, and it would shift more load onto the server since it would need to read and encode the file instead of offloading that to S3. Also, presigned URLs are already widely supported, so there isn’t really a compatibility concern.
Given that, I think presigned URLs are still a better fit for this use case for now. That said, if there’s a concrete need for base64, I could revisit this or make it a configurable option.
Adds a generic import architecture (format registry + detection) with Spliit-JSON as the first adapter, plus UI (upload dropzone, analysis/ progress/result views) and a batched tRPC import flow. This is what lets you export a group as JSON from spliit.app (or any other Spliit instance) and import it here as a new group - the two instances have entirely separate databases, so this was previously not possible at all (the 'Add group by URL' button only looks up groups in your own instance, it was never a cross-instance fetch). Conflict resolution notes: - expense-form.tsx: pr-472's side was stale noise (identical logic to our merge-base for this exact block, just missing spliit-app#367's EVENLY-for- reimbursements fix and spliit-app#499's imageId-based document ids) - kept our current version entirely rather than porting anything from pr-472 here. - create.procedure.ts: pr-472 intentionally changed groups.create's return shape from {groupId} to the full group object ({id, ...}) - its own new create-group.tsx UI code depends on this. Updated the stale test helpers (_app.test.ts, batch-api.ts) to match rather than reverting the shape. - Dropped the unconditionally-registered 'debug' import format (formats/debug-format.ts + its fixture) - a manual-testing aid with zero test coverage of its own that would have shipped live in production with no gate, matching content starting with a literal 'DEBUG_IMPORT'/'DEBUG_ERRORS' prefix. Low risk (it can only emit synthetic error messages, not fabricate real data) but provided nothing for us and no reason to ship it. - process-batch.procedure.ts imported 'nanoid' directly, which is ESM-only and broke Jest ('Must use import to load ES Module'). Replaced both call sites with this file's own randomId() (already the established replacement per the spliit-app#165 nanoid cleanup). - The same recurring cross-PR staleness pattern hit twice more: the new import-group.procedure.ts's own createGroup() call and spliit-json.ts's own expense-mapping function both predated spliit-app#530's now-required fixedExpenseDateGroups/location fields - added both. - spliit-json.test.ts's own test fixture caught this for real (not just gracefully skipped): 'should parse a valid export correctly' failed until spliit-json.ts set location on parsed expenses. Verified: check-types clean, all 206 tests pass, and (given the Turbopack/leaflet build failure found earlier wasn't caught by either of those) a full 'npm run build' also completes successfully.
3 participants
Context & Use Case
I host this Spliit instance for upcoming group holidays. To keep our data private, the entire instance sits behind an authenticated proxy, ensuring only our friends can access the tool.
The Problem: Currently, S3-uploaded documents (receipts/images) require the bucket to be public for them to render in the browser. This effectively bypasses the proxy's security, potentially exposing personal documents to the open web if a URL is leaked.
The Fix: This PR moves all S3 interactions to the server. The bucket can now be 100% private. Spliit fetches and streams the files server-side, meaning only users already authenticated through the proxy can see them.
Key Improvements
/api/documents/[id]).What Changed
Backend (The "Proxy" Logic)
POST /api/documents: Receives multipart uploads, stores them in S3, and saves metadata (dimensions, filename) to the DB.GET /api/documents/:id: The "secure gateway." It verifies the document exists and streams the data directly from S3 to the authenticated user.Client & UI
next-s3-uploaddependencies with a lightweight, native implementation.getImageData: A new client helper that calculates image dimensions using browser APIs to prevent layout shift without extra libraries.Infrastructure
src/lib/s3.ts: Centralized S3 client factory that only initializes if env vars are present.OPENAI_BASE_URLand model overrides, allowing for easy swapping between OpenAI and local/Scaleway providers.Security Note
By default, the SDK creates objects with private ACLs. This PR ensures that even if someone gets hold of the raw S3 link, the file remains inaccessible without the app's internal credentials.