Skip to content

feat: add Mosaic sandbox provider - #282

Merged
kisernl merged 2 commits into
masterfrom
feat/add-mosaic
Aug 7, 2026
Merged

feat: add Mosaic sandbox provider#282
kisernl merged 2 commits into
masterfrom
feat/add-mosaic

Conversation

@kisernl

@kisernl kisernl commented Aug 5, 2026 •
edited by devin-ai-integration Bot
Loading

Copy link
Copy Markdown
Collaborator

Summary

Wires the new @computesdk/mosaic sandbox provider into the benchmarks suite so it can be benchmarked alongside the existing providers once the MOSAIC_API_URL / MOSAIC_API_TOKEN secrets are available.

Changes

  • package.json / pnpm-lock.yaml: add @computesdk/mosaic@^0.1.1.
  • benchmarks/sandbox/providers.ts: register mosaic with baseUrl/apiKey from MOSAIC_API_URL / MOSAIC_API_TOKEN and sandboxOptions: { templateId: 'node-20' }.
  • benchmarks/sandbox/dax.bench.ts: add mosaic: { vcpus: 8, memoryMb: 16384 } to the standardized DAX resource map.
  • benchmarks/.env.example: document MOSAIC_API_URL and MOSAIC_API_TOKEN.
  • benchmarks/scripts/provider-vars.json: regenerate from providers.ts so the credential manifest stays in sync.
  • .github/workflows/sandbox-*.yml: add mosaic to every sandbox benchmark matrix and to the vault secret-loading regex.

Validation

  • pnpm typecheck passes.
  • pnpm --filter @benchsdk/client test and pnpm --filter @benchsdk/runner test pass.
  • npx tsx benchmarks/scripts/gen-provider-vars.ts regenerates the manifest with the new mosaic entry.

Link to Devin session: https://app.devin.ai/sessions/9e07e3983f4945e38c429dc0cd0ae66a
Requested by: @kisernl

Open in Devin Review

@open-cla

open-cla Bot commented Aug 5, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

devin-ai-integration[bot]
devin-ai-integration Bot reviewed Aug 5, 2026
View reviewed changes

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

View 1 additional finding in Devin Review.

Open in Devin Review

Comment thread benchmarks/sandbox/providers.ts
@devin-ai-integration

devin-ai-integration Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Addressed in 67152d6 — benchmarks/scripts/provider-vars.json has been regenerated to include the mosaic entry and the MOSAIC_API_URL / MOSAIC_API_TOKEN env vars are now wired into all sandbox-*.yml workflows and the .env.example file.

@kisernl
kisernl merged commit 6f4f3e3 into master Aug 7, 2026
4 checks passed
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Snapshot/Fork Benchmark Results

small dataset

# Provider Score Snapshot create Fork (snapshot) Fork (live) First read Status
1 Tensorlake 99.6 0.05s 0.35s 0.40s 0.21s 1/1
2 Tigris 99.4 0.19s 0.53s 0.47s 0.21s 1/1
3 Azure-blob 98.8 0.69s 0.52s 1.62s 0.01s 1/1
4 Cloudflare R2 81.5 9.29s 13.15s 18.51s 0.24s 1/1
5 Archil 0.0 0.00s 0.00s 0.00s 0.00s 0/1
6 AWS S3 0.0 0.00s 0.00s 0.00s 0.00s 0/1

View full run

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Browser Benchmark Results

# Provider Score Create Connect Navigate Release Total Status
1 Kernel 97.8 0.02s 0.06s 0.09s 0.05s 0.23s 10/10
2 Tilion 96.9 0.09s 0.04s 0.09s 0.03s 0.26s 10/10
3 Browseruse 95.9 0.16s 0.11s 0.08s 0.04s 0.41s 10/10
4 Hyperbrowser 93.5 0.25s 0.13s 0.13s 0.09s 0.62s 10/10
5 Browserbase 89.0 0.17s 0.31s 0.17s 0.14s 0.79s 10/10
6 Notte 87.2 0.28s 0.60s 0.27s 0.11s 1.44s 10/10
7 Steel 80.0 0.38s 0.73s 0.13s 0.24s 1.59s 10/10

View full run · SVG available as build artifact

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Storage Benchmark Results

1MB Files

# Provider Score Download Throughput Upload Status
1 Azure Blob Storage 96.5 0.03s 331.4 Mbps 0.07s 10/10
2 Archil 95.3 0.05s 155.6 Mbps 0.23s 10/10
3 Vercel Blob 95.1 0.07s 125.5 Mbps 0.23s 10/10
4 AWS S3 94.9 0.10s 83.6 Mbps 0.22s 10/10
5 Tensorlake 94.4 0.06s 143.3 Mbps 0.62s 10/10
6 Cloudflare R2 94.3 0.13s 63.1 Mbps 0.34s 10/10
7 Tigris 94.0 0.14s 59.5 Mbps 0.29s 10/10
8 Google Cloud Storage 0.0 0.00s 0.0 Mbps 0.00s 0/10

View full run · SVGs available as build artifacts

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Browser Throughput Benchmark Results

# Provider Score APS (med) Task (med) Task (p95) Screenshot Status
1 Kernel 80.8 5.94/s 1.68s 2.09s 257ms 3/3
2 Tilion 75.0 4.86/s 2.06s 3.88s 206ms 3/3
3 Browseruse 67.9 3.20/s 3.12s 3.22s 281ms 3/3
4 Browserbase 66.3 2.94/s 3.40s 3.71s 373ms 3/3
5 Hyperbrowser 65.7 2.89/s 3.46s 4.05s 463ms 3/3
6 Steel 49.8 1.08/s 9.23s 9.60s 785ms 3/3
7 Notte 46.5 0.93/s 10.73s 11.55s 1065ms 3/3

View full run · SVG available as build artifact

@github-actions

github-actions Bot commented Aug 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Sandbox Dax Benchmark Results

Provider Phases Total Prepare Clone Install Typecheck Status
archil 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
arker 7/7 59.20s 4.15s 2.37s 11.64s 35.38s 1/1 OK
beam 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
blaxel 6/7 43.96s 1.77s 2.56s 10.91s 0.00s 0/1 OK
cloud-run 2/7 1.25s 0.38s 0.00s 0.00s 0.00s 0/1 OK
codesandbox 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
createos 7/7 49.03s 1.99s 2.02s 11.11s 25.43s 1/1 OK
daytona 7/7 149.91s 5.11s 7.48s 39.83s 35.40s 1/1 OK
declaw 7/7 67.87s 5.47s 2.10s 12.44s 39.80s 1/1 OK
e2b 7/7 79.62s 9.35s 3.73s 16.31s 41.80s 1/1 OK
hopx -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
isorun 7/7 93.90s 0.97s 1.47s 10.97s 27.92s 1/1 OK
lightning 7/7 40.80s 4.72s 2.86s 12.57s 15.37s 1/1 OK
modal 7/7 92.55s 4.16s 3.93s 14.83s 61.47s 1/1 OK
mosaic -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
namespace 7/7 40.69s 4.69s 1.43s 8.26s 20.72s 1/1 OK
northflank -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
opencomputer 6/7 25.45s 4.35s 1.80s 16.92s 0.00s 0/1 OK
runloop 2/7 109.52s 0.00s 0.00s 28.83s 47.81s 1/1 OK
sail 6/7 60.61s 3.04s 3.85s 33.70s 0.00s 0/1 OK
sandbox0 7/7 136.58s 4.33s 7.53s 74.06s 35.38s 1/1 OK
superserve 7/7 101.92s 6.68s 5.92s 24.21s 60.61s 1/1 OK
tenki 7/7 74.38s 3.49s 4.06s 18.17s 42.66s 1/1 OK
tensorlake 7/7 56.68s 12.67s 1.73s 10.56s 27.53s 1/1 OK
upstash 7/7 51.41s 3.87s 2.08s 10.00s 25.09s 1/1 OK
vercel 7/7 73.23s 16.93s 1.90s 11.73s 35.92s 1/1 OK

View full run

@github-actions

github-actions Bot commented Aug 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Sequential

# Provider Score Median TTI P95 P99 Status
1 isorun 99.7 0.02s 0.04s 0.04s 10/10
2 declaw 99.0 0.01s 0.23s 0.23s 10/10
3 arker 98.9 0.08s 0.15s 0.15s 10/10
4 northflank 98.8 0.09s 0.16s 0.16s 10/10
5 createos 98.3 0.08s 0.32s 0.32s 10/10
6 archil 98.2 0.16s 0.21s 0.21s 10/10
7 lightning 98.2 0.16s 0.22s 0.22s 10/10
8 blaxel 97.1 0.23s 0.38s 0.38s 10/10
9 superserve 96.1 0.36s 0.44s 0.44s 10/10
10 beam 95.7 0.11s 0.92s 0.92s 10/10
11 upstash 95.3 0.45s 0.52s 0.52s 10/10
12 runloop 94.1 0.56s 0.65s 0.65s 10/10
13 cloud-run 93.8 0.55s 0.73s 0.73s 10/10
14 modal 93.5 0.57s 0.76s 0.76s 10/10
15 tensorlake 92.4 0.72s 0.82s 0.82s 10/10
16 sandbox0 91.5 0.81s 0.91s 0.91s 10/10
17 tenki 91.4 0.73s 1.05s 1.05s 10/10
18 vercel 91.0 0.56s 1.41s 1.41s 10/10
19 sail 90.5 0.60s 1.48s 1.48s 10/10
20 opencomputer 90.4 0.18s 2.13s 2.13s 10/10
21 e2b 87.8 1.02s 1.53s 1.53s 10/10
22 cloudflare 67.3 2.86s 3.88s 3.88s 10/10
23 daytona 48.1 1.08s 33.57s 33.57s 9/10
24 codesandbox 45.4 2.43s 26.26s 26.26s 10/10
25 hopx 0.0 0.00s 0.00s 0.00s 0/10
26 mosaic 0.0 0.00s 0.00s 0.00s 0/10

Staggered

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.02s 0.02s 0.02s 10/10
2 declaw 99.8 0.02s 0.03s 0.03s 10/10
3 arker 99.1 0.08s 0.10s 0.10s 10/10
4 createos 99.0 0.08s 0.12s 0.12s 10/10
5 beam 98.9 0.10s 0.14s 0.14s 10/10
6 northflank 98.8 0.10s 0.14s 0.14s 10/10
7 archil 98.4 0.15s 0.18s 0.18s 10/10
8 blaxel 97.6 0.23s 0.26s 0.26s 10/10
9 lightning 97.4 0.17s 0.41s 0.41s 10/10
10 opencomputer 95.2 0.09s 1.06s 1.06s 10/10
11 runloop 94.4 0.54s 0.59s 0.59s 10/10
12 cloud-run 94.2 0.56s 0.61s 0.61s 10/10
13 modal 93.9 0.57s 0.67s 0.67s 10/10
14 superserve 92.9 0.29s 1.35s 1.35s 10/10
15 tensorlake 92.6 0.69s 0.81s 0.81s 10/10
16 tenki 92.6 0.69s 0.82s 0.82s 10/10
17 vercel 92.1 0.60s 1.08s 1.08s 10/10
18 e2b 89.9 0.94s 1.11s 1.11s 10/10
19 sail 89.9 0.75s 1.41s 1.41s 10/10
20 sandbox0 88.1 1.07s 1.36s 1.36s 10/10
21 upstash 85.3 1.23s 1.82s 1.82s 10/10
22 codesandbox 74.9 2.39s 2.69s 2.69s 10/10
23 cloudflare 74.4 2.17s 3.14s 3.14s 10/10
24 daytona 54.6 0.89s 22.43s 22.43s 10/10
25 hopx 0.0 0.00s 0.00s 0.00s 0/10
26 mosaic 0.0 0.00s 0.00s 0.00s 0/10

Burst

# Provider Score Median TTI P95 P99 Status
1 declaw 99.5 0.04s 0.06s 0.06s 10/10
2 beam 98.6 0.13s 0.16s 0.16s 10/10
3 createos 98.5 0.15s 0.15s 0.15s 10/10
4 northflank 98.5 0.13s 0.18s 0.18s 10/10
5 lightning 98.2 0.18s 0.19s 0.19s 10/10
6 arker 98.1 0.19s 0.19s 0.19s 10/10
7 archil 98.1 0.18s 0.20s 0.20s 10/10
8 isorun 97.8 0.05s 0.49s 0.49s 10/10
9 blaxel 97.3 0.26s 0.29s 0.29s 10/10
10 superserve 96.2 0.38s 0.39s 0.39s 10/10
11 cloud-run 93.9 0.55s 0.70s 0.70s 10/10
12 runloop 93.8 0.58s 0.68s 0.68s 10/10
13 opencomputer 92.6 0.33s 1.35s 1.35s 10/10
14 vercel 92.3 0.50s 1.16s 1.16s 10/10
15 tensorlake 91.7 0.70s 1.04s 1.04s 10/10
16 modal 91.4 0.59s 1.27s 1.27s 10/10
17 upstash 91.3 0.61s 1.27s 1.27s 10/10
18 e2b 88.8 0.98s 1.33s 1.33s 10/10
19 tenki 86.2 0.90s 2.10s 2.10s 10/10
20 sail 85.5 1.45s 1.46s 1.46s 10/10
21 sandbox0 83.4 1.59s 1.76s 1.76s 10/10
22 cloudflare 74.9 2.13s 3.08s 3.08s 10/10
23 codesandbox 71.0 2.70s 3.19s 3.19s 10/10
24 daytona 54.9 0.84s 17.23s 17.23s 10/10
25 hopx 0.0 0.00s 0.00s 0.00s 0/10
26 mosaic 0.0 0.00s 0.00s 0.00s 0/10

View full run · SVGs available as build artifacts

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

AI Gateway Benchmark Results

# Gateway Score Cold E2E Warm TTFT Tok/sec Status
1 OpenRouter 91.1 612ms 619ms 87.8 20/20
2 Anthropic (direct) 90.8 658ms 506ms 89.8 20/20
3 Pydantic AI Gateway 90.5 730ms 675ms 89.4 20/20
4 Vercel AI Gateway 90.1 758ms 652ms 87.8 20/20
5 LLM Gateway 90.0 856ms 721ms 92.3 20/20
6 Concentrate AI 88.0 833ms 706ms 89.5 20/20
7 Cloudflare AI Gateway 87.6 616ms 566ms 90.9 20/20

View full run · SVG available as a build artifact

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@devin-ai-integration devin-ai-integration[bot] devin-ai-integration[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kisernl