RFC: Workflow Performance Profiler - full design & implementation walkthrough

[PG1204]

Following up on @Yicong-Huang's suggestion on PR #5098. This post documents the entire profiler implementation from the hackathon submission, at the level of detail needed to decide what to land and in what order.

What it does

A feedback layer over Texera's existing per-operator runtime stats. While a workflow runs:

• Canvas heatmap colors operators cold -> hot under one of three views (Runtime / Throughput / I/O imbalance).
• Rule-based hints in the property panel ("filter passes 2% of rows - push it upstream").
• Compare to past run - pick any completed execution; canvas recolors green/red by delta.
• Downloadable report (Markdown + JSON).
• (Optional) Ghost suggestions on canvas for two structural rewrites.

Design principle: read-only consumer of data we already produce. No new event types, no engine changes, no new HTTP endpoints in the critical path. Removing the profiler touches zero backend code paths.

Architecture

OperatorStatisticsUpdateEvent (existing)
│
▼
WorkflowStatusService (existing)
│
▼
ProfilerService ── reactive state (enabled, view, threshold, scores, baseline)
│
├── profiler-config (round-trip via WorkflowContent.profilerConfig)
├── profiler-hints (pure 6-rule engine)
├── profiler-delta (baseline diffing)
├── profiler-history (server-fetched past runs → BaselineReport)
├── profiler-report (MD + JSON export)
└── profiler-suggestions (optional ghost overlays)
│
▼
joint-ui (canvas color) · menu (controls) · property-panel (metrics + hints)

Components

Module	Purpose
ProfilerService	Throttled (500ms) subscriber to status stream; pure computeScores() per view; normalizes to [0,1]; resets on run restart
profiler-config	WorkflowContent.profilerConfig round-trip; defensive parser; equality guard against write-loop
profiler-hints	6 rules: IDLE_HEAVY, JOIN_HIGH_FANIN_LOW_FANOUT, LOW_PARALLELISM_HOT_OP, RUNTIME_OUTLIER, SCAN_FULL_TABLE_NO_FILTER, UPSTREAM_OVERPRODUCTION
profiler-delta	Per-op delta vs baseline; ±5% / <1ms counts as unchanged; surfaces matched / new / removed
profiler-history	Reuses existing /executions/{wid}/stats/{eid}; converts persisted rows → BaselineReport; memoized per (wid, eid)
profiler-report	MD + JSON; JSON is identical to the upload schema (no separate format)
profiler-suggestions (optional)	Structural rewrites (INSERT_FILTER, BUMP_WORKERS); per-workflow dismissed set in localStorage
ProfilerScoring.scala (optional)	Pure object mirroring TS scoring; zero call sites today; only useful if backend ever needs scoring

Test totals on the branch: ~250 Vitest specs (all green), tsc --noEmit clean, ScalaTest covers the scoring helper.

Things I'd like input on

1. Per-workflow config storage: one optional field on WorkflowContent. OK, or prefer a side table / user setting?
2. Backend scoring helper: include ProfilerScoring.scala now for parity, or drop until there's a call site?
3. Ghost suggestions vs agent: they cover overlapping use cases (the agent integration, deferred to a separate RFC, has similar Apply/Reject cards). Land both as complementary, or pick one?
4. Schema versioning: baseline JSON = exported report JSON. Add an explicit schemaVersion before merging, or trust the defensive parser?
5. Hint i18n: messages are English-only. Route through i18n now or defer?
6. Recompute scale: tested up to ~80 operators with no issue, no formal benchmark. Upper bound we care about?
7. Hint actionability for non-parallelizable operators: LOW_PARALLELISM_HOT_OP and its ghost (+ workers (N)) fire on Python/HF ops, which are non-parallelizable today (PythonOperatorDescriptor.parallelizable() = false). Suppress the hint, keep it as diagnostic-only with rephrased copy, or treat parallelizable-Python as out-of-scope follow-up?

Proposed merge order

7 independent PRs, each useful on its own:

Heatmap foundation: ProfilerService + config + canvas coloring + Execution-Settings controls + legend. (~1.2k LOC w/ tests)
Hints + property-panel metrics + threshold slider.
UX polish: displayName threading, hover tooltip, MD/JSON report download.
Per-workflow config round-trip.
Compare-across-runs + delta heatmap.
(optional) Ghost suggestions.
(optional, backend) ProfilerScoring.scala.
Agent integration -> separate RFC once 1–5 are in.

Happy to open PR 1 once there's directional agreement here. Specific yes/no on "Things I'd like input on" would unblock the most.

[Yicong-Huang]

I personally very much like this feature, I view this feature as three folds:

1. displays the power of our engine: very interactive, can quickly gather metadata during execution. super powerful
2. utilizes our workflow UI, to display/embed information on this UI.
3. agent suggestions to apply/fix/help user based on the profile results.
   So it is a very good unification of interactive engine + workflow UI + agent.

I think your draft architecture makes sense. very good start point! to move forward, we need smaller steps. we may not need all the features, though. so will do the cherry pick carefully.

A few comments:

1. I would not call it profile, at least not towards user. I think we want it to be low-tech user facing, and they may not understand profile or profiling. I still prefer to keep it as "workflow status". to the end user, it is "what's going on with my workflow?". Currently I think we have "state" and "statistics" two concepts inside status (please correct me if I'm wrong, or we might need to do some refactor). Can you extend a concept of "profile" inside the "WorkflowStatusService"? The WorkflowStatusService should become layer that contains the ground truth information.
2. it might be good to have different overlay layers, for instance you can introduce "heat map" layer. see other current workflow overlay layers, user can choose to toggle on/off any of those. The layer will just display the corresponding information from WorkflowStatusService.

for your suggest plan, I would not considered it in term of PRs, but in terms of issues. each issue is a todo task. and each issue could correspond to multiple PRs (feature, revert, fix, reapply, doc, etc.)

So high levelly I suggest we open an umbrella issue (don't name it profile/profiler though), and go with those three sub-issues first.

1. refactor existing service (e.g., WorkflowStatusService) to capture/store the information on the frontend which you need to display. this is purely information level, no UI change yet.
2. start to add one overlay layer. could be "heat map": which operator is the bottleneck, etc. this is UI visual part.
3. make sure any information displayed on the frontend should be able to be reconstructed from the backend. meaning that if user refreshes the page, he/she will see the same information.

we can come back to this discussion after you finish the first three.
WDYT?

[PG1204]

Sounds good, we can discuss and cherry pick the required features.

Reply to comments:

1. Yes, currently "WorkflowStatusService" has two concepts - "state" & "statistics". In my current impl, the "ProfilerService" is a separate service which subscribes to "WorkflowStatusService.getStatusUpdateStream()" as its input. Sure, I can extend the profile concept into "WorkflowStatusService", keeping it as the layer that contains the ground truth information.
   Worth noting that today operatorState and the aggregated counts share one flat OperatorStatistics record on a single stream, so part of sub-issue 1 will be splitting state and statistics into cleanly separated sub-concepts before adding the third. Also agreed on keeping user-facing language as "workflow status", I'll scrub "profile/profiler" from any UI surface.

2. Understood. I'll add the Heat Map as a single new sibling toggle in the Layers menu next to Grid / Regions / Workers / Status - reading from WorkflowStatusService and visualizing the new sub-concept. Future overlay layers can come in as separate work later.

Reply to PR plan:
This plan sounds good. I'll go ahead and work the three suggested changes.

Umbrella issue title: since "profile/profiler" is off the table, something like "Workflow status overlays and ground-truth refactor", or do you have any preferred phrasing?
Sub-issue 3 scope: should "same information after refresh" cover both live (mid-execution reconnect) and post-execution (loading a completed run), or only one of those for now?

Is it alright if I add you as the reviewer for all the three sub issues' PRs?

[Yicong-Huang]

sg. thanks.

• umbrella issue should more or less be a broad project name (instead of describing its steps), lets call it "workflow runtime performance visualization" for now. you can have sub-issues for exact steps. (hint: use commands introduced in Add /sub-issue and /parent-issue comment commands for linking sub-issues from either end #5147).
  It is my personal preference to stay away from profile/profiler as it is not that accurate and not user friendly. feel free to reject if you think "workflow runtime performance visualization" is not good though.
• for my proposed sub-issue 3, I think it should cover both cases.

regarding review: everyone is welcome to review. for a healthy community, everyone can jump in and share review/comments. feel free to tag me (using commands introduced in #4986) for any PRs if I did not see and comment on it.

[chenlica]

@PG1204 Is it possible to add an architecture diagram to illustrate the main design?

[chenlica]

My general comment is that visual diagrams can make your design easier to understand. Maybe you can add both diagrams then we can decide?

[PG1204]

Sure, I shall add both diagrams.

[PG1204]

My general comment is that visual diagrams can make your design easier to understand. Maybe you can add both diagrams then we can decide?

flowchart TD
    A[Execution Engine<br/><i>unchanged</i>] -->|OperatorStatisticsUpdateEvent<br/>over existing websocket| B[WorkflowStatusService<br/><i>existing</i>]
    B -->|getStatusUpdateStream<br/>throttled 500ms| C[ProfilerService<br/><b>new</b><br/>reactive state]

    C --> D[profiler-config<br/>WorkflowContent round-trip]
    C --> E[profiler-hints<br/>6-rule engine]
    C --> F[profiler-delta<br/>baseline diff]
    C --> G[profiler-history<br/>server-fetched runs]
    C --> H[profiler-report<br/>MD + JSON export]
    C --> I[profiler-suggestions<br/><i>optional</i> ghost overlays]

    D --> UI
    E --> UI
    F --> UI
    G --> UI
    H --> UI
    I --> UI

    UI[UI Integration<br/>joint-ui · menu · property-panel · workflow-editor]

    classDef existing fill:#e8e8e8,stroke:#888,color:#333
    classDef new fill:#d4edda,stroke:#28a745,color:#155724
    classDef optional fill:#fff3cd,stroke:#856404,color:#856404
    classDef ui fill:#cce5ff,stroke:#004085,color:#004085

    class A,B existing
    class C,D,E,F,G,H new
    class I optional
    class UI ui

Loading

Profiler data flow. Gray = existing Texera, green = new modules, yellow = optional, blue = UI integration points. All new code lives under frontend/src/app/workspace/service/profiler/ (plus the optional ProfilerScoring.scala helper). No new event types on the wire, no engine changes.

[PG1204]

My general comment is that visual diagrams can make your design easier to understand. Maybe you can add both diagrams then we can decide?

Profiler on a HuggingFace Sentiment Analysis workflow (100 rows). The heatmap surfaces HF inference as the bottleneck; RUNTIME_OUTLIER quantifies it at 379.5× the workflow median; LOW_PARALLELISM_HOT_OP explains why (single-worker Python op); and the orange ghost overlay (+ workers (4)) is the canvas-side actionable suggestion. Heatmap + diagnostics + suggestion, all live in one view. Note: for Python/HF ops the ghost's Apply is a no-op today, refer to open question no.7 in the description.

[PG1204]

@chenlica requesting your input on the visual diagrams.

[chenlica]

Thanks. @kunwp1 Can you chime in to provide input?

[kunwp1]

I really like the heatmap idea and also +1 to @Yicong-Huang's idea to make it toggleable. Worth landing the UI first on the codebase. My concern is the rule-based hints. Can you be more specific on how the six rules were chosen or which one wins when several rules fire on the same op? My suggestion would be to land the heatmap first and handle the hints as a separate issue, where we can reason why each rule exists, where its thresholds come from, and which hint to show when several fire at once.

[PG1204]

On the six rules. They're not a designed taxonomy but are the six bottleneck shapes I kept hitting on hackathon test workflows, each mapped to a recurring user question:

Rule	User question it answers
RUNTIME_OUTLIER	Which op dominates the runtime?
LOW_PARALLELISM_HOT_OP	Is the hot op slow because of compute or parallelism?
IDLE_HEAVY	Is this op the bottleneck, or starved upstream?
UPSTREAM_OVERPRODUCTION	Am I reading way more data than I consume?
JOIN_HIGH_FANIN_LOW_FANOUT	Is this join doing wasted work?
SCAN_FULL_TABLE_NO_FILTER	Could a predicate pushdown save this?

Thresholds (3× median, ≥10× upstream ratio, <5% join fan-out, ≥70% idle, ≥1M scan rows, score ≥ user-configured hot threshold + workers ≤1) were hand-tuned on those workflows for low false-positive rate over recall. No formal benchmark backs them, they're conservative defaults, open for revision.

Which one wins when several fire: None, as they're additive. The panel shows all hints that fire on the same op, in stable alphabetical-by-rule-id order. In the screenshot above, RUNTIME_OUTLIER and LOW_PARALLELISM_HOT_OP fire together by design: the first says "this is the bottleneck", the second "and here's likely why." They compound rather than conflict. Real overlaps exist (UPSTREAM_OVERPRODUCTION vs JOIN_HIGH_FANIN_LOW_FANOUT, IDLE_HEAVY vs LOW_PARALLELISM_HOT_OP) but dedup/priority isn't designed yet, but happy to make that part of the separate hints issue you suggested.

[PG1204]

My suggestion would be to land the heatmap first and handle the hints as a separate issue, where we can reason why each rule exists, where its thresholds come from, and which hint to show when several fire at once.

Agreed, this lines up with @Yicong-Huang's earlier proposal: umbrella issue "workflow runtime performance visualization" with sub-issues for (1) WorkflowStatusService refactor, (2) heatmap overlay, (3) persistence. Hints weren't in those three; they slot in naturally as future sub-issues which will be covering rule motivation, threshold justification, and conflict/dedup policy , landing only after the foundation is in.

I can open the umbrella + sub-issues 1 to 3 first upon the team's approval, hints sub-issue can follow later after thorough discussion. Thanks for the push.

[PG1204]

Hi team, any update on this? Shall I proceed with the impl of the heatmap on UI only? The hints can be discussed and worked on later.

[kunwp1]

You can proceed on the implementation.