Global streaming

[hanna-paasivirta]

Short Description

• Edits to streaming statuses in global_chat, workflow_chat and job_chat
• Sneaks in a parallelisation for step generation via the planner agent

Fixes #443

Requires a small tweak in Lightning to change the first status from "Generating code" to "Thinking" at minimum. Maybe a persistent event list in the future. OpenFn/lightning#4630

Implementation Details

Streaming status messages

All user-facing status messages shown during request processing. Status pools are defined once in streaming_util.py and composed by each service.

Frontend (Lightning)

When	Status
User sends message	"Thinking..."

Planner (via global agent)

Initial status (random from pool):

Context	Pool
Has existing workflow	"Reviewing the workflow...", "Reading your workflow...", "Looking over the current setup...", "Reviewing the current steps...", "Analyzing the workflow...", "Looking at what you have so far...", "Checking what's needed...", "Working out the details...", "Figuring out the approach...", "Planning things out..."
No existing workflow	"Mapping out the steps...", "Gathering requirements...", "Thinking about the design...", "Checking what's needed...", "Working out the details...", "Figuring out the approach...", "Planning things out..."

Tool execution statuses (fixed):

Tool	Context	Status
search_documentation	—	"Searching documentation for "{query}"..."
call_workflow_agent	Has existing workflow	"Editing workflow..."
call_workflow_agent	No existing workflow	"Building workflow outline..."
call_job_code_agent	Single job	"Writing code for "{Display Name}" step..."
call_job_code_agent	Multiple jobs	"Writing code for "{Name A}" and "{Name B}" steps..."
inspect_job_code	—	"Reading code for "{Display Name}" step..."

Job display names are resolved from the YAML name field first, then fall back to title-casing the key (e.g. fetch-patients → "Fetch Patients").

workflow_chat (direct via router)

Initial status (random from pool):

Context	Pool
Has existing workflow	"Reviewing the workflow...", "Reading your workflow...", "Looking over the current setup...", "Reviewing the current steps...", "Analyzing the workflow...", "Looking at what you have so far..."
No existing workflow	"Mapping out the steps...", "Gathering requirements...", "Thinking about the design...", "Designing the workflow...", "Planning the workflow...", "Working out the structure..."

Fixed status at end:

When	Status
YAML formatting phase	"Formatting workflow..."

job_chat (direct via router)

Initial status (random from pool):

Context	Pool
Has existing code	"Reviewing the code...", "Reading the current code...", "Looking over the job code...", "Checking the existing code...", "Reviewing what you have so far..."
No existing code	"Preparing to write code...", "Getting ready to write code...", "Setting things up...", "Getting started on the code...", "Preparing the job code..."

Conditional status (only shown when docs are actually needed):

When	Status
RAG decides docs are needed	"Searching documentation..."

Example narratives

Global agent, new workflow with multiple jobs:

Thinking... → Gathering requirements... → Searching documentation for "DHIS2 adaptor"... → Building workflow outline... → Writing code for "Fetch Patients" and "Send To FHIR" steps...

Global agent, editing existing workflow:

Thinking... → Reviewing the workflow... → Editing workflow... → Writing code for "Fetch Patients" step...

Direct workflow_chat, new workflow:

Thinking... → Designing the workflow... → Formatting workflow...

Direct workflow_chat, existing workflow:

Thinking... → Reading your workflow... → Formatting workflow...

Direct job_chat, new code, needs docs:

Thinking... → Preparing to write code... → Searching documentation...

Direct job_chat, editing code, simple request (no docs needed):

Thinking... → Reviewing the code...

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

[josephjclark]

Love the approach and structure of this PR

I'll bring some suggestions for those lovely tables

[josephjclark]

@hanna-paasivirta can you raise a small PR against lightning with your suggestion?

[josephjclark]

Lovely approach, well done

[josephjclark]

One possible optimisation here:

rather than picking a choice and then calling send_thinking, what if you passed send_thinking a list and it picked at random?

It would just make the streaming code slightly easier to write (which in turn will hopefully encourage more and better streaming updates)

[josephjclark]

It's a bit difficult to do this when generating many steps in parallel.

I think I'd prefer to just say "generating for step X" or just a flat "generating job code". The and is a bit cumbersome. And what would this look like if it's generating 4 steps at once? With long step names? I can't see it scaling

[hanna-paasivirta]

I tried just the first step, and it's confusing when the other steps appear in the workflow apparently without having been worked on. I also think it's quite satisfying seeing the different steps being named and worked on at once. I've replaced it to just be commas now. What do you think?

[josephjclark]

commas will scale better. Let's see how it goes

[josephjclark]

is the word step here a bit weird and confusing?

I think Writing code for "Fetch Patients" is pretty clear. If we really need the step` key word in there, I'd do it before (not after) the step name

[josephjclark]

Can we not add any more context to the job chat? Like "loading adaptor docs"? I guess it's mostly all done in one LLM call.

What about process_stream_event? Can we do more there? Or at least cycle different synonyms for "generating code"?

I don't want to spend ages on this, just probing it a bit to see if we can add a bit more texture and movement to the updates

[hanna-paasivirta]

I removed the adaptor docs because it's quite annoying seeing that in a turn when you ask an unrelated question. I've added one more status pool STATUS_WORKING that covers the prompt building steps. And expanded the STATUS_REVIEWING_CODE pool because users would see that more often than the STATUS_NEW_CODE.