Queue callback error: Error [WorkflowAPIError]: workflow step XXX not found #438
Description
Hi.
I'm building workflow to process through many items. The goal is to fetch all items from datasource and then proceed through each to replicate it in same datasource with different language. As of now it's just a simple duplication of item with its content and meta. Each duplication process may be different depending on its workflow state, tough each is about 2-4 API calls to data storage in a single step.
Some about 2 hours (~1k items) after the workflow started, it's is throwing an error of not found step, like :
Queue callback error: Error [WorkflowAPIError]: workflow step step_01KB03ZG4MHPKTSYP45DWESM8A not found
at eq (.next/server/chunks/354.js:41:41)
at async tI (.next/server/chunks/354.js:41:6880)
at async (.next/server/chunks/354.js:45:768)
at async (.next/server/chunks/354.js:5:8119)
at async (.next/server/chunks/354.js:5:7582)
at async (.next/server/chunks/354.js:45:316)
at async default (.next/server/chunks/354.js:41:9464)
at async L.processMessage (.next/server/chunks/354.js:40:14995)
at async L.consume (.next/server/chunks/354.js:40:16202) {
cause: undefined,
status: 404,
code: undefined,
url: 'https://vercel-workflow.com/api/v1/runs/wrun_01KB03ZE75MGRXH35KDCVBY731/steps/step_01KB03ZG4MHPKTSYP45DWESM8A?remoteRefBehavior=resolve'
}
This is a repeatable step that is invoked with different parameters and was successful many times before.
Additionally the workflow is not breaking at this point, continuously and endlessly trying to fetch the not found step (at least for couple next hours before manually canceled) - so additional question would be how to programmatically stop it or retry for workflow process clarity and observability?
I'm testing that in Preview environment of Vercel project.
Are there any limitations that i'm unaware and crossing the line in terms of memory, static chunks, timeouts etc ? Observability section is not quite useful when having so many separated steps (infinite scroll for load more), while consolidating them into one with substeps leads to Vercel Function Timeout as it takes longer than max of 800s.