I have a temporal workflow in my django application that processes the report. A…
- User
I have a temporal workflow in my django application that processes the report. A workflow has a bunch of activities and runs them in sequence. In the beginning of the workflow and at the end there is an activity that updates the status of the report to RUNNING and SUCCESS. Sometimes, one of the intermediate activities hang and they are getting killed because of timeout. In this case, even though workflow state is "Failed", my report is stuck in "RUNNING" state because it never gets to the last activity. Is there a way to define what happens when the workflow dies?
Here is an example workflow:
@workflow.defn class ExtractDataFromDynamicSections: @workflow.run async def run(self, payload: ExtractDataFromDynamicSectionsPayload) -> None: activity_options = { "start_to_close_timeout": timedelta(minutes=20), "retry_policy": default_retry_policy, } await workflow.execute_activity( reset_report, args=[payload.report_id], **activity_options, ) await workflow.execute_activity( update_agentic_extraction_status, args=[payload.report_id, Report.AgenticExtractionStatus.RUNNING], **activity_options, ) try: await workflow.execute_activity( extract_venture_from_dynamic_sections, args=[payload.report_id], **activity_options, ) await workflow.execute_activity( extract_people_from_dynamic_sections, args=[payload.report_id], **activity_options, ) await workflow.execute_activity( extract_report_details_from_dynamic_sections, args=[payload.report_id], **activity_options, ) await workflow.execute_activity( extract_skill_matrix_from_dynamic_sections, args=[payload.report_id], **activity_options, ) await workflow.execute_activity( update_agentic_extraction_status, args=[payload.report_id, Report.AgenticExtractionStatus.SUCCESS], **activity_options, ) except ApplicationError: await workflow.execute_activity( update_agentic_extraction_status, args=[payload.report_id, Report.AgenticExtractionStatus.FAILED], **activity_options, ) raise Assistant
Handling Workflow Failures in Temporal
In your current implementation, if an intermediate activity fails or times out, the workflow will fail but the report status remains "RUNNING" because the final status update activity never executes. Here are several ways to handle this:
Option 1: Use Workflow Interceptors
You can implement a workflow interceptor that will run cleanup logic when the workflow fails:
from temporalio import workflow class WorkflowInterceptor(workflow.WorkflowInterceptor): async def intercept_workflow(self, input, next): try: return await next(input) except Exception as e: # Get the report_id from the input if possible report_id = input.args[0].report_id if input.args else None if report_id: await workflow.execute_activity( update_agentic_extraction_status, args=[report_id, Report.AgenticExtractionStatus.FAILED], start_to_close_timeout=timedelta(minutes=1) ) raise # Then register it when starting the workflowOption 2: Use a Finally Block
Modify your workflow to include a
finallyblock that will execute regardless of success or failure:@workflow.defn class ExtractDataFromDynamicSections: @workflow.run async def run(self, payload: ExtractDataFromDynamicSectionsPayload) -> None: activity_options = { "start_to_close_timeout": timedelta(minutes=20), "retry_policy": default_retry_policy, } report_id = payload.report_id status = Report.AgenticExtractionStatus.FAILED # Default to failed try: await workflow.execute_activity( reset_report, args=[report_id], **activity_options, ) await workflow.execute_activity( update_agentic_extraction_status, args=[report_id, Report.AgenticExtractionStatus.RUNNING], **activity_options, ) await workflow.execute_activity( extract_venture_from_dynamic_sections, args=[report_id], **activity_options, ) # ... other activities ... status = Report.AgenticExtractionStatus.SUCCESS except Exception as e: raise finally: # This will run regardless of success or failure await workflow.execute_activity( update_agentic_extraction_status, args=[report_id, status], start_to_close_timeout=timedelta(minutes=1) )Option 3: Use a Saga Pattern
Implement a saga pattern where you track the state and can compensate if needed:
@workflow.defn class ExtractDataFromDynamicSections: @workflow.run async def run(self, payload: ExtractDataFromDynamicSectionsPayload) -> None: activity_options = { "start_to_close_timeout": timedelta(minutes=20), "retry_policy": default_retry_policy, } report_id = payload.report_id success = False try: await workflow.execute_activity( update_agentic_extraction_status, args=[report_id, Report.AgenticExtractionStatus.RUNNING], **activity_options, ) # Run all activities await workflow.execute_activity( extract_venture_from_dynamic_sections, args=[report_id], **activity_options, ) # ... other activities ... success = True finally: status = (Report.AgenticExtractionStatus.SUCCESS if success else Report.AgenticExtractionStatus.FAILED) await workflow.execute_activity( update_agentic_extraction_status, args=[report_id, status], start_to_close_timeout=timedelta(minutes=1) )Option 4: Use Workflow Signals
You could also implement a signal handler that updates the status when the workflow fails:
@workflow.defn class ExtractDataFromDynamicSections: @workflow.signal async def update_status_on_failure(self, status: str): await workflow.execute_activity( update_agentic_extraction_status, args=[self.report_id, status], start_to_close_timeout=timedelta(minutes=1) ) @workflow.run async def run(self, payload: ExtractDataFromDynamicSectionsPayload) -> None: self.report_id = payload.report_id try: # ... your workflow code ... except Exception: await self.update_status_on_failure(Report.AgenticExtractionStatus.FAILED) raiseThe
finallyblock approach (Option 2) is probably the simplest and most straightforward solution for your case. It ensures the status is always updated regardless of how the workflow exits.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.