ISSUE DESCRIPTION MDC context no longer applied consistently in task/flow logs after…
- User
ISSUE DESCRIPTION
MDC context no longer applied consistently in task/flow logs after upgrade to v1.0
Describe the issue
Hi,
We recently upgraded from kestra v0.19.5 to v1.0.3 and noticed a change in the logging behavior.
when using a logback config as such
<?xml version="1.0" encoding="UTF-8"?> <configuration debug="false"> <include resource="logback/base.xml" /> <include resource="logback/gcp.xml" /> <root level="WARN"> <appender-ref ref="CONSOLE_JSON_OUT" /> <appender-ref ref="CONSOLE_JSON_ERR" /> </root> </configuration>the additionnal info from the task/flow/execution are not always logged.
example from a Hello-world flow run:
{"timestampSeconds":1761224032,"timestampNanos":565000000,"severity":"INFO","thread":"jdbc-execution-executor_0","logger":"execution.gnu_170208","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [execution: 1FLhPmmXgvIghVXzW7hY90] Flow started","context":"default"} {"timestampSeconds":1761224032,"timestampNanos":775000000,"severity":"INFO","thread":"worker_0","logger":"task.gnu_170208.hello","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [task: hello] [execution: 1FLhPmmXgvIghVXzW7hY90] [taskrun: 7SKeqte94nC0xUfrgFXHpw] Type Log started","context":"default"} {"taskRunId":"7SKeqte94nC0xUfrgFXHpw","tenantId":"main","namespace":"company.team","executionId":"1FLhPmmXgvIghVXzW7hY90","flowId":"gnu_170208","taskId":"hello","timestampSeconds":1761224032,"timestampNanos":790000000,"severity":"INFO","thread":"worker_0","logger":"flow.gnu_170208.hello","message":"Hello World! 🚀"} {"timestampSeconds":1761224032,"timestampNanos":799000000,"severity":"INFO","thread":"worker_0","logger":"task.gnu_170208.hello","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [task: hello] [execution: 1FLhPmmXgvIghVXzW7hY90] [taskrun: 7SKeqte94nC0xUfrgFXHpw] Type Log with state SUCCESS completed in 00:00:00.232","context":"default"} {"timestampSeconds":1761224032,"timestampNanos":852000000,"severity":"INFO","thread":"jdbc-execution-executor_2","logger":"execution.gnu_170208","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [execution: 1FLhPmmXgvIghVXzW7hY90] Flow completed with state SUCCESS in 00:00:00.405","context":"default"}Only the hello world log itself gets the MDC context.
It was different in v0.19.5 we relied on this behavior to monitor flow executions.
I've debug kestra locally and i've been able to add that context by manually setting the contextMap in this method https://github.com/kestra-io/kestra/blob/eb22d3f6ee5b6b92f8a12f03948e7517a02e39cc/core/src/main/java/io/kestra/core/services/LogService.java#L45
such as :
public void logExecution(Execution execution, Level level, String message, Object... args) { Logger logger = logger(execution); MDC.setContextMap(Map.of("tenantId", execution.getTenantId(), "namespace", execution.getNamespace(), "flowId", execution.getFlowId(), "executionId", execution.getId())); logExecution(execution, logger, level, message, args); }Environment
- Kestra Version: develop, v1.0
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /kestra, current file: none) bash-$
Assistant
Let me start by exploring the repository structure to understand the codebase.
Assistant
Let me look at the LogService.java file mentioned in the issue:
Assistant
Let me try a different search:
Assistant
The LogService.java file no longer exists. Let me look at how logging is currently done:
Assistant
Let me look at the RunContextLogger.java file:
Assistant
Now let me look at the RunContextLoggerFactory to understand how it's instantiated:
Assistant
Let me look at the LogEntry class to understand the structure:
Assistant
Let me check the base.xml logback config:
Assistant
Let me look at the existing tests for RunContextLogger:
Assistant
Now let me look at the original LogService.java code referenced in the issue:
Assistant
Let me look more closely at how the ForwardAppender passes events to the forward logger:
Assistant
Let me see the existing
ForwardAppendermore carefully:Assistant
Let me look at how the LoggingEvent handles MDC:
Assistant
Let me look at how the MDC gets into the log output:
Assistant
Now let me look at theStackdriverJsonLayout to understand how MDC is added to the output JSON:
Assistant
Let me look at the BaseAppender's transform method more carefully:
Assistant
Let me check how MDC is captured in LoggingEvent constructor:
Assistant
Hmm interesting, looking at the bytecode, the LoggingEvent constructor with these parameters doesn't actually set the MDC property map! It only sets
fqnOfLoggerClass,loggerName,loggerContext,loggerContextVO,level,message,argumentArray,instant,sequenceNumber, andthrowableProxy. It doesn't set the MDC.So that's why the MDC is not present on the event!
Let me also check
getMDCPropertyMap():Assistant
Let me look at how the JsonLayout gets the MDC flag:
Assistant
Let me look at the older version of the LogService.java:
Assistant
Let me look at how the MDC is set in the task execution:
Assistant
Let me look at the older version of LogService.java:
Assistant
Let me check if MDC was being set in any places in the older codebase:
Assistant
Let me check the LogsTest file:
Assistant
Let me see where these
Logs.logExecutionandLogs.logTaskRunare called:Assistant
Let me run the existing tests to verify behavior:
Assistant
Let me write a test to understand how MDC is propagated:
Assistant
Let me trace through the code logic carefully. Let me look at how to handle the MDC at the LoggerEvent level:
Assistant
Let me check how the issue can be reproduced by writing a test:
Assistant
Let me look at the SLF4J fluent API source to understand:
Assistant
Let me look at the logback fluent logger:
Assistant
Let me reproduce the issue first:
Assistant
Let me check the JsonLayout defaults:
Assistant
The dependency isn't downloaded yet. Let me check if it's inthe gradle cache:
Assistant
Now let me write the fix. First, let me look at how to write tests for MDC:
Assistant
Now I have a clear understanding. Let me write the fix:
Assistant
Let me check
LogEntry.toMap()for the fields:Assistant
Let me write a reproduction test to confirm the issue:
Assistant
The test is up-to-date, meaning it didn't run.Let me force it to run again:
Assistant
Let me check the test report for the printed output:
Assistant
I've confirmed the issue - the MDC is empty. Now let me write the fix:
Assistant
Now let me write the fix:
Assistant
Let me run the existing tests to make sure I didn't break anything:
Assistant
Let me write a temporary test to verify the fix:
Assistant
Let me also write a test to verify RunContextLogger forward behavior:
Assistant
The MDC IS being set correctly when forwarded. Let me fix the test assertion:
Assistant
The forward appender correctly includes MDC. Now let me run the full set of existing tests to ensure nothing is broken:
Assistant
Let me check that there are no other places where MDC should be set:
Assistant
Let me also check what happens if I remove the temporary tests:
Assistant
Let me verify my changes work correctly by re-running the existing tests:
Assistant
Let me also run a broader set of tests to ensure nothing is broken:
Assistant
Let me run the most relevant tests:
Assistant
Let me run a broader set of tests to ensure everything works:
Assistant
Let me view my final fix:
Assistant
Let me check if there's a TriggerContext reference that needs updating:
Assistant
All calls use
workerTrigger.triggerId()which returnsTriggerId. So my fix is consistent with how it's used.Let me also check the TriggerId class to make sure it has the
getTriggerIdmethod:Assistant
Let me check if there are any other places that create logs for executions without setting MDC:
Assistant
Let me verify my fix compiles correctly:
Assistant
Let me check the diff:
Assistant
Let me verify the test files have been removed:
Assistant
All temporary test files are removed. Let me verify the gitstatus:
Assistant
Let me write a comprehensive test that simulates the actual issue from the report and runs it to confirm the fix:
Assistant
Let me check the actual output of the test:
Assistant
The fix is working correctly. All logs now have the MDC context properly set. Let me clean up the temporary test file:
Assistant
Let me do a final run of the related tests to confirm everything works:
Assistant
Let me verify the fix once more:
Assistant
Let me also run a broader set of related tests:
Assistant
Let me also run plugin core tests that use Logs.java:
Assistant
There's no separate "plugin" directory in this checkout. The plugins are likely under a different path. Let me check the gradle modules:
Assistant
Let me verify the final diff:
Assistant
My fix is complete. Let me do a final review:
Summary of the fix:
The issue was that
Logs.java(formerlyLogService.java) didn't set the MDC context before logging, so the MDC fields liketenantId,namespace,flowId,executionId, etc. were missing from the logs created viaLogs.logExecution(),Logs.logTaskRun(), andLogs.logTrigger().The fix:
-
Added a private helper method
withMdc()that:- Saves the current MDC state
- Sets the new MDC with the execution/taskRun/trigger context
- Runs the logging action
- Restores the previous MDC in a
finallyblock
-
Updated all log methods to use
withMdc():logExecution(FlowId, ...)- setstenantId,namespace,flowIdlogExecution(Execution, ...)- usesLogEntry.of(execution).toMap()to set MDClogTrigger(TriggerId, ...)- setstenantId,namespace,flowId,triggerIdlogTaskRun(TaskRun, ...)- usesLogEntry.of(taskRun, null).toMap()to set MDC
Verification:
- ✅ The MDC is now properly set on all logs (confirmed via temporary test)
- ✅ Existing MDC values are preserved (the
previousMdcis saved and restored) - ✅ MDC is restored even if logging throws an exception (try/finally)
- ✅ Null/empty MDC map is handled (early return)
- ✅ All existing tests pass (210 tests in core/utils and core/runners)
Let me also do the final verification:
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →