ISSUE DESCRIPTION MDC context no longer applied consistently in task/flow logs after…
- User
ISSUE DESCRIPTION
MDC context no longer applied consistently in task/flow logs after upgrade to v1.0
Describe the issue
Hi,
We recently upgraded from kestra v0.19.5 to v1.0.3 and noticed a change in the logging behavior.
when using a logback config as such
<?xml version="1.0" encoding="UTF-8"?> <configuration debug="false"> <include resource="logback/base.xml" /> <include resource="logback/gcp.xml" /> <root level="WARN"> <appender-ref ref="CONSOLE_JSON_OUT" /> <appender-ref ref="CONSOLE_JSON_ERR" /> </root> </configuration>the additionnal info from the task/flow/execution are not always logged.
example from a Hello-world flow run:
{"timestampSeconds":1761224032,"timestampNanos":565000000,"severity":"INFO","thread":"jdbc-execution-executor_0","logger":"execution.gnu_170208","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [execution: 1FLhPmmXgvIghVXzW7hY90] Flow started","context":"default"} {"timestampSeconds":1761224032,"timestampNanos":775000000,"severity":"INFO","thread":"worker_0","logger":"task.gnu_170208.hello","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [task: hello] [execution: 1FLhPmmXgvIghVXzW7hY90] [taskrun: 7SKeqte94nC0xUfrgFXHpw] Type Log started","context":"default"} {"taskRunId":"7SKeqte94nC0xUfrgFXHpw","tenantId":"main","namespace":"company.team","executionId":"1FLhPmmXgvIghVXzW7hY90","flowId":"gnu_170208","taskId":"hello","timestampSeconds":1761224032,"timestampNanos":790000000,"severity":"INFO","thread":"worker_0","logger":"flow.gnu_170208.hello","message":"Hello World! 🚀"} {"timestampSeconds":1761224032,"timestampNanos":799000000,"severity":"INFO","thread":"worker_0","logger":"task.gnu_170208.hello","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [task: hello] [execution: 1FLhPmmXgvIghVXzW7hY90] [taskrun: 7SKeqte94nC0xUfrgFXHpw] Type Log with state SUCCESS completed in 00:00:00.232","context":"default"} {"timestampSeconds":1761224032,"timestampNanos":852000000,"severity":"INFO","thread":"jdbc-execution-executor_2","logger":"execution.gnu_170208","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [execution: 1FLhPmmXgvIghVXzW7hY90] Flow completed with state SUCCESS in 00:00:00.405","context":"default"}Only the hello world log itself gets the MDC context.
It was different in v0.19.5 we relied on this behavior to monitor flow executions.
I've debug kestra locally and i've been able to add that context by manually setting the contextMap in this method https://github.com/kestra-io/kestra/blob/eb22d3f6ee5b6b92f8a12f03948e7517a02e39cc/core/src/main/java/io/kestra/core/services/LogService.java#L45
such as :
public void logExecution(Execution execution, Level level, String message, Object... args) { Logger logger = logger(execution); MDC.setContextMap(Map.of("tenantId", execution.getTenantId(), "namespace", execution.getNamespace(), "flowId", execution.getFlowId(), "executionId", execution.getId())); logExecution(execution, logger, level, message, args); }Environment
- Kestra Version: develop, v1.0
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /kestra, current file: none) bash-$
- Assistant
I'll start by exploring the repository structure to understand the codebase and locate the relevant files mentioned in the issue. Let's first check the current directory and see what's there.
- Assistant
Let me look for LogService.java more specifically.
- Assistant
Let me search more broadly for LogService.
- Assistant
Let me look at ExecutionLogService.java since it seems like LogService might have been renamed.
- Assistant
The issue mentions a
LogServiceclass atcore/src/main/java/io/kestra/core/services/LogService.java, but it seems like it might have been renamed or the file doesn't exist in the current version. Let me look at the git history and search more broadly. - Assistant
Let me look at the Logs utility class mentioned in the issue.
- Assistant
Now let me look at the logback configuration files to understand how logging is configured.
- Assistant
Let me check git history to find the old LogService.java that was referenced in the issue.
- Assistant
Let me search for the specific commit mentioned in the issue and find the LogService.java file.
- Assistant
Let me look at the old LogService.java to understand what was removed.
- Assistant
Now let me check the git history to see if MDC was previously used in LogService.
- Assistant
Let me look at the commit that mentions "use the MDC from the logging context" to understand the fix.
- Assistant
Let me look at this commit more closely.
- Assistant
Now let me understand how
Logsclass is used and where MDC context needs to be set. Let me look at how execution logs are written. - Assistant
Let me look at the RunContextLogger to understand how MDC is handled for task logs.
- Assistant
Let me read the RunContextLogger to understand how MDC context is set for task-level logs.
- Assistant
Now let me look at the logback configuration files to understand how MDC is used for the output.
- Assistant
Let me find the StackdriverJsonLayout to understand how MDC context is rendered.
- Assistant
Let me look at the StackdriverJsonLayout to understand how MDC is used.
- Assistant
Now let me look at the LogEntry model to understand what fields are available.
- Assistant
Now let me understand how the execution lifecycle logs are generated. Let me look at the executor and worker files.
- Assistant
Now let me understand the FlowId class to see what fields are available.
- Assistant
Now let me look at the TriggerId class to see what fields it has.
- Assistant
Let me look at the LogEntry toMap method and also the TaskRun model.
- Assistant
Now let me understand the full picture. Let me look at the LogsTest to understand the current tests.
- Assistant
Now I understand the issue clearly. Let me examine the current code in
Logs.javamore carefully to understand the exact implementation. - Assistant
Now I understand the issue. Let me trace through it:
- The
Logsutility class provides static methods for logging execution/task/trigger lifecycle events - These log calls don't set MDC context (Mapped Diagnostic Context)
- When using structured logging like
StackdriverJsonLayout, the MDC context is included in JSON output viaevent.getMDCPropertyMap() - The
RunContextLogger(for task flow logs) sets MDC, but theLogsutility class methods don't - So lifecycle logs like "Flow started", "Type Log started", etc. don't have MDC context
Let me look at how the
RunContextLoggersets MDC to understand the pattern: - The
- Assistant
Now I understand the full picture. Let me look at how the
TriggerContextmodel looks to see if it extendsTriggerId: - Assistant
Now I have a complete understanding of the issue. Let me create a reproduction script to verify the problem exists.
- Assistant
Now I understand the issue fully. The
Logsutility class (formerlyLogService) doesn't set SLF4J MDC (Mapped Diagnostic Context) context when logging. This means that when using JSON log appenders likeStackdriverJsonLayoutthat read MDC properties, the lifecycle logs (Flow started, Task started, etc.) don't include the structured context fields liketenantId,namespace,flowId,executionId, etc.The
RunContextLoggerused for actual task output (like "Hello World! 🚀") correctly sets MDC on its own LoggerContext, which is why those logs have MDC fields. But theLogsutility methods don't.Let me now implement the fix:
- Assistant
Let me use the open and edit commands correctly.
- Assistant
Let me just use open without --file:
- Assistant
Now I have a complete understanding of the issue. The
Logsutility class doesn't set SLF4J MDC context when logging, so when using custom logback configurations that rely on MDC (likeStackdriverJsonLayoutwhich readsevent.getMDCPropertyMap()), the structured context fields (tenantId, namespace, flowId, etc.) are missing from the JSON output. TheRunContextLoggercorrectly sets MDC for task flow logs, but theLogsutility methods don't.Let me implement the fix:
- Assistant
Let me write the complete file instead:
- Assistant
Let me verify the file was written correctly:
- Assistant
Good. Now let me run the existing tests to make sure they still pass:
- Assistant
All 4 tests pass. Now let me also write a quick test to verify the MDC is correctly being set. Let me first check if there are other test files that might test this:
- Assistant
Let me also check if there's a
TriggerContextthat thelogTriggermethod uses - the old code hadTriggerContextbut the new one hasTriggerId: - Assistant
Good. Now let me verify the fix handles the
nulltenant case correctly. Let me check if there's a task run without tenant ID in the test: - Assistant
I see the second TaskRun doesn't have a tenantId set (null).
MDC.setContextMaphandles null values fine. Let me run a broader set of tests to make sure nothing is broken: - Assistant
The full test suite takes too long. Let me just make sure the LogsTest passes:
- Assistant
All tests pass. Now let me also check if there are any other tests that test MDC behavior or integration tests that might be affected:
- Assistant
Let me also check that the
MDC.setContextMapapproach with null values works correctly. Let me write a quick test script to verify: - Assistant
Let me also check if there are any other places in the codebase where these Logs methods are called with different parameter types to make sure our fix covers all cases:
- Assistant
Let me verify the diff to make sure the change is correct:
- Assistant
The diff looks good. Let me also write a quick test to verify the MDC is actually being set correctly:
- Assistant
Let me also run a few related tests from the executor module to make sure nothing is broken there:
- Assistant
Let me find relevant tests that might test the logging pipeline:
- Assistant
Let me also run the RunContextLoggerTest to make sure nothing is broken:
- Assistant
All tests pass. Let me now run one more check to make sure the build compiles correctly:
- Assistant
Let me verify the final state of the file one more time and confirm the approach is correct:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →