ISSUE DESCRIPTION MDC context no longer applied consistently in task/flow logs after…
- User
ISSUE DESCRIPTION
MDC context no longer applied consistently in task/flow logs after upgrade to v1.0
Describe the issue
Hi,
We recently upgraded from kestra v0.19.5 to v1.0.3 and noticed a change in the logging behavior.
when using a logback config as such
<?xml version="1.0" encoding="UTF-8"?> <configuration debug="false"> <include resource="logback/base.xml" /> <include resource="logback/gcp.xml" /> <root level="WARN"> <appender-ref ref="CONSOLE_JSON_OUT" /> <appender-ref ref="CONSOLE_JSON_ERR" /> </root> </configuration>the additionnal info from the task/flow/execution are not always logged.
example from a Hello-world flow run:
{"timestampSeconds":1761224032,"timestampNanos":565000000,"severity":"INFO","thread":"jdbc-execution-executor_0","logger":"execution.gnu_170208","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [execution: 1FLhPmmXgvIghVXzW7hY90] Flow started","context":"default"} {"timestampSeconds":1761224032,"timestampNanos":775000000,"severity":"INFO","thread":"worker_0","logger":"task.gnu_170208.hello","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [task: hello] [execution: 1FLhPmmXgvIghVXzW7hY90] [taskrun: 7SKeqte94nC0xUfrgFXHpw] Type Log started","context":"default"} {"taskRunId":"7SKeqte94nC0xUfrgFXHpw","tenantId":"main","namespace":"company.team","executionId":"1FLhPmmXgvIghVXzW7hY90","flowId":"gnu_170208","taskId":"hello","timestampSeconds":1761224032,"timestampNanos":790000000,"severity":"INFO","thread":"worker_0","logger":"flow.gnu_170208.hello","message":"Hello World! 🚀"} {"timestampSeconds":1761224032,"timestampNanos":799000000,"severity":"INFO","thread":"worker_0","logger":"task.gnu_170208.hello","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [task: hello] [execution: 1FLhPmmXgvIghVXzW7hY90] [taskrun: 7SKeqte94nC0xUfrgFXHpw] Type Log with state SUCCESS completed in 00:00:00.232","context":"default"} {"timestampSeconds":1761224032,"timestampNanos":852000000,"severity":"INFO","thread":"jdbc-execution-executor_2","logger":"execution.gnu_170208","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [execution: 1FLhPmmXgvIghVXzW7hY90] Flow completed with state SUCCESS in 00:00:00.405","context":"default"}Only the hello world log itself gets the MDC context.
It was different in v0.19.5 we relied on this behavior to monitor flow executions.
I've debug kestra locally and i've been able to add that context by manually setting the contextMap in this method https://github.com/kestra-io/kestra/blob/eb22d3f6ee5b6b92f8a12f03948e7517a02e39cc/core/src/main/java/io/kestra/core/services/LogService.java#L45
such as :
public void logExecution(Execution execution, Level level, String message, Object... args) { Logger logger = logger(execution); MDC.setContextMap(Map.of("tenantId", execution.getTenantId(), "namespace", execution.getNamespace(), "flowId", execution.getFlowId(), "executionId", execution.getId())); logExecution(execution, logger, level, message, args); }Environment
- Kestra Version: develop, v1.0
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /kestra, current file: none) bash-$
- Assistant
I'll start by exploring the repository structure to understand the codebase and locate the relevant files. Let's first look at the root directory.
- Assistant
Let me look at the specific file mentioned in the issue:
core/src/main/java/io/kestra/core/services/LogService.java. - Assistant
It seems the file wasn't found. Let me search more broadly.
- Assistant
Hmm, the find didn't return results. Let me try a different approach.
- Assistant
Let me find the LogService.java file.
- Assistant
That's odd. Let me try a broader search.
- Assistant
The file might have been renamed or restructured. Let me look at ExecutionLogService.java and also search for the specific method mentioned in the issue.
- Assistant
Now let me search for MDC usage and log creation in the codebase to understand how MDC context is managed.
- Assistant
Let me understand the issue better. The problem is that MDC (Mapped Diagnostic Context) is not being set consistently for all log messages in task/flow logs. Looking at the example logs:
- "Flow started" - has MDC context in message text but NOT in JSON fields
- "Type Log started" - has MDC context in message text but NOT in JSON fields
- "Hello World! 🚀" - has MDC context in JSON fields (taskRunId, tenantId, etc.)
- "Type Log with state SUCCESS" - has MDC context in message text but NOT in JSON fields
- "Flow completed with state SUCCESS" - has MDC context in message text but NOT in JSON fields
So only the user's log (Hello World) gets the MDC context set. The issue is that
logExecutionand similar methods don't set the MDC context before logging.Let me look at the RunContextLogger to understand how MDC is managed.
- Assistant
Now I need to understand how logs are being generated for flow/task execution events (started, completed, etc.). The issue mentions that these events don't get the MDC context in the JSON fields. Let me look at the
ForwardAppender- it forwards to a logger with the same name, and this is the logger that will be affected by the user's logback config.Looking at the
ForwardAppender:- It uses
LoggerFactory.getLogger(runContextLogger.loggerName)- this creates a logger in the global LoggerContext (not the custom one created byRunContextLogger). - When the custom LoggerContext's MDC adapter is set, it only affects that specific LoggerContext, not the global one.
- The
ForwardAppenderforwards events to the global logger, but the MDC context from the custom LoggerContext isn't available in the global LoggerContext.
So the problem is that the
ForwardAppenderforwards logging events from the custom LoggerContext to a global logger. The global logger uses the standard MDC (which might not have the context set), so the JSON layout doesn't include the MDC fields.Now let me look at where execution-level logs are being generated (e.g., "Flow started", "Flow completed").
- It uses
- Assistant
Now let me look at the
Logsutility class. - Assistant
Now I understand the problem. The
Logsutility class usesLoggerFactory.getLogger()which gets a logger from the global logger context, not from theRunContextLogger's custom logger context. TheRunContextLoggerhas its own MDC set up ininitializeLogger(), but whenLogs.logExecution()or similar methods log, they use the global logger which doesn't have the MDC context.Let me check what fields
LogEntry.toMap()returns: - Assistant
Let me look at the full LogEntry class to understand the toMap method.
- Assistant
Now I understand the issue. The
Logsclass logs using standardLoggerFactory.getLogger()loggers (global context loggers), which means the MDC context is not set for these log messages. The issue is about the JSON output format - when usingCONSOLE_JSON_OUTorCONSOLE_JSON_ERRappenders, they read MDC properties from the logging event, but the standard loggers used byLogsclass don't have MDC set.The fix should be to set the MDC context (using
org.slf4j.MDC) before logging execution-related messages and clear it after, so that when the JSON layout reads MDC properties, they are present.Let me look at the existing test for this class to understand the expected behavior.
- Assistant
Now I understand the issue clearly. When the
Logsutility class logs execution messages, it uses loggers from the global SLF4J context (LoggerFactory.getLogger(...)), which doesn't have MDC set. TheRunContextLoggercreates its own LoggerContext with MDC, which is used for task/flow-specific logs, but theLogsclass uses the global logger without setting MDC context.The fix should set the MDC context in the
Logsclass before logging, and clear it afterward. MDC is thread-local, so setting it in the current thread before logging and clearing it after should be safe.Let me implement the fix:
- Assistant
Now let me look at the StackdriverJsonLayout to understand how it reads MDC context:
- Assistant
Now I can see the issue clearly. Looking at the
StackdriverJsonLayout.toJsonMap():if (this.includeMDC) { map.putAll(event.getMDCPropertyMap()); }It reads MDC from
event.getMDCPropertyMap(). For logs created by theRunContextLogger(task logs like "Hello World"), the MDC is set on the custom logger context, and theForwardAppenderforwards these events to the global logger - but the events retain their MDC from the custom context. So they show up in JSON with MDC fields.But for logs created by the
Logsclass (execution-level logs like "Flow started"), they use the standard SLF4J MDC (org.slf4j.MDC), which is global. If nobody sets the standard MDC before these log calls, theevent.getMDCPropertyMap()will be empty.Now, the question is: where should we set the MDC? We have a few options:
- Set it in each
Logsmethod before logging and clear it after - Set it in the calling code (e.g.,
ExecutorService,WorkerTaskProcessor, etc.) - Add MDC context propagation to the
ForwardAppender
Looking at the issue more carefully, I see that:
- The issue reporter shows logs from the
ForwardAppender(forwarded to global logger), not from theLogsclass - Wait, looking at the example output:
- "execution.gnu_170208" logger - this is from
Logs.logExecutionusing the executor logger - "task.gnu_170208.hello" logger - this is from
Logs.logTaskRunusing the worker logger - "flow.gnu_170208.hello" logger - this is from theForwardAppenderforwarding fromRunContextLogger
- "execution.gnu_170208" logger - this is from
So the logs from
execution.andtask.loggers are missing MDC context, while theflow.logger (fromForwardAppender) has MDC because it's forwarded with the custom context.Actually wait, looking at the example again:
{"timestampSeconds":1761224032,"timestampNanos":565000000,"severity":"INFO","thread":"jdbc-execution-executor_0","logger":"execution.gnu_170208","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [execution: 1FLhPmmXgvIghVXzW7hY90] Flow started","context":"default"}This doesn't have tenantId, namespace, etc. as JSON fields - only in the message text.
{"taskRunId":"7SKeqte94nC0xUfrgFXHpw","tenantId":"main","namespace":"company.team","executionId":"1FLhPmmXgvIghVXzW7hY90","flowId":"gnu_170208","taskId":"hello","timestampSeconds":1761224032,"timestampNanos":790000000,"severity":"INFO","thread":"worker_0","logger":"flow.gnu_170208.hello","message":"Hello World! 🚀"}This one has all the MDC fields as JSON keys.
So the fix is to set MDC in the
Logsclass before logging and clear it after.Let me implement this:
Now I understand the full picture. The
StackdriverJsonLayoutreads MDC fromevent.getMDCPropertyMap(). For logs created by theRunContextLogger(task logs like "Hello World"), the MDC is set on the custom logger context and theForwardAppenderforwards these events to the global logger while retaining their MDC from the custom context. So they show up in JSON with MDC fields.But for logs created by the
Logsclass (execution-level logs like "Flow started", "Type started", "Type completed", "Flow completed"), they use the standard SLF4J MDC, and nobody sets it before these log calls.The fix should set MDC in the
Logsclass before logging and clear it after. MDC is thread-local, so this is safe in a multi-threaded environment.Let me implement this:
- Set it in each
- Assistant
Let me try using a different approach to edit the file:
- Assistant
Good, the file has been updated. Now let me also update the test to verify the MDC context is being set. Let me first check the existing test more closely:
Now let me verify the file looks correct and run the existing tests:
- Assistant
Now let me also check the
WorkerTaskProcessorto see how it logs: - Assistant
Good, I can see that the
WorkerTaskProcessorcallsLogs.logTaskRun(...)for the "Type {} started" and "Type {} with state {} completed" messages. TheLogsclass uses global loggers without setting MDC on the global MDC adapter, which is why the JSON output doesn't include the MDC fields.Now let me run the existing tests to make sure my changes don't break anything:
- Assistant
All 4 existing tests pass. Now let me also check if there are other tests that might be affected by this change:
- Assistant
All the existing tests pass. Let me now run a broader set of related tests to make sure nothing is broken:
- Assistant
All 191 tests pass. Let me also verify the
FlowIdclass to make sure our null check is appropriate: - Assistant
Let me find FlowId:
- Assistant
Let me check the FlowId interface to understand the tenantId handling.
- Assistant
Now let me also check the executor code that logs "Flow started" and "Flow completed" to confirm those use the
Logsclass: - Assistant
Good, the "Flow started" and "Flow completed" messages are indeed using
Logs.logExecution(...), which my fix addresses.Now let me also check whether there's a potential issue with the
FlowIdmethod - specifically ifgetTenantId()could be null for some callers. I need to make sure theMap.ofcall in thelogExecution(FlowId, Logger, Level, String, Object...)method handles null tenantId: - Assistant
Let me also check if
FlowextendsFlowIdand ifgetTenantId()can be null: - Assistant
Let me check if
FlowimplementsFlowId: - Assistant
Let me check what
Flowextends/implements: - Assistant
Let me check if AbstractFlow implements FlowId.
- Assistant
The
Flowclass extendsAbstractFlowand implementsHasUID, notFlowId. But thelogExecution(FlowId, Logger, Level, String, Object...)method takes aFlowIdparameter. SoFlowmight implementFlowIdthrough its parent class or some other mechanism.Let me check the
Flowclass more carefully: - Assistant
Let me check the
AbstractFlowclass to see if it implementsFlowId. - Assistant
Let me check FlowInterface to see if it extends FlowId.
- Assistant
Let me check if FlowInterface extends FlowId:
- Assistant
Good, so
FlowInterface extends FlowIdandFlow extends AbstractFlow implements FlowInterfacesoFlowimplementsFlowId. ThegetTenantId()can be null for single-tenant setups, which my code handles.Let me also check the
TriggerIdclass: - Assistant
Let me also look at the
WorkerTriggerProcessorto check how it callsLogs.logExecution: - Assistant
Good. Now let me run the test again to make sure everything passes and let me also run a broader set of executor/worker tests:
- Assistant
All tests pass. Let me also verify that the
ForwardAppenderinRunContextLoggerproperly handles MDC. Let me check if there's an issue with how the event's MDC is preserved when forwarding: - Assistant
The diff looks clean and correct. Let me now run a broader set of tests to make sure nothing is broken:
- Assistant
Let me just run the core tests without the checkstyle exclusion.
- Assistant
The full core tests take too long. Let me just run the specific test and a few related ones:
- Assistant
All 9 tests pass. Let me do a final review of my changes:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-lunaCompare side by side →