Skip to content
TheChatArchiveThe Chat Archive
xiaomi/mimo-v2.5-pro· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION MDC context no longer applied consistently in task/flow logs after…

  1. User

    ISSUE DESCRIPTION

    MDC context no longer applied consistently in task/flow logs after upgrade to v1.0

    Describe the issue

    Hi,

    We recently upgraded from kestra v0.19.5 to v1.0.3 and noticed a change in the logging behavior.

    when using a logback config as such

    <?xml version="1.0" encoding="UTF-8"?>
    <configuration debug="false">
        <include resource="logback/base.xml" />
        <include resource="logback/gcp.xml" />
    
        <root level="WARN">
            <appender-ref ref="CONSOLE_JSON_OUT" />
            <appender-ref ref="CONSOLE_JSON_ERR" />
        </root>
    </configuration>
    

    the additionnal info from the task/flow/execution are not always logged.

    example from a Hello-world flow run:

    {"timestampSeconds":1761224032,"timestampNanos":565000000,"severity":"INFO","thread":"jdbc-execution-executor_0","logger":"execution.gnu_170208","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [execution: 1FLhPmmXgvIghVXzW7hY90] Flow started","context":"default"}
    {"timestampSeconds":1761224032,"timestampNanos":775000000,"severity":"INFO","thread":"worker_0","logger":"task.gnu_170208.hello","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [task: hello] [execution: 1FLhPmmXgvIghVXzW7hY90] [taskrun: 7SKeqte94nC0xUfrgFXHpw] Type Log started","context":"default"}
    {"taskRunId":"7SKeqte94nC0xUfrgFXHpw","tenantId":"main","namespace":"company.team","executionId":"1FLhPmmXgvIghVXzW7hY90","flowId":"gnu_170208","taskId":"hello","timestampSeconds":1761224032,"timestampNanos":790000000,"severity":"INFO","thread":"worker_0","logger":"flow.gnu_170208.hello","message":"Hello World! 🚀"}
    {"timestampSeconds":1761224032,"timestampNanos":799000000,"severity":"INFO","thread":"worker_0","logger":"task.gnu_170208.hello","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [task: hello] [execution: 1FLhPmmXgvIghVXzW7hY90] [taskrun: 7SKeqte94nC0xUfrgFXHpw] Type Log with state SUCCESS completed in 00:00:00.232","context":"default"}
    {"timestampSeconds":1761224032,"timestampNanos":852000000,"severity":"INFO","thread":"jdbc-execution-executor_2","logger":"execution.gnu_170208","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [execution: 1FLhPmmXgvIghVXzW7hY90] Flow completed with state SUCCESS in 00:00:00.405","context":"default"}
    
    

    Only the hello world log itself gets the MDC context.

    It was different in v0.19.5 we relied on this behavior to monitor flow executions.

    I've debug kestra locally and i've been able to add that context by manually setting the contextMap in this method https://github.com/kestra-io/kestra/blob/eb22d3f6ee5b6b92f8a12f03948e7517a02e39cc/core/src/main/java/io/kestra/core/services/LogService.java#L45

    such as :

    public void logExecution(Execution execution, Level level, String message, Object... args) {
        Logger logger = logger(execution);
        MDC.setContextMap(Map.of("tenantId", execution.getTenantId(), "namespace", execution.getNamespace(), "flowId", execution.getFlowId(), "executionId", execution.getId()));
    
        logExecution(execution, logger, level, message, args);
    }
    

    Environment

    • Kestra Version: develop, v1.0

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /kestra, current file: none) bash-$

  2. Assistant

    I'll start by exploring the repository structure to understand the codebase and locate the relevant files. Let's first look at the root directory.

  3. Assistant

    Let me look at the specific file mentioned in the issue: core/src/main/java/io/kestra/core/services/LogService.java.

  4. Assistant

    It seems the file wasn't found. Let me search more broadly.

  5. Assistant

    Hmm, the find didn't return results. Let me try a different approach.

  6. Assistant

    Let me find the LogService.java file.

  7. Assistant

    That's odd. Let me try a broader search.

  8. Assistant

    The file might have been renamed or restructured. Let me look at ExecutionLogService.java and also search for the specific method mentioned in the issue.

  9. Assistant

    Now let me search for MDC usage and log creation in the codebase to understand how MDC context is managed.

  10. Assistant

    Let me understand the issue better. The problem is that MDC (Mapped Diagnostic Context) is not being set consistently for all log messages in task/flow logs. Looking at the example logs:

    1. "Flow started" - has MDC context in message text but NOT in JSON fields
    2. "Type Log started" - has MDC context in message text but NOT in JSON fields
    3. "Hello World! 🚀" - has MDC context in JSON fields (taskRunId, tenantId, etc.)
    4. "Type Log with state SUCCESS" - has MDC context in message text but NOT in JSON fields
    5. "Flow completed with state SUCCESS" - has MDC context in message text but NOT in JSON fields

    So only the user's log (Hello World) gets the MDC context set. The issue is that logExecution and similar methods don't set the MDC context before logging.

    Let me look at the RunContextLogger to understand how MDC is managed.

  11. Assistant

    Now I need to understand how logs are being generated for flow/task execution events (started, completed, etc.). The issue mentions that these events don't get the MDC context in the JSON fields. Let me look at the ForwardAppender - it forwards to a logger with the same name, and this is the logger that will be affected by the user's logback config.

    Looking at the ForwardAppender:

    1. It uses LoggerFactory.getLogger(runContextLogger.loggerName) - this creates a logger in the global LoggerContext (not the custom one created by RunContextLogger).
    2. When the custom LoggerContext's MDC adapter is set, it only affects that specific LoggerContext, not the global one.
    3. The ForwardAppender forwards events to the global logger, but the MDC context from the custom LoggerContext isn't available in the global LoggerContext.

    So the problem is that the ForwardAppender forwards logging events from the custom LoggerContext to a global logger. The global logger uses the standard MDC (which might not have the context set), so the JSON layout doesn't include the MDC fields.

    Now let me look at where execution-level logs are being generated (e.g., "Flow started", "Flow completed").

  12. Assistant

    Now let me look at the Logs utility class.

  13. Assistant

    Now I understand the problem. The Logs utility class uses LoggerFactory.getLogger() which gets a logger from the global logger context, not from the RunContextLogger's custom logger context. The RunContextLogger has its own MDC set up in initializeLogger(), but when Logs.logExecution() or similar methods log, they use the global logger which doesn't have the MDC context.

    Let me check what fields LogEntry.toMap() returns:

  14. Assistant

    Let me look at the full LogEntry class to understand the toMap method.

  15. Assistant

    Now I understand the issue. The Logs class logs using standard LoggerFactory.getLogger() loggers (global context loggers), which means the MDC context is not set for these log messages. The issue is about the JSON output format - when using CONSOLE_JSON_OUT or CONSOLE_JSON_ERR appenders, they read MDC properties from the logging event, but the standard loggers used by Logs class don't have MDC set.

    The fix should be to set the MDC context (using org.slf4j.MDC) before logging execution-related messages and clear it after, so that when the JSON layout reads MDC properties, they are present.

    Let me look at the existing test for this class to understand the expected behavior.

  16. Assistant

    Now I understand the issue clearly. When the Logs utility class logs execution messages, it uses loggers from the global SLF4J context (LoggerFactory.getLogger(...)), which doesn't have MDC set. The RunContextLogger creates its own LoggerContext with MDC, which is used for task/flow-specific logs, but the Logs class uses the global logger without setting MDC context.

    The fix should set the MDC context in the Logs class before logging, and clear it afterward. MDC is thread-local, so setting it in the current thread before logging and clearing it after should be safe.

    Let me implement the fix:

  17. Assistant

    Now let me look at the StackdriverJsonLayout to understand how it reads MDC context:

  18. Assistant

    Now I can see the issue clearly. Looking at the StackdriverJsonLayout.toJsonMap():

    if (this.includeMDC) {
        map.putAll(event.getMDCPropertyMap());
    }
    

    It reads MDC from event.getMDCPropertyMap(). For logs created by the RunContextLogger (task logs like "Hello World"), the MDC is set on the custom logger context, and the ForwardAppender forwards these events to the global logger - but the events retain their MDC from the custom context. So they show up in JSON with MDC fields.

    But for logs created by the Logs class (execution-level logs like "Flow started"), they use the standard SLF4J MDC (org.slf4j.MDC), which is global. If nobody sets the standard MDC before these log calls, the event.getMDCPropertyMap() will be empty.

    Now, the question is: where should we set the MDC? We have a few options:

    1. Set it in each Logs method before logging and clear it after
    2. Set it in the calling code (e.g., ExecutorService, WorkerTaskProcessor, etc.)
    3. Add MDC context propagation to the ForwardAppender

    Looking at the issue more carefully, I see that:

    • The issue reporter shows logs from the ForwardAppender (forwarded to global logger), not from the Logs class
    • Wait, looking at the example output:
      • "execution.gnu_170208" logger - this is from Logs.logExecution using the executor logger
      • "task.gnu_170208.hello" logger - this is from Logs.logTaskRun using the worker logger - "flow.gnu_170208.hello" logger - this is from the ForwardAppender forwarding from RunContextLogger

    So the logs from execution. and task. loggers are missing MDC context, while the flow. logger (from ForwardAppender) has MDC because it's forwarded with the custom context.

    Actually wait, looking at the example again:

    {"timestampSeconds":1761224032,"timestampNanos":565000000,"severity":"INFO","thread":"jdbc-execution-executor_0","logger":"execution.gnu_170208","message":"[tenant: main] [namespace: company.team] [flow: gnu_170208] [execution: 1FLhPmmXgvIghVXzW7hY90] Flow started","context":"default"}
    

    This doesn't have tenantId, namespace, etc. as JSON fields - only in the message text.

    {"taskRunId":"7SKeqte94nC0xUfrgFXHpw","tenantId":"main","namespace":"company.team","executionId":"1FLhPmmXgvIghVXzW7hY90","flowId":"gnu_170208","taskId":"hello","timestampSeconds":1761224032,"timestampNanos":790000000,"severity":"INFO","thread":"worker_0","logger":"flow.gnu_170208.hello","message":"Hello World! 🚀"}
    

    This one has all the MDC fields as JSON keys.

    So the fix is to set MDC in the Logs class before logging and clear it after.

    Let me implement this:

    Now I understand the full picture. The StackdriverJsonLayout reads MDC from event.getMDCPropertyMap(). For logs created by the RunContextLogger (task logs like "Hello World"), the MDC is set on the custom logger context and the ForwardAppender forwards these events to the global logger while retaining their MDC from the custom context. So they show up in JSON with MDC fields.

    But for logs created by the Logs class (execution-level logs like "Flow started", "Type started", "Type completed", "Flow completed"), they use the standard SLF4J MDC, and nobody sets it before these log calls.

    The fix should set MDC in the Logs class before logging and clear it after. MDC is thread-local, so this is safe in a multi-threaded environment.

    Let me implement this:

  19. Assistant

    Let me try using a different approach to edit the file:

  20. Assistant

    Good, the file has been updated. Now let me also update the test to verify the MDC context is being set. Let me first check the existing test more closely:

    Now let me verify the file looks correct and run the existing tests:

  21. Assistant

    Now let me also check the WorkerTaskProcessor to see how it logs:

  22. Assistant

    Good, I can see that the WorkerTaskProcessor calls Logs.logTaskRun(...) for the "Type {} started" and "Type {} with state {} completed" messages. The Logs class uses global loggers without setting MDC on the global MDC adapter, which is why the JSON output doesn't include the MDC fields.

    Now let me run the existing tests to make sure my changes don't break anything:

  23. Assistant

    All 4 existing tests pass. Now let me also check if there are other tests that might be affected by this change:

  24. Assistant

    All the existing tests pass. Let me now run a broader set of related tests to make sure nothing is broken:

  25. Assistant

    All 191 tests pass. Let me also verify the FlowId class to make sure our null check is appropriate:

  26. Assistant

    Let me find FlowId:

  27. Assistant

    Let me check the FlowId interface to understand the tenantId handling.

  28. Assistant

    Now let me also check the executor code that logs "Flow started" and "Flow completed" to confirm those use the Logs class:

  29. Assistant

    Good, the "Flow started" and "Flow completed" messages are indeed using Logs.logExecution(...), which my fix addresses.

    Now let me also check whether there's a potential issue with the FlowId method - specifically if getTenantId() could be null for some callers. I need to make sure the Map.of call in the logExecution(FlowId, Logger, Level, String, Object...) method handles null tenantId:

  30. Assistant

    Let me also check if Flow extends FlowId and if getTenantId() can be null:

  31. Assistant

    Let me check if Flow implements FlowId:

  32. Assistant

    Let me check what Flow extends/implements:

  33. Assistant

    Let me check if AbstractFlow implements FlowId.

  34. Assistant

    The Flow class extends AbstractFlow and implements HasUID, not FlowId. But the logExecution(FlowId, Logger, Level, String, Object...) method takes a FlowId parameter. So Flow might implement FlowId through its parent class or some other mechanism.

    Let me check the Flow class more carefully:

  35. Assistant

    Let me check the AbstractFlow class to see if it implements FlowId.

  36. Assistant

    Let me check FlowInterface to see if it extends FlowId.

  37. Assistant

    Let me check if FlowInterface extends FlowId:

  38. Assistant

    Good, so FlowInterface extends FlowId and Flow extends AbstractFlow implements FlowInterface so Flow implements FlowId. The getTenantId() can be null for single-tenant setups, which my code handles.

    Let me also check the TriggerId class:

  39. Assistant

    Let me also look at the WorkerTriggerProcessor to check how it calls Logs.logExecution:

  40. Assistant

    Good. Now let me run the test again to make sure everything passes and let me also run a broader set of executor/worker tests:

  41. Assistant

    All tests pass. Let me also verify that the ForwardAppender in RunContextLogger properly handles MDC. Let me check if there's an issue with how the event's MDC is preserved when forwarding:

  42. Assistant

    The diff looks clean and correct. Let me now run a broader set of tests to make sure nothing is broken:

  43. Assistant

    Let me just run the core tests without the checkstyle exclusion.

  44. Assistant

    The full core tests take too long. Let me just run the specific test and a few related ones:

  45. Assistant

    All 9 tests pass. Let me do a final review of my changes:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.