For a few days my computer is intermittently very laggy. When that happens, there's a…
- User
For a few days my computer is intermittently very laggy. When that happens, there's a bunch of kworker threads, all on CPU core 0, making it run at up to almost 100%, while the computer is essentially idle. How can I find more about what these threads are doing?
Assistant
Those
kworkerthreads can be a bit mysterious! Don't worry, I'm here to help you dig into what's going on.kworkerthreads are kernel threads that run in the background, performing various tasks such as handling interrupts, managing hardware, and executing deferred work. Since they're kernel threads, they don't show up in traditional user-space tools liketoporhtop.To investigate what's going on with these
kworkerthreads, you'll need to use some advanced Linux tools. Here's a step-by-step guide:- Identify the kworker threads: Run
ps -ef | grep kworkerto see the list ofkworkerthreads. Note down the PIDs and the CPU core they're running on (in your case, CPU core 0). - Use
topwith the-Hoption: Runtop -H -p <PID>(replace<PID>with the actual PID of one of thekworkerthreads). The-Hoption enables thread mode, and-pspecifies the PID to focus on. This will show you the thread's CPU usage, memory usage, and other details. - Check the kernel stack: Run
stack <PID>(part of theprocpspackage) to see the kernel stack for thekworkerthread. This will give you an idea of what system calls the thread is currently executing.
sudo stack <PID>- Use
systemtaporperf: These tools allow you to profile and trace kernel functions. You can use them to see what's causing thekworkerthreads to run at high CPU usage.
systemtap (if installed):
sudo stap -L -v -F -p 1 -x <PID> -c 'kprobe.function("*") { println(ppfunc()); }'This will print out the kernel function names that are currently being executed by the
kworkerthread.perf (if installed):
sudo perf top -p <PID> -F 99 -aThis will show you a live update of the kernel functions that are consuming CPU cycles.
- sysrq: You can also try triggering a sysrq ( System Request) to get more information about the current system state.
echo 'w' > /proc/sysrq-triggerThis will write a warning message to the kernel log with information about the current tasks and CPU usage.
- dmesg and kernel logs: Check the kernel logs for any error or warning messages that might be related to the
kworkerthreads.
dmesg | grep -i error dmesg | grep -i warnIf none of these steps help you identify the issue, you can also consider:
- Checking for firmware updates for your hardware
- Rebooting your system to see if the issue persists
- Looking into system-specific issues (e.g., ACPI, intel_idle, or other platform-specific drivers)
By following these steps, you should be able to gather more information about what's going on with those
kworkerthreads and potentially identify the root cause of the issue. Good luck!- Identify the kworker threads: Run
- User
sudo perf topshows the top "overhead" symbol is io_idle. I already checked the drives health (one HDD and one NVMe), and they both look fine, and there's no real disk activity when that happens. Assistant
io_idleis a kernel function that's related to I/O idle management. It's not directly related to disk I/O, but rather a mechanism to handle idle periods on various I/O devices, including CPUs, disks, and network interfaces.Given that you've already checked the drives' health and there's no significant disk activity, let's explore other possibilities:
-
CPU idle management:
io_idlemight be related to CPU idle management, specifically theintel_idleoracpi_idledrivers. These drivers manage CPU idle states, which can affect power consumption and performance.- Check if you're using the latest kernel and firmware for your CPU.
- Look into the
intel_idleoracpi_idledrivers' configuration and see if there are any issues.
-
Network interfaces: Although you mentioned no disk activity, it's possible that network interfaces are involved. Check if there are any network-related issues:
- Run
ip link showto verify that all network interfaces are up and running. - Use
tcpdumporWiresharkto monitor network traffic and see if there are any unusual patterns.
- Run
-
Other I/O devices: There might be other I/O devices contributing to the
io_idleoverhead:- Check if you have any USB devices connected and if they're functioning correctly.
- Verify that your system's clock and timer are accurate.
-
Power management: Power management features can sometimes cause issues:
- Check your system's power management settings and see if there are any unusual configurations.
- Disable any power-saving features that might be causing issues.
-
Kernel configuration: The kernel configuration might be affecting the behavior:
- Check if the
CONFIG_NO_HZandCONFIG_NO_HZ_FULLoptions are enabled. - Verify that the
intel_idleoracpi_idledrivers are properly configured.
- Check if the
To further investigate, you can try:
- Disable CPU governors: Temporarily disable CPU governors to see if it affects the behavior:
sudo bash -c "for cpu in /sys/devices/system/cpu/cpu[0-9]*; do echo performance > $cpu/cpufreq/scaling_governor; done"- sysrq: Trigger a sysrq to get more information about the current system state:
echo 'w' > /proc/sysrq-trigger- ftrace: Use
ftraceto enable tracing for theio_idlefunction:
sudo bash -c "echo io_idle > /sys/kernel/tracing/current_tracer" sudo bash -c "echo 1 > /sys/kernel/tracing/tracing_on"This will enable tracing for the
io_idlefunction and might provide more insights into what's causing the issue.If none of these steps help, you might want to consider:
- Kernel debugging: Enable kernel debugging and collect more information about the issue.
- System logs: Check system logs for any error or warning messages that might be related to the issue.
By exploring these possibilities, you should be able to gather more information about what's causing the
io_idleoverhead and potentially identify the root cause of the issue.-
- User
no, this
io_idleseems to be irrelevant, it's always at ~50%, whether core 0 is swamped by kworker threads or not.. Assistant
In that case, the
io_idlesymbol might not be directly related to the issue with thekworkerthreads.Let's focus on the
kworkerthreads again. Since they're running on core 0 and causing high CPU usage, we can try to:- Disable core 0: Temporarily disable core 0 to see if the issue persists:
sudo bash -c "echo 0 > /sys/devices/system/cpu/cpu0/online"If the issue goes away, it might indicate that there's a problem with core 0 or the kernel's scheduling on that core.
- Isolate core 0: Try isolating core 0 using the
isolcpuskernel parameter:
sudo grubby --update-kernel=ALL --args="isolcpus=0"Reboot your system and see if the issue persists.
- Check for IRQ affinity: Some interrupts might be pinned to core 0, causing the
kworkerthreads to run on that core:
sudo bash -c "for i in /sys/class/net/*/device; do echo $i; cat $i/msi_irqs; done"This will show you which interrupts are associated with each network interface. You can then use
irqbalanceorirqaffinityto rebalance the interrupts across cores.- Kernel scheduler: The kernel scheduler might be causing the
kworkerthreads to run on core 0:
sudo bash -c "echo 1 > /sys/kernel/debug/sched_debug"This will enable scheduler debugging, which might provide more insights into the scheduling decisions.
- sysrq: Trigger a sysrq to get more information about the current system state:
echo 'w' > /proc/sysrq-triggerBy exploring these possibilities, you should be able to gather more information about what's causing the
kworkerthreads to run on core 0 and potentially identify the root cause of the issue.If none of these steps help, you might want to consider:
- Kernel debugging: Enable kernel debugging and collect more information about the issue.
- System logs: Check system logs for any error or warning messages that might be related to the issue.
Keep in mind that some of these steps might require a reboot or have system-wide effects, so be cautious when experimenting.
- User
oh wait, I'm just seeing messages in journalctl that seem to coincide in time (during previous investigations I saw nothing of note in dmesg and journalctl):
Jul 02 20:08:58 hostname kernel: Uhhuh. NMI received for unknown reason 2d on CPU 15. Jul 02 20:08:58 hostname kernel: Dazed and confused, but trying to continue Assistant
Those messages in
journalctlare quite interesting. TheUhhuh. NMI received for unknown reasonmessage typically indicates that the kernel has received a Non-Maskable Interrupt (NMI) that it doesn't know how to handle.NMI's are special interrupts that can't be masked or ignored by the kernel. They're usually used for critical system events, such as:
- Machine checks (e.g., CPU errors)
- I/O errors
- Watchdog timer events
The
2din the message refers to the reason code for the NMI. Unfortunately, the kernel doesn't provide a clear explanation for this reason code.The
Dazed and confused, but trying to continuemessage is a hint that the kernel is trying to recover from the NMI, but it's not sure what's going on.Given that you're seeing these messages coincide with the
kworkerthread activity on core 0, it's possible that there's a hardware issue or a kernel bug that's causing the NMI's.Here are some potential next steps:
- Check for hardware issues: Run a stress test on your system using tools like
stressormemtest86+to see if you can reproduce the issue. - Kernel debugging: Enable kernel debugging and collect more information about the issue. You can try enabling the
CONFIG_DEBUG_NMIkernel option and see if it provides more insights. - Firmware updates: Check if there are any firmware updates available for your system, especially for the CPU, chipset, and other critical components.
- BIOS updates: Verify that your system's BIOS is up-to-date, as outdated BIOS versions can cause issues with NMI handling.
To further investigate, you can try:
- sysrq: Trigger a sysrq to get more information about the current system state:
echo 'w' > /proc/sysrq-trigger- NMI watchdog: Check if the NMI watchdog is enabled:
cat /proc/sys/kernel/nmi_watchdogIf it's enabled, you can try disabling it to see if it makes a difference:
echo 0 > /proc/sys/kernel/nmi_watchdogKeep in mind that NMI's can be caused by a variety of factors, including hardware issues, kernel bugs, or even firmware problems. It's essential to be thorough in your investigation and consider multiple possibilities.
- User
yeah, it's just
perftriggering these unknown NMI, doesn't seem to be a concern.. Assistant
In that case, it's likely that the unknown NMI's are being triggered by
perfitself, rather than being a symptom of a larger issue.perfuses various kernel mechanisms to collect performance data, including sampling and tracing. In some cases, this can trigger NMI's that the kernel doesn't recognize, resulting in the "unknown reason" messages.As long as your system is otherwise stable and functioning correctly, it's probably not a concern. You can safely ignore these messages if you're using
perffor performance analysis.In that case, let's focus on the original issue with the
kworkerthreads. Have you tried:- Disabling the
kworkerthreads: You can try disabling thekworkerthreads temporarily to see if it makes a difference:
echo 0 > /sys/kernel/softirq/threads- Checking for softirq activity: You can use
cat /proc/softirqsto see if there are any softirq activities that might be causing thekworkerthreads to run:
cat /proc/softirqs- Kernel configuration: You can try adjusting the kernel configuration to see if it affects the behavior of the
kworkerthreads.
If none of these steps help, you might want to consider:
- Kernel debugging: Enable kernel debugging and collect more information about the issue.
- System logs: Check system logs for any error or warning messages that might be related to the issue.
Let me know if you have any further questions or if there's anything else I can help you with!
- Disabling the
- User
ok, when I manage to get a stack quickly enough (not easy), what I get are amdgpu interrupts. Can I get aggregated data about that, to see which interrupts may be the cause?
Assistant
Getting a stack quickly enough can be challenging.
To get aggregated data about the interrupts, you can use the
irqbalancetool or the/proc/interruptsfile.Here's how you can use
/proc/interruptsto get aggregated data:cat /proc/interruptsThis will show you the interrupt counts for each interrupt number on each CPU. Look for the interrupts related to
amdgpuand see if any of them are incrementing rapidly.You can also use
irqbalanceto get more detailed information about the interrupts:irqbalance -dThis will show you the interrupt distribution across CPUs.
To get more specific information about the
amdgpuinterrupts, you can try:cat /sys/class/drm/card0/device/msi_irqsThis will show you the MSI (Message Signaled Interrupts) IRQs used by the
amdgpudevice.You can also use
perfto get more detailed information about the interrupts:perf irqThis will show you the interrupt counts and distribution across CPUs.
By analyzing this data, you should be able to identify which interrupts are being triggered rapidly and potentially causing the
kworkerthreads to run.Keep in mind that interrupts can be triggered by various factors, including hardware events, driver activity, and system calls. So, you may need to dig deeper to understand the root cause of the issue.
If you're still having trouble, feel free to share more data or details, and I'll do my best to help you troubleshoot!
- User
ok, after changing the graphics card, the symptoms haven't reoccured so far. I still have no idea what the card was telling the kernel making one core freeze..
Assistant
Changing the graphics card seems to have resolved the issue, at least for now.
It's not uncommon for graphics cards to have issues that can cause system instability, especially if they're not compatible with the system or have firmware issues.
As for what the card was telling the kernel to make one core freeze, it's possible that it was a combination of factors, such as:
- Firmware issues: The card's firmware might have been buggy or outdated, causing it to send incorrect or malformed interrupts to the kernel.
- Driver issues: The kernel driver for the graphics card might have been misinterpreting the card's signals or not handling them correctly.
- Hardware issues: The card itself might have had a hardware fault or a manufacturing defect that caused it to malfunction.
Without more information or debugging data, it's difficult to pinpoint the exact cause. However, it's clear that replacing the card resolved the issue, which suggests that the problem was likely related to the card.
If you're interested in further investigation, you could try:
- Enabling kernel debugging: You can enable kernel debugging and collect more information about the issue. This might help you understand what was happening when the card was causing the system to freeze.
- Checking for firmware updates: You can check if there are any firmware updates available for the graphics card. This might help resolve any issues related to the card's firmware.
- Testing with different drivers: You can try testing the system with different graphics drivers to see if the issue persists.
However, since the issue seems to be resolved, it's probably not worth investing too much time and effort into further investigation. If you have any other issues or concerns, feel free to ask, and I'll do my best to help!
- User
yes, to clarify that again, when the issues occured, dmesg and systemd were reporting nothing of note, no errors with the GPU in particular. Also nvtop was showing normal diagnostics, load and temperature in particular.
Assistant
So, to summarize:
- The issues occurred with the original graphics card, but not with the replacement card.
- During the issues,
dmesgandsystemdlogs showed no errors or warnings related to the GPU. nvtop(or presumablyamdgpu_topfor AMD GPUs) showed normal diagnostics, including load and temperature.
This suggests that the issue was likely related to a low-level interaction between the GPU and the kernel, possibly due to a firmware or hardware issue with the original card.
Given that
dmesgandsystemdlogs showed no errors, it's possible that the issue was not related to a traditional kernel or driver error, but rather a more subtle or intermittent problem.The fact that
nvtop(oramdgpu_top) showed normal diagnostics also suggests that the issue was not related to a simple GPU overload or thermal issue.It's great that replacing the card resolved the issue, and it's likely that the problem was specific to the original card. If you have any other questions or concerns, feel free to ask!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.