My Linux Kernel Development Journey: From First Patch to Race Condition Hell
Over the course of several months in early 2025, I contributed multiple patches to the Linux kernel mainline, focusing primarily on concurrency issues, string safety improvements, and hardware-specific driver fixes. This work involved identifying and resolving deadlocks in CPU frequency management, fixing argument parsing problems in a kbuild script, and investigating complex race conditions reported by Syzbot.
This post shares the technical approaches and tools that helped me contribute to the kernel, hoping it might provide some help to others interested in kernel development.
Easy Entry Points: String Safety Improvements
Why String Functions Matter
One of the most approachable ways to contribute to kernel security is fixing unsafe string operations. The difference between snprintf() and scnprintf() might seem minor, but it’s crucial for security:
snprintf()returns the number of characters that would have been written (including truncated characters)scnprintf()returns the number of characters actually written
Consider this pattern:
1
2
len = snprintf(buf, sizeof(buf), "format %s", user_input);
return len; // This could be larger than sizeof(buf)!
Using scnprintf() prevents this sort of bugs entirely.
Finding String Safety Issues
The Linux Kernel Security Project (KSPP) maintains a list of ongoing security hardening efforts. One easy starting point is removing all strcpy() uses in favor of strscpy(), which provides a systematic approach to improving string safety across the kernel. Don’t do it without understanding the code. These fixes are generally welcomed by subsystem maintainers, but not all maintainers get satisfied by a simple replacement. You might need to come up with a better approach instead.
Deadlock Hunting: CPU Frequency Management
WARNING: Deadlocks aren’t the easiest bugs for beginners to fix, avoid them at all cost. Let’s say I’m a bit… greedy.
The Discovery Process
It’s worth understanding one of the most powerful tools in kernel debugging: lockdep. Lockdep is a runtime lock validator that tracks lock dependencies and detects potential deadlocks before they occur in production.
When you enable CONFIG_PROVE_LOCKING, lockdep monitors every lock acquisition and builds a dependency graph. If it detects a scenario where Thread A holds Lock X and wants Lock Y, while Thread B holds Lock Y and wants Lock X, it immediately reports a potential deadlock, even if the actual deadlock hasn’t occurred yet.
With lockdep enabled on my development laptop, I encountered this report:
1
WARNING: possible circular locking dependency detected
The warning pointed to the CPU frequency subsystem where store_local_boost() was acquiring cpus_read_lock() while already holding other locks that could create a circular dependency.
The Evolution of a Fix
1
a0982afa0992 cpufreq: drop redundant cpus_read_lock() from store_local_boost()
This seemingly simple patch actually went through an interesting evolution that demonstrates an important principle: sometimes the best solution is to remove code, not add more.
Initial Approach: Fix the Lock Ordering
My first instinct was to fix the circular dependency by acquiring locks in the correct order. The lockdep report showed that store_local_boost() was acquiring cpu_hotplug_lock after the policy lock, violating the expected hierarchy. So I developed a complex patch that:
- Acquired
cpu_hotplug_lockin the store() handler before the policy lock - Used
scoped_guard()with GCC’s cleanup attribute for automatic lock cleanup - Restructured the locking to respect the proper hierarchy
Here’s what that looked like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c
index 21fa733a2..b349adbeb 100644
--- a/drivers/cpufreq/cpufreq.c
+++ b/drivers/cpufreq/cpufreq.c
@@ -622,10 +622,7 @@ static ssize_t store_local_boost(struct cpufreq_policy *policy,
if (!policy->boost_supported)
return -EINVAL;
- cpus_read_lock();
ret = policy_set_boost(policy, enable);
- cpus_read_unlock();
-
if (!ret)
return count;
@@ -1006,16 +1003,28 @@ static ssize_t store(struct kobject *kobj, struct attribute *attr,
{
struct cpufreq_policy *policy = to_policy(kobj);
struct freq_attr *fattr = to_attr(attr);
+ int ret = -EBUSY;
if (!fattr->store)
return -EIO;
- guard(cpufreq_policy_write)(policy);
+ /*
+ * store_local_boost() requires cpu_hotplug_lock to be held, and must be
+ * called with that lock acquired *before* taking policy->rwsem to avoid
+ * lock ordering violations.
+ */
+ if (fattr == &local_boost)
+ cpus_read_lock();
- if (likely(!policy_is_inactive(policy)))
- return fattr->store(policy, buf, count);
+ scoped_guard(cpufreq_policy_write, policy) {
+ if (likely(!policy_is_inactive(policy)))
+ ret = fattr->store(policy, buf, count);
+ }
- return -EBUSY;
+ if (fattr == &local_boost)
+ cpus_read_unlock();
+
+ return ret;
}
Learning About Scoped Guards
guard() and scoped_guard() are neat usages of modern C techniques for resource management. They use GCC’s __attribute__((cleanup)) to automatically call cleanup functions when variables go out of scope:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#define CLASS(_name, var) \
class_##_name##_t var __cleanup(class_##_name##_destructor) = \
class_##_name##_constructor
#define guard(_name) \
CLASS(_name, __UNIQUE_ID(guard))
void hmm(void) {
guard(mutex)(&foo->lock); /* Automatically unlocked when function exits */
/* ... critical section code ... */
}
#define __scoped_guard(_name, _label, args...) \
for (CLASS(_name, scope)(args); \
__guard_ptr(_name)(&scope) || !__is_cond_ptr(_name); \
({ goto _label; })) \
if (0) { \
_label: \
break; \
} else
#define scoped_guard(_name, args...) \
__scoped_guard(_name, __UNIQUE_ID(label), args)
/*
* When 'scope' goes out of scope, the cleanup function automatically
* releases the lock
*/
This pattern prevents lock leaks and makes the code more maintainable by ensuring locks are always released properly. You can use it for memory management as well, like how systemd uses it.
The Better Solution: Question the Lock’s Purpose
During code review, maintainers asked a crucial question: “Why does this lock exist at all?”
After some analysis, we discovered that store_local_boost() was acquiring cpu_hotplug_lock redundantly. The calling context already provided the necessary protection. It was only called from places where protection was already in place.
The Final Fix
The final solution was to simply remove the lock acquisition:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c
index 731ecfc17..759dd178a 100644
--- a/drivers/cpufreq/cpufreq.c
+++ b/drivers/cpufreq/cpufreq.c
@@ -622,10 +622,7 @@ static ssize_t store_local_boost(struct cpufreq_policy *policy,
if (!policy->boost_supported)
return -EINVAL;
- cpus_read_lock();
ret = policy_set_boost(policy, enable);
- cpus_read_unlock();
-
if (!ret)
return count;
Not Every Fix Is a Cure for Cancer
1
f757f6011c92 kbuild: fix argument parsing in scripts/config
The scripts/config script previously assumed that --file was always the first argument, which caused issues when it appeared later. This commit updates the parsing logic to scan all arguments for –file and set the config file correctly. It also fixes --refresh so that it respects --file by passing KCONFIG_CONFIG=$FN to make oldconfig (a change that had been marked as a TODO!).
You can look for TODO, FIXME, and XXX comments to address current issues in the kernel, but be aware that not all of them are as straightforward as this one.
Race Condition Analysis
Understanding Syzbot
Syzbot is Google’s automated kernel fuzzing system that continuously tests the Linux kernel by generating random system calls and monitoring for crashes, hangs, or other anomalies. It’s an incredible resource for finding real bugs that affect production systems.
The ZRAM Mystery
One of the most challenging bugs I investigated was Syzbot report extid+1a281a451fd8c0945d07, involving a race condition in the ZRAM module.
The Problem: During device reset operations, there’s a narrow window where the compression algorithm pointer becomes NULL, but concurrent sysfs reads can still access it, causing crashes.
The Complexity: The race window logically doesn’t even exist (at least that’s how it seems) and occurs under lock protection, making it theoretically impossible according to the locking semantics.
Race Condition Debugging Techniques
For learning about the fundamental concepts behind race conditions, deadlocks, and concurrent programming in systems code, I highly recommend Is Parallel Programming Hard, And, If So, What Can You Do About It? by Paul McKenney. This book is freely available and covers everything from basic synchronization primitives to advanced lock-free programming techniques used in the kernel.
Community Collaboration
Understanding Multi-Contributor Patches
1
2b9f84e7dc86 platform/x86: thinkpad_acpi: disable ACPI fan access for T495* and E560
Sometimes you might have to carry a patch on behalf of someone else. The patch mentioned above is an example. The problem involved ThinkPad laptops where ACPI fan control methods (FANG+FANW) existed in the DSDT table but didn’t actually function, causing “No such device or address” errors. A previous commit (57d0557dfa49) added support for newer ACPI fan control methods, but certain laptop models (T495, T495s, E560) have these methods in their firmware without proper implementation. The solution required disabling the broken ACPI methods and falling back to legacy fan control.
his patch involved:
- Reported-by: Vlastimil Holer (original bug reporter)
- Main author: Eduard Christian Dumitrescu (primary patch developer)
- Co-developed-by: Myself (contributed additional laptop models and refinements)
- Tested-by: Alireza Elikahi (verified the fix worked)
- Reviewed-by: Kurt Borja and Ilpo Järvinen (code review)
Proper Patch Attribution
The kernel uses specific tags to credit contributions properly, as detailed in the patch formatting documentation:
1
2
3
4
5
6
Reported-by: Person who found the bug
Fixes: commit-hash ("commit title") - Links to the commit that introduced the issue
Tested-by: Person who verified the fix works
Reviewed-by: Person who performed code review
Co-developed-by: Significant contributor to the patch
Signed-off-by: Legal certification that you can contribute this code
This attribution system ensures proper credit while creating a clear audit trail of who was involved in identifying, developing, testing, and reviewing each change.
Complex patches often build on existing community work. In this case, Eduard had already developed a solution but needed help with adding additional affected laptop models, proper patch formatting for submission and ensuring the fix was comprehensive.
Using Co-developed-by acknowledges collaborative development while maintaining clear authorship. Kernel development isn’t just about writing code… it’s about participating in a community process.
Final Notes
Huge thanks to Shuah Khan for helping me at the Linux Foundation and giving me the courage to actually contribute to the kernel. Also, thank you to the other maintainers and reviewers who were kind enough to keep my sanity during contributions—or strict (especially the inotify maintainers)—from whom I learned a lot.