diff --git a/ch 8. Spinlocks.md b/ch 8. Spinlocks.md new file mode 100644 index 0000000..92aa50d --- /dev/null +++ b/ch 8. Spinlocks.md @@ -0,0 +1,347 @@ +# Spinlocks + +This document aims to describe the inner workings of a spin-lock in AtOS. +Before discussing that, first we try to motivate the requirement for it, then +discuss the various implementation details and setup some groundwork for +mutexes. + +## Historical Background and Motivation + +Processors in the old days used to run on a single core. This meant that there +was only one CPU, running a scheduler and context-switching between processes. +Things were simpler, whatever operations you needed to perform, you perform it +and wait till it completes and do the next set of operations. Then, we figured +how to have multiple processors, effectively introducing parallelism, making +computers faster. However, if you're executing a program on multiple +processors, you have the benefit of running chunks of the program on different +processors, effectively dividing the work. Having multiple processors basically +allows you to perform operations faster and do so in parallel and thus a modern +system must utilize that capability. + +Our second need is a bit subtler. Let's say a programmer writes a game and the +game needs do some I/O operation. In a single core, the I/O might take a lot of +time and thus greatly reduce the performance and user experience. Imagine +waiting for eternity for a background I/O task to complete. That would be +devastating. Also, while the I/O operation is being performed, the CPU just +sits idly, blocked/waiting for the I/O task to finish, effectively wasting that +computing power being idle. + +## Locks and Threads + +Now, we introduce the concept of locks. Let us say you have two threads working +on a program, trying to speed it up. Your main function starts running, creates +the two threads, and the threads can execute in any way since you don't control +how the scheduler decides to schedule them. So it may happen that a thread, +say, T1 runs first followed by the other thread, T2. But it is equally possible +that the CPU decides to run T2 and then run T1. + +This is not a big problem when both the threads are doing different tasks but +big problems occur when you're accessing a shared resource. For example, let's +say in an operating system process demands a memory page from the kernel (if you +don't know what pages are, don't worry, just know that in AtOS we divide the +memory into fixed-size blocks called pages and this technique is called +paging). To get a page, you need to traverse through something called a +freelist, which is a linked list keeping track of the pages that are free for +use i.e., other programs are not using that chunk of memory. Now, multiple +processes can run on different cores of the CPU at the same time and this means, +that there might be ONE small possibility that two different programs access the +freelist and choose the first available chunk of memory and now, you got +yourself a problem. + +Depending on the code, you are going to mess up your freelist data structure. +Let's say another process just freed some memory and it tried to modify the +freelist at the same time as the other two processes and now you really can't +make any guarantees on the integrity of the data. There are many cases of +things like this happening where the expected deterministic output becomes +nondeterministic and so we need locks. + +There is a lot to read on concurrency, and this short text can't cover all the +intricacies of the topic and do them justice so we entrust the reader's ability +to find the necessary resources to aid the understanding of the material before +proceeding. + +## Implementation + +Hoping that we have successfully demonstrated the correct motivation to +implement a lock, let us look at the modern approach to implementing locks. +With sufficient intuition, you will also be able to deduce why the decisions +were taken the way they were. In summary, just know that initially programmers +tried to implement software locks and it didn't work out so we sought some aid +of the hardware and implemented the locks the way we did. Pure software mutual +exclusion algorithms still exist, but practical modern locks rely on atomic +primitives aided by the hardware. + +### Atomicity + +Before we read go through the structure itself, first let us discuss an +important thing known as atomicity. When an operation appears to happen all at +once, it is said to be atomic. We will discuss more about its use ahead. + +### The `Spinlock` structure + +In AtOS, you can see the spinlock structure in `kernel/spinlock.rs`: + +```rs +pub struct Spinlock { + lock: AtomicBool, + holding_cpuid: Cell>, + + // For debugging: + name: &'static str, // Name of the lock +} +``` + +As you can see, `lock` is just a boolean, you could've also used an integer. +Note that the boolean used is an AtomicBool. This is a Rust compiler thing, it +provides atomic operations with guarantees from the underlying hardware/memory +model. When the value is `false`, the lock is free; and when it is `true` it +indicates the lock being held. + +This is where atomicity comes into play. Since two threads can call to acquire +our spinlock, we need to make sure that whoever calls it either completely gains +access to it or doesn't. This functionality of atomic operation is guaranteed +by the hardware by various methods. + +The `holding_cpuid` records the CPU (logical processor/core) that currently owns +the lock. It is not used to synchronize the access to the lock itself (that is +handled by the bool), it just exists mainly to facilitate debugging. The `Cell` +wrapper is a Rust thing, that allows debugging information to be updated even +when `Spinlock` is only accessed through a shared reference `&Spinlock`. In +other words, `Cell` lets us change the stored CPU ID without requiring a mutable +reference to the entire Spinlock. + +Finally, `name` is just a human-readable name for the lock, again finding its +use in debugging more than the lock itself. + +### Hardware atomic operations + +On AArch64, there are two common ways to implement atomic operations. The older +systems used load-exclusive/store-exclusive pair of instructions (`LDXR` and +`STXR`) while the newer ARMv8.1-A processors introduced Large System Extensions +(LSE), which provide single-instruction atomic operations such as `CAS` +(compare-and-swap), `SWP` (atomic swap), and atomic arithmetic instructions. + +Regardless of which ISA we use, the goal is the same: if two cores attempt to +acquire the same lock, we just want the first thread that successfully acquires +the lock by setting the value to true, atomically. Since, we're using the Rust +Compiler, it will implement the atomic instructions for us and convert it to the +required target assembly. + +### The `acquire` function + +Now, having defined the structure, let us look at how we have implemented the +`acquire` function, which lets a thread executing inside the process acquire the +lock, effectively spinning and consuming CPU cycles while waiting. Since this +is a Spinlock implementation, the thread that acquires the lock acquires it and +does its work while the other thread that also tries to acquire the lock after +the aforementioned thread has already acquired waits for the acquiring thread to +release the lock. + +Try to read the following code: + +```rs +pub fn acquire(&self) { + Interrupts::push_off(); + + let current_cpuid = mycpu().cid; + + // Prevent the exact same lock from deadlocking itself. + // If it is already locked on a single core, we'd spin forever. + if self.lock.load(Ordering::Relaxed) && self.holding_cpuid.get() == Some(current_cpuid) { + panic!("acquire({})", self.name); + } + + while self.lock.compare_exchange(false, + true, + Ordering::Acquire, + Ordering::Relaxed).is_err() { + core::hint::spin_loop(); + } + + self.holding_cpuid.set(Some(current_cpuid)); +} +``` + +As you can see, when another thread tries to acquire an already-acquired lock, +we just loop, or more accurately, "spin" to wait until the lock gets released. +So once the lock is released, the while condition will become false and the +calling thread will acquire the lock by the virtue of the compare-exchange +instruction right after. Now, spinning does waste a lot of CPU cycles, but in +some cases you need a spinlock and in many others, you would do well with +something called a "mutex", a topic for another chapter. + +Hopefully now a lot of locking makes sense. If you notice, we did not yet cover +the `push_off` and its counterpart `pop_off` which will be used in releasing the +lock, but we will cover it in the subsequent sections. + +### The `release` function + +Now, since a thread can acquire a lock to do some work on the shared resource, +it should also have a way to release such a lock. The `release` function does +exactly that: check if release is only called by the thread that initially +called acquire and then, set our boolean to false, indicating that the lock has +been released. You can now read the implementation: + +```rs +pub fn release(&self) { + let current_cpuid = mycpu().cid; + + if !self.lock.load(Ordering::Relaxed) { + panic!("release({})", self.name); + } + + if self.holding_cpuid.get() != Some(current_cpuid) { + panic!("Spinlock release error: ({}) owned by CPU {:?}, but CPU {} tried to release it!", + self.name, self.holding_cpuid.get(), current_cpuid); + } + + self.holding_cpuid.set(None); + self.lock.store(false, Ordering::Release); + + Interrupts::pop_off(); +} +``` + +## Interrupts, push_off and pop_off + +Now we first describe the need for enabling and disabling interrupts and then +discuss the missing piece: the push_off and pop_off calls. Suppose a timer +interrupt fires when you're in the midst of a critical section. The CPU +switches to the interrupt handler and now imagine the interrupt handler tries to +acquire the same lock. But since the lock was already acquired by the +now-interrupted thread, the interrupt handler will spin forever, waiting to +acquire that lock to do its thing and since the interrupted code can't continue +because of the interrupt and do its thing and release the lock, we are stuck in +the interrupt handler. That's a deadlock. So whenever you acquire a lock, you +must disable the interrupts. + +Okay, so why not just disable them the normal way? Why do we need 2 entirely +new functions for it? Suppose we have two functions (pseudocode): + +```c +void foo() +{ + interrupts_off(); + bar(); + interrupts_on(); +} +``` + +```c +void bar() +{ + interrupts_off(); + thing(); + interrupts_on(); +} +``` + +Looks harmless? Let's execute it line by line. + +```text +foo(): + interrupts_off(): + bar(): + interrupts_off() + ... + interrupts_on() <--- + interrupts_on(); +``` + +Interrupts are enabled when bar() exits and now while foo() was expecting +interrupts to be turned off, we introduced a point of failure: foo can be +interrupted by a timer interrupt. And in such a critical piece of software like +an Operating System, if something has even the slightest chance of going nuclear +and ruining progress, it is better to eliminate the problem. This one is a +fairly easy problem to fix. + +Look at this code in `kernel/processes.rs`: + +```rs +pub struct Cpu { + pub cid: usize, + pub current_pid: Option, // Tracking the running proc by id + + pub ncli: usize, // Depth of nested interrupt disabling on this CPU + pub interrupts_enabled: bool, // Were interrupts enabled BEFORE the very first push_off? +} +``` + +We only cover ncli and interrupts_enabled here. The other members hopefully +speak for themselves. + +Instead of just turning off interrupts, we can keep a counter, call it `ncli` to +tell us how nested we are into disabling interrupts. So every call to +`push_off` will increase ncli to indicate the number of levels we are nested +into and every call to `pop_off` will decrease the said counter. Member +`interrupts_enabled` keeps track of whether the interrupts were already enabled +at the very first push_off call so that we can later restore back to this state. +Here is a simple simulated example: + +Current state: +ncli = 0; +interrupts enabled + +Call push_off: + interrupts disabled + ncli = 1; +Call push_off: + ncli = 2; + +Now the unwinding + +Call pop_off: + ncli = 1; + +Call pop_off: + ncli = 0; + since interrupts were previously enabled, we restore the previous + interrupt state. + +We hope the example above explains the intent. Now, you are ready to look at +its implementation in `kernel/interrupts.rs`: + +```rs +// push_off and pop_off are like enabling and disabling interrupts but it +// doesn't just toggle blindly. Each push_off matches a pop_off. If +// interrupts were originally off, these functions keep them off. +pub fn push_off() { + let enabled = Self::irq_enabled(); + Self::irq_disable(); + + let c = mycpu(); + if c.ncli == 0 { + c.interrupts_enabled = enabled; + } + c.ncli += 1; +} + +pub fn pop_off() { + let c = mycpu(); + + if Self::irq_enabled() { + panic!("pop_off: interrupts active when they should be masked"); + } + + if c.ncli < 1 { + panic!("pop_off: nesting underflow!"); + } + + c.ncli -= 1; + if c.ncli == 0 && c.interrupts_enabled { + Self::irq_enable(); + } +} +``` + +These set of functions do exactly as described above, while also maintaining +checks to avoid misuse to a certain extent. + +# Conclusion + +With this, we have covered the fundamentals of concurrency, implemented a +working version of spinlocks and hopefully established the need for concurrency +in the first place. In the next chapter, we aim to cover a more sophisticated +implementation of mutexes. It is recommended that you are clear on concurrency +and scheduling before moving to mutexes as much of it involves those topics. +