1169695Skan==================================================================== 2169695SkanInteraction of Suspend code (S3) with the CPU hotplug infrastructure 3169695Skan==================================================================== 4169695Skan 5169695Skan(C) 2011 - 2014 Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com> 6169695Skan 7169695Skan 8169695SkanI. Differences between CPU hotplug and Suspend-to-RAM 9169695Skan====================================================== 10169695Skan 11169695SkanHow does the regular CPU hotplug code differ from how the Suspend-to-RAM 12169695Skaninfrastructure uses it internally? And where do they share common code? 13169695Skan 14169695SkanWell, a picture is worth a thousand words... So ASCII art follows :-) 15169695Skan 16169695Skan[This depicts the current design in the kernel, and focusses only on the 17169695Skaninteractions involving the freezer and CPU hotplug and also tries to explain 18169695Skanthe locking involved. It outlines the notifications involved as well. 19169695SkanBut please note that here, only the call paths are illustrated, with the aim 20169695Skanof describing where they take different paths and where they share code. 21169695SkanWhat happens when regular CPU hotplug and Suspend-to-RAM race with each other 22169695Skanis not depicted here.] 23169695Skan 24169695SkanOn a high level, the suspend-resume cycle goes like this:: 25169695Skan 26169695Skan |Freeze| -> |Disable nonboot| -> |Do suspend| -> |Enable nonboot| -> |Thaw | 27169695Skan |tasks | | cpus | | | | cpus | |tasks| 28169695Skan 29169695Skan 30169695SkanMore details follow:: 31169695Skan 32169695Skan Suspend call path 33169695Skan ----------------- 34169695Skan 35169695Skan Write 'mem' to 36169695Skan /sys/power/state 37169695Skan sysfs file 38169695Skan | 39169695Skan v 40169695Skan Acquire system_transition_mutex lock 41169695Skan | 42169695Skan v 43169695Skan Send PM_SUSPEND_PREPARE 44169695Skan notifications 45169695Skan | 46169695Skan v 47169695Skan Freeze tasks 48169695Skan | 49169695Skan | 50169695Skan v 51169695Skan freeze_secondary_cpus() 52169695Skan /* start */ 53169695Skan | 54169695Skan v 55169695Skan Acquire cpu_add_remove_lock 56169695Skan | 57169695Skan v 58169695Skan Iterate over CURRENTLY 59169695Skan online CPUs 60169695Skan | 61169695Skan | 62169695Skan | ---------- 63169695Skan v | L 64169695Skan ======> _cpu_down() | 65169695Skan | [This takes cpuhotplug.lock | 66169695Skan Common | before taking down the CPU | 67169695Skan code | and releases it when done] | O 68169695Skan | While it is at it, notifications | 69169695Skan | are sent when notable events occur, | 70169695Skan ======> by running all registered callbacks. | 71169695Skan | | O 72169695Skan | | 73169695Skan | | 74169695Skan v | 75169695Skan Note down these cpus in | P 76169695Skan frozen_cpus mask ---------- 77169695Skan | 78169695Skan v 79169695Skan Disable regular cpu hotplug 80169695Skan by increasing cpu_hotplug_disabled 81169695Skan | 82169695Skan v 83169695Skan Release cpu_add_remove_lock 84169695Skan | 85169695Skan v 86169695Skan /* freeze_secondary_cpus() complete */ 87169695Skan | 88169695Skan v 89169695Skan Do suspend 90169695Skan 91169695Skan 92169695Skan 93169695SkanResuming back is likewise, with the counterparts being (in the order of 94169695Skanexecution during resume): 95169695Skan 96169695Skan* thaw_secondary_cpus() which involves:: 97169695Skan 98169695Skan | Acquire cpu_add_remove_lock 99169695Skan | Decrease cpu_hotplug_disabled, thereby enabling regular cpu hotplug 100169695Skan | Call _cpu_up() [for all those cpus in the frozen_cpus mask, in a loop] 101169695Skan | Release cpu_add_remove_lock 102169695Skan v 103169695Skan 104* thaw tasks 105* send PM_POST_SUSPEND notifications 106* Release system_transition_mutex lock. 107 108 109It is to be noted here that the system_transition_mutex lock is acquired at the 110very beginning, when we are just starting out to suspend, and then released only 111after the entire cycle is complete (i.e., suspend + resume). 112 113:: 114 115 116 117 Regular CPU hotplug call path 118 ----------------------------- 119 120 Write 0 (or 1) to 121 /sys/devices/system/cpu/cpu*/online 122 sysfs file 123 | 124 | 125 v 126 cpu_down() 127 | 128 v 129 Acquire cpu_add_remove_lock 130 | 131 v 132 If cpu_hotplug_disabled > 0 133 return gracefully 134 | 135 | 136 v 137 ======> _cpu_down() 138 | [This takes cpuhotplug.lock 139 Common | before taking down the CPU 140 code | and releases it when done] 141 | While it is at it, notifications 142 | are sent when notable events occur, 143 ======> by running all registered callbacks. 144 | 145 | 146 v 147 Release cpu_add_remove_lock 148 [That's it!, for 149 regular CPU hotplug] 150 151 152 153So, as can be seen from the two diagrams (the parts marked as "Common code"), 154regular CPU hotplug and the suspend code path converge at the _cpu_down() and 155_cpu_up() functions. They differ in the arguments passed to these functions, 156in that during regular CPU hotplug, 0 is passed for the 'tasks_frozen' 157argument. But during suspend, since the tasks are already frozen by the time 158the non-boot CPUs are offlined or onlined, the _cpu_*() functions are called 159with the 'tasks_frozen' argument set to 1. 160[See below for some known issues regarding this.] 161 162 163Important files and functions/entry points: 164------------------------------------------- 165 166- kernel/power/process.c : freeze_processes(), thaw_processes() 167- kernel/power/suspend.c : suspend_prepare(), suspend_enter(), suspend_finish() 168- kernel/cpu.c: cpu_[up|down](), _cpu_[up|down](), 169 [disable|enable]_nonboot_cpus() 170 171 172 173II. What are the issues involved in CPU hotplug? 174------------------------------------------------ 175 176There are some interesting situations involving CPU hotplug and microcode 177update on the CPUs, as discussed below: 178 179[Please bear in mind that the kernel requests the microcode images from 180userspace, using the request_firmware() function defined in 181drivers/base/firmware_loader/main.c] 182 183 184a. When all the CPUs are identical: 185 186 This is the most common situation and it is quite straightforward: we want 187 to apply the same microcode revision to each of the CPUs. 188 To give an example of x86, the collect_cpu_info() function defined in 189 arch/x86/kernel/microcode_core.c helps in discovering the type of the CPU 190 and thereby in applying the correct microcode revision to it. 191 But note that the kernel does not maintain a common microcode image for the 192 all CPUs, in order to handle case 'b' described below. 193 194 195b. When some of the CPUs are different than the rest: 196 197 In this case since we probably need to apply different microcode revisions 198 to different CPUs, the kernel maintains a copy of the correct microcode 199 image for each CPU (after appropriate CPU type/model discovery using 200 functions such as collect_cpu_info()). 201 202 203c. When a CPU is physically hot-unplugged and a new (and possibly different 204 type of) CPU is hot-plugged into the system: 205 206 In the current design of the kernel, whenever a CPU is taken offline during 207 a regular CPU hotplug operation, upon receiving the CPU_DEAD notification 208 (which is sent by the CPU hotplug code), the microcode update driver's 209 callback for that event reacts by freeing the kernel's copy of the 210 microcode image for that CPU. 211 212 Hence, when a new CPU is brought online, since the kernel finds that it 213 doesn't have the microcode image, it does the CPU type/model discovery 214 afresh and then requests the userspace for the appropriate microcode image 215 for that CPU, which is subsequently applied. 216 217 For example, in x86, the mc_cpu_callback() function (which is the microcode 218 update driver's callback registered for CPU hotplug events) calls 219 microcode_update_cpu() which would call microcode_init_cpu() in this case, 220 instead of microcode_resume_cpu() when it finds that the kernel doesn't 221 have a valid microcode image. This ensures that the CPU type/model 222 discovery is performed and the right microcode is applied to the CPU after 223 getting it from userspace. 224 225 226d. Handling microcode update during suspend/hibernate: 227 228 Strictly speaking, during a CPU hotplug operation which does not involve 229 physically removing or inserting CPUs, the CPUs are not actually powered 230 off during a CPU offline. They are just put to the lowest C-states possible. 231 Hence, in such a case, it is not really necessary to re-apply microcode 232 when the CPUs are brought back online, since they wouldn't have lost the 233 image during the CPU offline operation. 234 235 This is the usual scenario encountered during a resume after a suspend. 236 However, in the case of hibernation, since all the CPUs are completely 237 powered off, during restore it becomes necessary to apply the microcode 238 images to all the CPUs. 239 240 [Note that we don't expect someone to physically pull out nodes and insert 241 nodes with a different type of CPUs in-between a suspend-resume or a 242 hibernate/restore cycle.] 243 244 In the current design of the kernel however, during a CPU offline operation 245 as part of the suspend/hibernate cycle (cpuhp_tasks_frozen is set), 246 the existing copy of microcode image in the kernel is not freed up. 247 And during the CPU online operations (during resume/restore), since the 248 kernel finds that it already has copies of the microcode images for all the 249 CPUs, it just applies them to the CPUs, avoiding any re-discovery of CPU 250 type/model and the need for validating whether the microcode revisions are 251 right for the CPUs or not (due to the above assumption that physical CPU 252 hotplug will not be done in-between suspend/resume or hibernate/restore 253 cycles). 254 255 256III. Known problems 257=================== 258 259Are there any known problems when regular CPU hotplug and suspend race 260with each other? 261 262Yes, they are listed below: 263 2641. When invoking regular CPU hotplug, the 'tasks_frozen' argument passed to 265 the _cpu_down() and _cpu_up() functions is *always* 0. 266 This might not reflect the true current state of the system, since the 267 tasks could have been frozen by an out-of-band event such as a suspend 268 operation in progress. Hence, the cpuhp_tasks_frozen variable will not 269 reflect the frozen state and the CPU hotplug callbacks which evaluate 270 that variable might execute the wrong code path. 271 2722. If a regular CPU hotplug stress test happens to race with the freezer due 273 to a suspend operation in progress at the same time, then we could hit the 274 situation described below: 275 276 * A regular cpu online operation continues its journey from userspace 277 into the kernel, since the freezing has not yet begun. 278 * Then freezer gets to work and freezes userspace. 279 * If cpu online has not yet completed the microcode update stuff by now, 280 it will now start waiting on the frozen userspace in the 281 TASK_UNINTERRUPTIBLE state, in order to get the microcode image. 282 * Now the freezer continues and tries to freeze the remaining tasks. But 283 due to this wait mentioned above, the freezer won't be able to freeze 284 the cpu online hotplug task and hence freezing of tasks fails. 285 286 As a result of this task freezing failure, the suspend operation gets 287 aborted. 288