1.. _rcu_dereference_doc:
2
3PROPER CARE AND FEEDING OF RETURN VALUES FROM rcu_dereference()
4===============================================================
5
6Proper care and feeding of address and data dependencies is critically
7important to correct use of things like RCU.  To this end, the pointers
8returned from the rcu_dereference() family of primitives carry address and
9data dependencies.  These dependencies extend from the rcu_dereference()
10macro's load of the pointer to the later use of that pointer to compute
11either the address of a later memory access (representing an address
12dependency) or the value written by a later memory access (representing
13a data dependency).
14
15Most of the time, these dependencies are preserved, permitting you to
16freely use values from rcu_dereference().  For example, dereferencing
17(prefix "*"), field selection ("->"), assignment ("="), address-of
18("&"), casts, and addition or subtraction of constants all work quite
19naturally and safely.  However, because current compilers do not take
20either address or data dependencies into account it is still possible
21to get into trouble.
22
23Follow these rules to preserve the address and data dependencies emanating
24from your calls to rcu_dereference() and friends, thus keeping your RCU
25readers working properly:
26
27-	You must use one of the rcu_dereference() family of primitives
28	to load an RCU-protected pointer, otherwise CONFIG_PROVE_RCU
29	will complain.  Worse yet, your code can see random memory-corruption
30	bugs due to games that compilers and DEC Alpha can play.
31	Without one of the rcu_dereference() primitives, compilers
32	can reload the value, and won't your code have fun with two
33	different values for a single pointer!  Without rcu_dereference(),
34	DEC Alpha can load a pointer, dereference that pointer, and
35	return data preceding initialization that preceded the store
36	of the pointer.  (As noted later, in recent kernels READ_ONCE()
37	also prevents DEC Alpha from playing these tricks.)
38
39	In addition, the volatile cast in rcu_dereference() prevents the
40	compiler from deducing the resulting pointer value.  Please see
41	the section entitled "EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH"
42	for an example where the compiler can in fact deduce the exact
43	value of the pointer, and thus cause misordering.
44
45-	In the special case where data is added but is never removed
46	while readers are accessing the structure, READ_ONCE() may be used
47	instead of rcu_dereference().  In this case, use of READ_ONCE()
48	takes on the role of the lockless_dereference() primitive that
49	was removed in v4.15.
50
51-	You are only permitted to use rcu_dereference() on pointer values.
52	The compiler simply knows too much about integral values to
53	trust it to carry dependencies through integer operations.
54	There are a very few exceptions, namely that you can temporarily
55	cast the pointer to uintptr_t in order to:
56
57	-	Set bits and clear bits down in the must-be-zero low-order
58		bits of that pointer.  This clearly means that the pointer
59		must have alignment constraints, for example, this does
60		*not* work in general for char* pointers.
61
62	-	XOR bits to translate pointers, as is done in some
63		classic buddy-allocator algorithms.
64
65	It is important to cast the value back to pointer before
66	doing much of anything else with it.
67
68-	Avoid cancellation when using the "+" and "-" infix arithmetic
69	operators.  For example, for a given variable "x", avoid
70	"(x-(uintptr_t)x)" for char* pointers.	The compiler is within its
71	rights to substitute zero for this sort of expression, so that
72	subsequent accesses no longer depend on the rcu_dereference(),
73	again possibly resulting in bugs due to misordering.
74
75	Of course, if "p" is a pointer from rcu_dereference(), and "a"
76	and "b" are integers that happen to be equal, the expression
77	"p+a-b" is safe because its value still necessarily depends on
78	the rcu_dereference(), thus maintaining proper ordering.
79
80-	If you are using RCU to protect JITed functions, so that the
81	"()" function-invocation operator is applied to a value obtained
82	(directly or indirectly) from rcu_dereference(), you may need to
83	interact directly with the hardware to flush instruction caches.
84	This issue arises on some systems when a newly JITed function is
85	using the same memory that was used by an earlier JITed function.
86
87-	Do not use the results from relational operators ("==", "!=",
88	">", ">=", "<", or "<=") when dereferencing.  For example,
89	the following (quite strange) code is buggy::
90
91		int *p;
92		int *q;
93
94		...
95
96		p = rcu_dereference(gp)
97		q = &global_q;
98		q += p > &oom_p;
99		r1 = *q;  /* BUGGY!!! */
100
101	As before, the reason this is buggy is that relational operators
102	are often compiled using branches.  And as before, although
103	weak-memory machines such as ARM or PowerPC do order stores
104	after such branches, but can speculate loads, which can again
105	result in misordering bugs.
106
107-	Be very careful about comparing pointers obtained from
108	rcu_dereference() against non-NULL values.  As Linus Torvalds
109	explained, if the two pointers are equal, the compiler could
110	substitute the pointer you are comparing against for the pointer
111	obtained from rcu_dereference().  For example::
112
113		p = rcu_dereference(gp);
114		if (p == &default_struct)
115			do_default(p->a);
116
117	Because the compiler now knows that the value of "p" is exactly
118	the address of the variable "default_struct", it is free to
119	transform this code into the following::
120
121		p = rcu_dereference(gp);
122		if (p == &default_struct)
123			do_default(default_struct.a);
124
125	On ARM and Power hardware, the load from "default_struct.a"
126	can now be speculated, such that it might happen before the
127	rcu_dereference().  This could result in bugs due to misordering.
128
129	However, comparisons are OK in the following cases:
130
131	-	The comparison was against the NULL pointer.  If the
132		compiler knows that the pointer is NULL, you had better
133		not be dereferencing it anyway.  If the comparison is
134		non-equal, the compiler is none the wiser.  Therefore,
135		it is safe to compare pointers from rcu_dereference()
136		against NULL pointers.
137
138	-	The pointer is never dereferenced after being compared.
139		Since there are no subsequent dereferences, the compiler
140		cannot use anything it learned from the comparison
141		to reorder the non-existent subsequent dereferences.
142		This sort of comparison occurs frequently when scanning
143		RCU-protected circular linked lists.
144
145		Note that if the pointer comparison is done outside
146		of an RCU read-side critical section, and the pointer
147		is never dereferenced, rcu_access_pointer() should be
148		used in place of rcu_dereference().  In most cases,
149		it is best to avoid accidental dereferences by testing
150		the rcu_access_pointer() return value directly, without
151		assigning it to a variable.
152
153		Within an RCU read-side critical section, there is little
154		reason to use rcu_access_pointer().
155
156	-	The comparison is against a pointer that references memory
157		that was initialized "a long time ago."  The reason
158		this is safe is that even if misordering occurs, the
159		misordering will not affect the accesses that follow
160		the comparison.  So exactly how long ago is "a long
161		time ago"?  Here are some possibilities:
162
163		-	Compile time.
164
165		-	Boot time.
166
167		-	Module-init time for module code.
168
169		-	Prior to kthread creation for kthread code.
170
171		-	During some prior acquisition of the lock that
172			we now hold.
173
174		-	Before mod_timer() time for a timer handler.
175
176		There are many other possibilities involving the Linux
177		kernel's wide array of primitives that cause code to
178		be invoked at a later time.
179
180	-	The pointer being compared against also came from
181		rcu_dereference().  In this case, both pointers depend
182		on one rcu_dereference() or another, so you get proper
183		ordering either way.
184
185		That said, this situation can make certain RCU usage
186		bugs more likely to happen.  Which can be a good thing,
187		at least if they happen during testing.  An example
188		of such an RCU usage bug is shown in the section titled
189		"EXAMPLE OF AMPLIFIED RCU-USAGE BUG".
190
191	-	All of the accesses following the comparison are stores,
192		so that a control dependency preserves the needed ordering.
193		That said, it is easy to get control dependencies wrong.
194		Please see the "CONTROL DEPENDENCIES" section of
195		Documentation/memory-barriers.txt for more details.
196
197	-	The pointers are not equal *and* the compiler does
198		not have enough information to deduce the value of the
199		pointer.  Note that the volatile cast in rcu_dereference()
200		will normally prevent the compiler from knowing too much.
201
202		However, please note that if the compiler knows that the
203		pointer takes on only one of two values, a not-equal
204		comparison will provide exactly the information that the
205		compiler needs to deduce the value of the pointer.
206
207-	Disable any value-speculation optimizations that your compiler
208	might provide, especially if you are making use of feedback-based
209	optimizations that take data collected from prior runs.  Such
210	value-speculation optimizations reorder operations by design.
211
212	There is one exception to this rule:  Value-speculation
213	optimizations that leverage the branch-prediction hardware are
214	safe on strongly ordered systems (such as x86), but not on weakly
215	ordered systems (such as ARM or Power).  Choose your compiler
216	command-line options wisely!
217
218
219EXAMPLE OF AMPLIFIED RCU-USAGE BUG
220----------------------------------
221
222Because updaters can run concurrently with RCU readers, RCU readers can
223see stale and/or inconsistent values.  If RCU readers need fresh or
224consistent values, which they sometimes do, they need to take proper
225precautions.  To see this, consider the following code fragment::
226
227	struct foo {
228		int a;
229		int b;
230		int c;
231	};
232	struct foo *gp1;
233	struct foo *gp2;
234
235	void updater(void)
236	{
237		struct foo *p;
238
239		p = kmalloc(...);
240		if (p == NULL)
241			deal_with_it();
242		p->a = 42;  /* Each field in its own cache line. */
243		p->b = 43;
244		p->c = 44;
245		rcu_assign_pointer(gp1, p);
246		p->b = 143;
247		p->c = 144;
248		rcu_assign_pointer(gp2, p);
249	}
250
251	void reader(void)
252	{
253		struct foo *p;
254		struct foo *q;
255		int r1, r2;
256
257		rcu_read_lock();
258		p = rcu_dereference(gp2);
259		if (p == NULL)
260			return;
261		r1 = p->b;  /* Guaranteed to get 143. */
262		q = rcu_dereference(gp1);  /* Guaranteed non-NULL. */
263		if (p == q) {
264			/* The compiler decides that q->c is same as p->c. */
265			r2 = p->c; /* Could get 44 on weakly order system. */
266		} else {
267			r2 = p->c - r1; /* Unconditional access to p->c. */
268		}
269		rcu_read_unlock();
270		do_something_with(r1, r2);
271	}
272
273You might be surprised that the outcome (r1 == 143 && r2 == 44) is possible,
274but you should not be.  After all, the updater might have been invoked
275a second time between the time reader() loaded into "r1" and the time
276that it loaded into "r2".  The fact that this same result can occur due
277to some reordering from the compiler and CPUs is beside the point.
278
279But suppose that the reader needs a consistent view?
280
281Then one approach is to use locking, for example, as follows::
282
283	struct foo {
284		int a;
285		int b;
286		int c;
287		spinlock_t lock;
288	};
289	struct foo *gp1;
290	struct foo *gp2;
291
292	void updater(void)
293	{
294		struct foo *p;
295
296		p = kmalloc(...);
297		if (p == NULL)
298			deal_with_it();
299		spin_lock(&p->lock);
300		p->a = 42;  /* Each field in its own cache line. */
301		p->b = 43;
302		p->c = 44;
303		spin_unlock(&p->lock);
304		rcu_assign_pointer(gp1, p);
305		spin_lock(&p->lock);
306		p->b = 143;
307		p->c = 144;
308		spin_unlock(&p->lock);
309		rcu_assign_pointer(gp2, p);
310	}
311
312	void reader(void)
313	{
314		struct foo *p;
315		struct foo *q;
316		int r1, r2;
317
318		rcu_read_lock();
319		p = rcu_dereference(gp2);
320		if (p == NULL)
321			return;
322		spin_lock(&p->lock);
323		r1 = p->b;  /* Guaranteed to get 143. */
324		q = rcu_dereference(gp1);  /* Guaranteed non-NULL. */
325		if (p == q) {
326			/* The compiler decides that q->c is same as p->c. */
327			r2 = p->c; /* Locking guarantees r2 == 144. */
328		} else {
329			spin_lock(&q->lock);
330			r2 = q->c - r1;
331			spin_unlock(&q->lock);
332		}
333		rcu_read_unlock();
334		spin_unlock(&p->lock);
335		do_something_with(r1, r2);
336	}
337
338As always, use the right tool for the job!
339
340
341EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH
342-----------------------------------------
343
344If a pointer obtained from rcu_dereference() compares not-equal to some
345other pointer, the compiler normally has no clue what the value of the
346first pointer might be.  This lack of knowledge prevents the compiler
347from carrying out optimizations that otherwise might destroy the ordering
348guarantees that RCU depends on.  And the volatile cast in rcu_dereference()
349should prevent the compiler from guessing the value.
350
351But without rcu_dereference(), the compiler knows more than you might
352expect.  Consider the following code fragment::
353
354	struct foo {
355		int a;
356		int b;
357	};
358	static struct foo variable1;
359	static struct foo variable2;
360	static struct foo *gp = &variable1;
361
362	void updater(void)
363	{
364		initialize_foo(&variable2);
365		rcu_assign_pointer(gp, &variable2);
366		/*
367		 * The above is the only store to gp in this translation unit,
368		 * and the address of gp is not exported in any way.
369		 */
370	}
371
372	int reader(void)
373	{
374		struct foo *p;
375
376		p = gp;
377		barrier();
378		if (p == &variable1)
379			return p->a; /* Must be variable1.a. */
380		else
381			return p->b; /* Must be variable2.b. */
382	}
383
384Because the compiler can see all stores to "gp", it knows that the only
385possible values of "gp" are "variable1" on the one hand and "variable2"
386on the other.  The comparison in reader() therefore tells the compiler
387the exact value of "p" even in the not-equals case.  This allows the
388compiler to make the return values independent of the load from "gp",
389in turn destroying the ordering between this load and the loads of the
390return values.  This can result in "p->b" returning pre-initialization
391garbage values on weakly ordered systems.
392
393In short, rcu_dereference() is *not* optional when you are going to
394dereference the resulting pointer.
395
396
397WHICH MEMBER OF THE rcu_dereference() FAMILY SHOULD YOU USE?
398------------------------------------------------------------
399
400First, please avoid using rcu_dereference_raw() and also please avoid
401using rcu_dereference_check() and rcu_dereference_protected() with a
402second argument with a constant value of 1 (or true, for that matter).
403With that caution out of the way, here is some guidance for which
404member of the rcu_dereference() to use in various situations:
405
4061.	If the access needs to be within an RCU read-side critical
407	section, use rcu_dereference().  With the new consolidated
408	RCU flavors, an RCU read-side critical section is entered
409	using rcu_read_lock(), anything that disables bottom halves,
410	anything that disables interrupts, or anything that disables
411	preemption.  Please note that spinlock critical sections
412	are also implied RCU read-side critical sections, even when
413	they are preemptible, as they are in kernels built with
414	CONFIG_PREEMPT_RT=y.
415
4162.	If the access might be within an RCU read-side critical section
417	on the one hand, or protected by (say) my_lock on the other,
418	use rcu_dereference_check(), for example::
419
420		p1 = rcu_dereference_check(p->rcu_protected_pointer,
421					   lockdep_is_held(&my_lock));
422
423
4243.	If the access might be within an RCU read-side critical section
425	on the one hand, or protected by either my_lock or your_lock on
426	the other, again use rcu_dereference_check(), for example::
427
428		p1 = rcu_dereference_check(p->rcu_protected_pointer,
429					   lockdep_is_held(&my_lock) ||
430					   lockdep_is_held(&your_lock));
431
4324.	If the access is on the update side, so that it is always protected
433	by my_lock, use rcu_dereference_protected()::
434
435		p1 = rcu_dereference_protected(p->rcu_protected_pointer,
436					       lockdep_is_held(&my_lock));
437
438	This can be extended to handle multiple locks as in #3 above,
439	and both can be extended to check other conditions as well.
440
4415.	If the protection is supplied by the caller, and is thus unknown
442	to this code, that is the rare case when rcu_dereference_raw()
443	is appropriate.  In addition, rcu_dereference_raw() might be
444	appropriate when the lockdep expression would be excessively
445	complex, except that a better approach in that case might be to
446	take a long hard look at your synchronization design.  Still,
447	there are data-locking cases where any one of a very large number
448	of locks or reference counters suffices to protect the pointer,
449	so rcu_dereference_raw() does have its place.
450
451	However, its place is probably quite a bit smaller than one
452	might expect given the number of uses in the current kernel.
453	Ditto for its synonym, rcu_dereference_check( ... , 1), and
454	its close relative, rcu_dereference_protected(... , 1).
455
456
457SPARSE CHECKING OF RCU-PROTECTED POINTERS
458-----------------------------------------
459
460The sparse static-analysis tool checks for non-RCU access to RCU-protected
461pointers, which can result in "interesting" bugs due to compiler
462optimizations involving invented loads and perhaps also load tearing.
463For example, suppose someone mistakenly does something like this::
464
465	p = q->rcu_protected_pointer;
466	do_something_with(p->a);
467	do_something_else_with(p->b);
468
469If register pressure is high, the compiler might optimize "p" out
470of existence, transforming the code to something like this::
471
472	do_something_with(q->rcu_protected_pointer->a);
473	do_something_else_with(q->rcu_protected_pointer->b);
474
475This could fatally disappoint your code if q->rcu_protected_pointer
476changed in the meantime.  Nor is this a theoretical problem:  Exactly
477this sort of bug cost Paul E. McKenney (and several of his innocent
478colleagues) a three-day weekend back in the early 1990s.
479
480Load tearing could of course result in dereferencing a mashup of a pair
481of pointers, which also might fatally disappoint your code.
482
483These problems could have been avoided simply by making the code instead
484read as follows::
485
486	p = rcu_dereference(q->rcu_protected_pointer);
487	do_something_with(p->a);
488	do_something_else_with(p->b);
489
490Unfortunately, these sorts of bugs can be extremely hard to spot during
491review.  This is where the sparse tool comes into play, along with the
492"__rcu" marker.  If you mark a pointer declaration, whether in a structure
493or as a formal parameter, with "__rcu", which tells sparse to complain if
494this pointer is accessed directly.  It will also cause sparse to complain
495if a pointer not marked with "__rcu" is accessed using rcu_dereference()
496and friends.  For example, ->rcu_protected_pointer might be declared as
497follows::
498
499	struct foo __rcu *rcu_protected_pointer;
500
501Use of "__rcu" is opt-in.  If you choose not to use it, then you should
502ignore the sparse warnings.
503