mm/gup: cache p4d in follow_p4d_mask()

Add a variable to cache p4d in follow_p4d_mask().  It's a good practise to
make sure all the following checks will have a consistent view of the
entry.

Link: https://lkml.kernel.org/r/20240318200404.448346-3-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Andreas Larsson <andreas@gaisler.com>
Cc: "Aneesh Kumar K.V" <aneesh.kumar@kernel.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Bjorn Andersson <andersson@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Fabio Estevam <festevam@denx.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Konrad Dybcio <konrad.dybcio@linaro.org>
Cc: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Cc: Lucas Stach <l.stach@pengutronix.de>
Cc: Mark Salter <msalter@redhat.com>
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Mike Rapoport (IBM) <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: "Naveen N. Rao" <naveen.n.rao@linux.ibm.com>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Shawn Guo <shawnguo@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/mm/gup.c b/mm/gup.c
index 1611e73b1121b1b9031356ccac6b765c60327ba1..19f6a458683d45a312afdc783ecdb38f5740fd1f 100644 (file)
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -772,16 +772,17 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma,
                                    unsigned int flags,
                                    struct follow_page_context *ctx)
 {
-       p4d_t *p4d;
+       p4d_t *p4dp, p4d;
 
-       p4d = p4d_offset(pgdp, address);
-       if (p4d_none(*p4d))
+       p4dp = p4d_offset(pgdp, address);
+       p4d = READ_ONCE(*p4dp);
+       if (p4d_none(p4d))
                return no_page_table(vma, flags);
-       BUILD_BUG_ON(p4d_huge(*p4d));
-       if (unlikely(p4d_bad(*p4d)))
+       BUILD_BUG_ON(p4d_huge(p4d));
+       if (unlikely(p4d_bad(p4d)))
                return no_page_table(vma, flags);
 
-       return follow_pud_mask(vma, address, p4d, flags, ctx);
+       return follow_pud_mask(vma, address, p4dp, flags, ctx);
 }
 
 /**