Skip to content

Commit e29a709

Browse files
Muhammad Usama Anjum1Naim
authored andcommitted
mm/page_alloc: Optimize __free_contig_frozen_range()
Apply the same batch-freeing optimization from free_contig_range() to the frozen page path. The previous __free_contig_frozen_range() freed each order-0 page individually via free_frozen_pages(), which is slow for the same reason the old free_contig_range() was: each page goes to the order-0 pcp list rather than being coalesced into higher-order blocks. Rewrite __free_contig_frozen_range() to call free_pages_prepare() for each order-0 page, then batch the prepared pages into the largest possible power-of-2 aligned chunks via free_prepared_contig_range(). If free_pages_prepare() fails (e.g. HWPoison, bad page) the page is deliberately not freed; it should not be returned to the allocator. I've tested CMA through debugfs. The test allocates 16384 pages per allocation for several iterations. There is 3.5x improvement. Before: 1406 usec per iteration After: 402 usec per iteration Before: 70.89% 0.69% cma [kernel.kallsyms] [.] free_contig_frozen_range | |--70.20%--free_contig_frozen_range | | | |--46.41%--__free_frozen_pages | | | | | --36.18%--free_frozen_page_commit | | | | | --29.63%--_raw_spin_unlock_irqrestore | | | |--8.76%--_raw_spin_trylock | | | |--7.03%--__preempt_count_dec_and_test | | | |--4.57%--_raw_spin_unlock | | | |--1.96%--__get_pfnblock_flags_mask.isra.0 | | | --1.15%--free_frozen_page_commit | --0.69%--el0t_64_sync After: 23.57% 0.00% cma [kernel.kallsyms] [.] free_contig_frozen_range | ---free_contig_frozen_range | |--20.45%--__free_contig_frozen_range | | | |--17.77%--free_pages_prepare | | | --0.72%--free_prepared_contig_range | | | --0.55%--__free_frozen_pages | --3.12%--free_pages_prepare Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Suggested-by: Zi Yan <ziy@nvidia.com> Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com>
1 parent 199df20 commit e29a709

File tree

1 file changed

+1
-2
lines changed

1 file changed

+1
-2
lines changed

mm/page_alloc.c

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7094,8 +7094,7 @@ static int __alloc_contig_verify_gfp_mask(gfp_t gfp_mask, gfp_t *gfp_cc_mask)
70947094

70957095
static void __free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages)
70967096
{
7097-
for (; nr_pages--; pfn++)
7098-
free_frozen_pages(pfn_to_page(pfn), 0);
7097+
__free_contig_range_common(pfn, nr_pages, /* is_frozen= */ true);
70997098
}
71007099

71017100
/**

0 commit comments

Comments
 (0)