Improve performance of listTemplates API#13566
Conversation
* added bypass logic to create template_pair from 6 tables, fallback for hard filters * Removed batching, checking against per pair * follow try-with-resources design * unit tests for the bypass logic * Fix cross-zone template lookup in listTemplates Phase 2 When temp_zone_pair has dcId=0 (cross-zone template with no data_center), use IS NULL predicate instead of broken EQ/IN with literal 0 which never matches NULL rows. * Add filter for non-root domain-admin users --------- Co-authored-by: anishadas <adas33@apple.com> Co-authored-by: Aaron Chung <aaron_chung@apple.com>
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## 4.22 #13566 +/- ##
=============================================
- Coverage 17.67% 3.69% -13.98%
=============================================
Files 5923 448 -5475
Lines 533349 38117 -495232
Branches 65248 7067 -58181
=============================================
- Hits 94253 1409 -92844
+ Misses 428437 36521 -391916
+ Partials 10659 187 -10472
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ove-listtemplates-perf
|
@blueorangutan package |
|
@vladimirpetrov a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18552 |
|
thanks @Pearl1594 , do you have any performance figures? (before vs. after?) |
Description
This PR Adds a fast-path for listTemplates Phase 1 that bypasses template_view,
behind a runtime config flag (default off).
Why
The captured incident query against template_view runs 7,000+ seconds and
drains the DB pool. View materializes ~71k rows for ~4.7k templates due to
LEFT-JOIN row multiplication, OR-join on data_center, and predicates on a
computed column. Phase 1 only needs 6 of the 13 tables.
What changed
BypassTemplateViewConfigKey (template.list.bypass.view, defaultfalse,Global, runtime-toggleable).
TemplateListFilterPOJO +canBypass()predicate.TemplateJoinDao.findDistinctTempZonePairs(filter)— hand-tuned SQL overvm_template + account + template_store_ref + image_store + template_zone_ref
COALESCE.QueryManagerImplbuilds the filter and dispatches to bypass when flag ison AND
canBypass()is true. Otherwise falls through to existing path.Coverage
Bypass handles:
id,name,keyword,hypervisor,format,type,public,featured,bootable,parenttemplateid,accountType,accountIdIN,zoneid,templateState,removed,onlyReady, pagination,templatefilter∈ {self/selfexecutablenon-domain-admin,executable,alladmin}.Falls back to view path:
tags,templatefilter∈ {featured,community,sharedexecutable,shared,allnon-admin,self/selfexecutableforDOMAIN_ADMIN/RESOURCE_DOMAIN_ADMIN}.
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
How did you try to break this feature and the system with this change?