Skip to content

evere upload/create latency in Team Folders with large group membership #5119

Description

@mosi-kha

Bug description:

Uploading or creating a file inside a Team Folder is extremely slow when the folder is shared with a group that has a large membership (~10,000 users). The same operation (create/upload) in a user's home folder, or in a smaller-membership team folder, completes normally. PROPFIND against the same team folder is fast — only write operations (PUT, file/folder creation) are affected.

We suspect an N+1 query / unpaginated group-membership resolution pattern is triggered during the write-permission check for team folders backed by large groups, similar in shape to the documented IGroup::getUsers() unpaginated-load issue (30k+ queries / ~3s / ~200MB RAM for a 10k-member group lookup), but we have not yet isolated the exact call site.

Steps to reproduce:

  1. Create a group with ~10,000 members (or an equivalent-size representative test group)
  2. Create a Team Folder and assign that group with write access
  3. As a member of that group, create a new text file (or upload a small file) directly inside the team folder via the web UI or WebDAV PUT
  4. Observe request latency

Expected behavior:

File creation/upload in a team folder should complete in a time comparable to the same operation in a user's home folder, regardless of the size of the group(s) with access to that folder.

Actual behavior:

  • File creation/upload in the team folder takes 10+ seconds (observed via PHP-FPM request_slowlog_timeout — see trace note below)
  • The same operation in a user's home folder is unaffected
  • PROPFIND on the same team folder is fast; only writes are slow
  • A Symfony Profiler trace of the same request ran out of memory before completing, consistent with a large unbounded result set being loaded into memory rather than a simple slow query

Environment:

  • Nextcloud Server version: 33.0.7
  • Group Folders app version: 21.0.15
  • [PHP version] 8.3
  • [Database engine + version] postgres
  • [Web server] nginx + php-fpm
  • [User backend: LDAP / DB / other — note group size and backend type here, this may be relevant]
  • [Redis/APCu caching config if relevant] redis + APCu

Additional context:

  • We reviewed recent Group Folders performance work (v21.0.10: PROPFIND query reduction, per-request user-mapping caching; v21.0.14: userInMappings optimization with Circles) — these are already present in our version and do not appear to resolve this specific write-path case.
  • We're also tracking Missing index: oc_group_folders_groups #5009 (missing index on oc_group_folders_groups) as a possibly-related but distinct issue, and nextcloud/server#51483 (etag lock contention on the __groupfolders filecache root) as another candidate we've ruled out based on our profiling so far, since our symptom scales with group size rather than with concurrent write volume.
  • We have a PHP-FPM slowlog trace showing the write completing via a POST /ocs/v2.php sub-request taking 10.7s; happy to attach the full backtrace if useful — will follow up with it once we isolate the exact call path.
  • Available to help profile further or test a patch against our staging environment (~10k-member group is reproducible for us).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions