[pull] master from rclone:master - #238
Open
pull[bot] wants to merge 4680 commits into
Open
Conversation
setAttr left attr.Ino unset (0) and the NewInode sites left StableAttr.Ino unset, so the kernel saw inode 0 while the node identity differed, which breaks NFS file-handle validation. Set both to the stable VFS inode. The bazil cmd/mount backend does not hit this because its framework assigns stable inodes automatically; go-fuse needs them set explicitly. Before: chmod/chown/truncate on a just-written file through an NFS-exported mount2 mount failed with ESTALE. After: they succeed. Exercising the NFS handle-validation path needs a kernel NFS server, so it isn't covered by the local vfstest harness; validated against a real Linux nfs-kernel-server export over NFSv3, NFSv4.0 and NFSv4.2. #9547
… offsets Seekdir handled only a rewind to offset 0 and returned ENOTSUP otherwise. The stateless kernel NFS server opens a fresh directory handle and seeks to the last returned cookie on every readdir continuation, so any listing spanning more than one readdir batch failed over NFS. dirStream is a snapshot taken at Readdir time and go-fuse assigns each entry a sequential offset, so seeking to off positions the stream at index off; off == 0 still resets to the start, preserving the rewind/re-read behaviour. Before: ls of a directory that spans more than one readdir batch failed over NFS with "Unknown error 524". After: it lists correctly. Adds TestDirStreamSeekdir covering rewind, mid-stream resume and the EOF clamp. The full NFS path was validated against a real Linux nfs-kernel-server export over NFSv3, NFSv4.0 and NFSv4.2. Fixes #9547
Mknod is now implemented by mount, mount2 and cmount, so exercise it from the shared vfstest suite: create a regular file (S_IFREG) through the mounted path and check it reads back as a 0-byte regular file. This is the path the kernel NFS server drives when a client creates a file over an exported mount. Before this there was no shared coverage for Mknod; it now runs against each backend via RunTests. Suggested in #9548. Signed-off-by: Sandy Luppino <s.luppino@opendrives.com>
…jobs NewStatsGroup started the averageLoop goroutine unconditionally at group creation. In an rcd daemon driven by many short rc sync/move calls (a common pattern for scheduled spool flushes), each call gets a fresh job/N stats group. When such a job transferred zero files the loop was never stopped, because _stopAverageLoop is only reached via DoneTransferring once transferring and checking both go from non-empty back to empty, which never happens if nothing was ever transferring in the first place. The result was one leaked goroutine per rc call, growing unbounded until the daemon was OOM-killed (reported: ~61k goroutines and ~640 MB RSS after ~7 days from a per-minute timer over 6 mappings). This is the same class of leak as #8571, which fixed the equivalent auto-start in NewStats. Fix it the same way: do not start the average loop at group creation. NewTransfer and NewTransferRemoteSize already call startAverageLoop when real transfer activity begins, and DoneTransferring already stops it when the last transfer completes, so on-demand behaviour is unchanged for groups that actually do work. Groups that never transfer anything now cost zero goroutines. Adds a regression test that fails without the fix. Fixes #9567
Signed-off-by: blackflytech <blackflytech@outlook.com>
A Range header requesting a suffix longer than the object (e.g. "bytes=-90407" against a 5 byte object) caused RangeOption.Decode to compute a negative offset (size - End), which serve.Object then used directly as a slice/seek offset and panicked with "slice bounds out of range". FixRangeOption (used by backends like OneDrive/Box that lack native suffix-range support) had the same root cause: it produced a RangeOption with a negative Start, which Header() silently dropped, turning the request into the wrong byte range instead of erroring or serving the whole object. Per RFC 7233 section 2.1, when the suffix-length exceeds the representation size, the entire representation should be served. Clamp the computed offset/start to 0 in both places. Fixes #6310
Per RFC 4918 section 10.6, when the Overwrite header is omitted from a COPY or MOVE request the resource MUST treat the request as if Overwrite: T had been sent. The upstream golang.org/x/net/webdav library mishandles this for MOVE by checking == "T" instead of != "F", so an absent header is treated as Overwrite: F and the request fails with 412 Precondition Failed. Normalise the header to T in the rclone WebDAV server before delegating to the upstream handler when the client did not send one. This restores RFC-compliant default behaviour and can be removed once the upstream fix in golang/go#66059 lands and the golang.org/x/net dependency is bumped. Fixes #9496
Zoho WorkDrive now sends a Retry-After header on 429 (it did not when the backend was written). Waiting the hard-coded 60s retried too early when the server asked for more (Retry-After: 299 is common) and the penalty escalated (observed 84s -> 239s). Honour the header plus a 1s margin - retrying at exactly Retry-After still finds an empty token bucket and burns ~16 immediate 429s - and keep 60s as the fallback when the header is absent. shouldRetry becomes a method on *Fs so the retry decision has access to the remote's state; later commits build on this. See #9570
Zoho throttling is account/plan-dependent; measurements show a sustainable listing rate of ~6 requests/s on a production account - going faster drains a token bucket and stalls ~2 minutes per Retry-After, which is strictly slower overall. Add per-remote pacer options (default 6/1) using the same token-bucket pacer as the Google Drive backend. Set --zoho-tpslimit 0 to disable the cap. See #9570
A 429 stall was only visible as a DEBUG pacer line, so without -vv rclone appeared to hang for 2-5 minutes. In one night's batch logs 17 job starts produced only 4 completions because the silent stalls looked like hangs and the jobs kept getting killed, re-triggering the throttle. Log the first 429 of each throttle episode at NOTICE with the server message and the wait time. An episode ends when a request succeeds after the penalty window; retries within an episode stay at DEBUG via the existing pacer logging. State is two atomics behind a pointer on Fs, so shallow Fs copies share it and concurrent checkers are safe. See #9570
WorkDrive throttles its listing API (GET files/{id}/files) PER
folder, independently of the overall request rate: at most ~19
listings of one folder are allowed in any rolling ~60s window and
the 20th returns F7008 with a ~300s Retry-After (measured live -
every observed trip landed exactly on the 20th listing inside a
window). fstests re-lists the same working directory after almost
every sub-operation, which is why the integration suite could not
pass.
Add a per-folder listing limiter with a true per-window cap: each
window starts with --zoho-list-folder-burst listings passing
back-to-back (the burst re-arms at every window boundary, so a
sync re-listing one directory a few times never waits), the rest
of the budget is spaced evenly across the window, and a sliding
log of recent listings guarantees no rolling window ever exceeds
--zoho-list-folder-limit (default 19) per --zoho-list-folder-window
(default 60s) for any traffic pattern. The registry is
process-wide and keyed by region+folder id so every Fs instance
shares one budget per physical folder; idle entries are evicted
after a window, which is lossless because Zoho's window has also
cleared by then.
Defaults were validated against the live service: bursts of 4-6
under the 19-per-60s cap ran clean while an over-cap probe tripped
F7008 exactly at the 20th listing, and a full test_all -backends
zoho run passes cleanly.
Fixes #9570
The --disable-zip flag was registered manually and was missing from OptionsInfo, so it could not be set over the rc interface. Move it into OptionsInfo like serve webdav does, which keeps the command line flag and also makes it settable via rc.
The shared Google Drive and Google Photos client_id is being retired and will stop working during 2026. Warn users who rely on it (ie who have not configured their own client_id) so they can create their own in advance. The warning is only shown for auth flows that actually use the shared client_id, not for service account, environment or anonymous auth. See: https://forum.rclone.org/t/google-drive-and-google-photos-users-action-required/54005
…ient_id #9580 rclone's shared Google Drive and Google Photos client_id is being retired and will stop working during 2026. When creating a new remote that would use it, the config wizard now warns the user and asks the user to enter their own client_id and secret instead. Service account and environment auth are unaffected as they don't use the shared client_id. See: https://forum.rclone.org/t/google-drive-and-google-photos-users-action-required/54005
When mounting or otherwise opening an S3 prefix without a trailing slash, rclone probes the path with a HEAD request to see whether it is actually a file. Since v1.72.0 (#8975) any error other than "not found" from that probe was fatal, so credentials scoped to a prefix - which return 403 rather than 404 for the prefix key - could no longer open the prefix at all. 6440052 s3: fix single file copying behavior with low permission A 403 on the probe is ambiguous: it can mean either "this is the file you named but you may not HEAD it" or "this is a prefix you may list but not HEAD". When the HEAD is not permitted we now fall back to a listing to disambiguate: if the path has children it is treated as a directory, otherwise it is treated as a file. Fixes #9582
- CVE-2026-39822: os: Root escape via symlink plus trailing slash - CVE-2026-42505: crypto/tls: Encrypted Client Hello privacy leak
…SA-8v25-v8p6-qf7v S3 object keys are opaque names that may legally contain `..` segments. `serve s3` built backend paths with `path.Join(bucket, key)`, which normalised the key so a request such as `GET /bucket/../root-secret.txt` resolved to a file outside the selected bucket elsewhere under the serve root. Listing prefixes and multipart uploads were affected also. This did not allow reading of files outside the root, but did allow reading of files in the root which normally aren't visible; only directories are visible as buckets normally. Because `serve s3` maps keys to file paths it cannot represent every opaque S3 key, so rather than normalising keys (which would alias distinct keys onto one file as well as allow traversal) it now rejects any key that is not already in canonical path form - containing `..`, `.`, `//` or a leading or trailing slash - with a 400 Bad Request, as MinIO does. Directory listing prefixes are validated the same way but allow the empty bucket-root prefix and an optional trailing slash. Fixes: GHSA-8v25-v8p6-qf7v
…nation CVE-2026-59732 Archive entry names are attacker controlled. `rclone archive extract` stripped only a leading `./` and then joined the entry name onto the destination directory with `path.Join`, which collapses `..` segments. An entry such as `../escaped.txt` extracted into `:s3:bucket/safe/prefix` therefore resolved to `bucket/safe/escaped.txt`, outside the selected `prefix` directory - a path traversal ("Zip Slip") attack that could create or overwrite sibling objects on any destination remote. Entry names are now validated before use: a leading `./` is still stripped (tar archives created with `tar -czf archive.tar.gz .` rely on this), but any entry with a `..` path component is rejected. Both `/` and `\` are treated as separators when looking for `..`, as the local backend treats `\` as a path separator on Windows. Fixes: GHSA-4vr5-p2gc-h23p
A user could reach another user's private repository by sending a path such as /<me>/../<victim>/config. The authorization check compares the first path segment against the authenticated user, while the backend object key was built from the raw, un-cleaned URL path. Reject any non-canonical request path so the authorization segment and the backend object key can no longer disagree. Fixes GHSA-fqj9-69pf-6pjg
…f44-9pgv-m4xc The CheckRedirect policy strips the X-Amz-Security-Token header when a redirect chain crosses a host, but it only compared the host and ignored the scheme. A redirect that kept the same host:port but downgraded https:// to http:// was treated as the same host, so the STS session token was re-sent over a plaintext connection where it could be observed. Fixes GHSA-cf44-9pgv-m4xc
…pping in-flight requests This switches the calls to the dropbox SDK over to use the context taking variants. This allows context cancellation to propagate properly. Fixes #9688
The cmd/gitannex end to end tests routinely take 230-530s and on slow CI runners the package exceeds the go test default timeout of 10m. Raise the per-package timeout to 20m in quicktest and racequicktest to give slow runners headroom.
The migration test ran the full "git annex testremote" suite for every layout mode after verifying the migration with "git annex fsck". The fsck calls already prove the migrated data is accessible via the builtin special remote, and TestEndToEnd covers the special remote protocol with testremote, so the extra five full testremote runs duplicated coverage at a cost of tens of seconds each on CI.
Run the full "git annex testremote" suite for a single layout mode and use "testremote --fast" for the rest. The full suite repeats the same protocol operations across a matrix of key sizes and chunk configurations, which exercises client side git-annex behaviour rather than rclone.
The end to end tests exercise rclone via a separate subprocess spawned by git-annex, and that subprocess is not built with race instrumentation. Running them in the race test therefore adds several minutes to CI without providing any race coverage. The unit tests in this package still run under the race detector.
The subtests within each end to end test function already run in parallel, but the three test functions themselves ran one after another. Marking the functions parallel lets all their subtests overlap.
Write the git configuration for each test's fake home directory as a file instead of running three "git config --global" commands, and drop the "git annex version" invocation from repository setup. This removes four subprocess launches from each of the fifteen test cases.
…ash - fixes #9681 purgeCheck listed a directory's children with includeAll set, so the Drive API returned trashed children as well as live ones. The only thing that came of that was the trashedFiles flag, which is used solely to decide whether to trash the directory or delete it permanently, and when use_trash is on the directory gets trashed either way. That made the common case pay for nothing: after trashing N files in a directory, removing the now empty directory paged through all N trashed entries, about one API call and pacer delay per 1000, instead of running a single query that returns nothing. Ask the server to filter out trashed children when the directory is going to be trashed anyway. Hard deletes still enumerate them, since there the distinction decides whether the directory can be removed permanently (#1040), and --drive-trashed-only is left alone because it deliberately lists trashed items.
Object.Update held its connection until the deferred putConnection ran at function exit, so the SetModTime it does at the end of every upload had to take a second connection from the pool, dialling a whole new SMB session when the pool was empty. With N transfers in flight the pool grew to roughly 2N sessions for no reason. Return the connection as soon as the file is closed. At that point the upload has succeeded and remove() can no longer be reached, so nothing else needs it, and SetModTime picks the same connection straight back out of the pool. putConnection nils the pointer, so the deferred putConnection becomes a no-op and the connection is not returned twice.
…ared size A source reader which returns EOF before supplying as many bytes as it declared should not cause a backend Put to hang, panic or create an object claiming the declared size. The layers above detect a truncated transfer by comparing sizes after the upload, so a backend may either return an error or create a shorter object. See #9704
#9704 A source which returned EOF before supplying as many bytes as it declared would either commit a truncated file (if the shortfall was within the final chunk) or loop forever appending empty chunks to the upload session. Return an error wrapping io.ErrUnexpectedEOF instead. Note that all dropbox uploads use the chunked upload path with the default batch_mode of sync, so this affected uploads of every size.
… cancelled The append loop retries everything once the upload session has started, so a cancelled context error was retried through all the low level retries with exponential backoff before the upload gave up.
When the source supplied fewer bytes than its declared size, the compressed data file was stored under a name containing the declared size while the metadata recorded the actual number of bytes read. NewObject looks the data file up by the size in the metadata, so the resulting object could never be read again, and the upload reported success. Check that the number of bytes read matches the declared size after uploading the data and before writing the metadata, and remove the data file and return an error if it does not. This was found by the new FsPutShortEOF integration test.
The SFTP page said `--dump-auth`, which no longer exists: it became a value of `--dump`, so the docs asked for a flag rclone would reject. The same line, and a line in the Swift troubleshooting section, also used `--dump-headers` and `--dump-bodies`. Those still parse, but SetFlags logs "--dump-headers is obsolete - please use --dump headers instead", so the docs were steering readers onto a deprecated form. The generated flag listings in docs/content/flags.md and docs/content/commands/rclone.md are left alone: those flags do still exist, so `--help` output should keep showing them.
…nds early Uploads through RcatSize with a known size - used by rcat --size, the rc operations/uploadfile and the serve backends, eg serve restic - did not check the size of the uploaded object. If the source stream ended before the declared size worth of data had been read, the truncated object was reported as a successful upload. This could corrupt data for callers which trust the result, eg a restic repository accessed via serve restic (see #9722). This adds the same size check operations.Copy performs after a copy, respecting --ignore-size and backends which do not report sizes.
#9722 When an upload failed part way through with a retryable error (eg a 502 from the block storage servers) the pacer retried the whole upload call with the same input stream. The stream had already been partially consumed, so the retry re-created the upload draft and committed just the remainder of the stream as a complete file, silently truncating it. With restic over serve restic this corrupted the repository as the truncated pack was reported as successfully uploaded. This fixes it by using CallNoRetry for the upload, as the other backends do, so retryable errors are returned wrapped in a RetryError for the caller to retry the upload with a fresh stream.
…itectures - fixes #9723 The precision field in the backend overview YAML files can hold fs.ModTimeNotSupported (100 years in nanoseconds) which overflows int on 32 bit platforms, making the YAML for those backends fail to parse and causing rclone to log 18 internal errors on every invocation. Use int64 for the precision field and add a test that parses every embedded backend YAML file so this is caught on 32 bit test runs.
When using Microsoft Entra ID credentials, Azure Blob server-side copy uses a user delegation SAS URL for the private copy source. The SAS start time was set to the current local time. Azure Storage validates the copy source from the service side, and small clock differences can make that SAS appear not yet valid. The service then returns 403 CannotVerifyCopySource with AuthenticationFailed. Start the copy-source SAS 15 minutes in the past, matching Microsoft SAS guidance for clock skew.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )