Q... is progress report possible?

[boelle]

so it just sits there, the normal Rsync at least shows progress

are there any options to get a more "quickly" progress report?

[xuanyuanaosheng]

@boelle @nathanhaigh
I meet the same problem:

I have "5.3T"data to sync and the data have too much small file. The tmp dir:

[root@test tmp.1RwTcdyHWc]# pwd
/tmp/tmp.1RwTcdyHWc
[root@test tmp.1RwTcdyHWc]# ll -lh
total 160M
-rw-r--r--. 1 root root 7.3M Sep  4 10:21 chunk.0
-rw-r--r--. 1 root root 7.3M Sep  4 10:21 chunk.1
-rw-r--r--. 1 root root 7.3M Sep  4 10:21 chunk.2
-rw-r--r--. 1 root root 7.3M Sep  4 10:21 chunk.3
-rw-r--r--. 1 root root 7.3M Sep  4 10:21 chunk.4
-rw-r--r--. 1 root root 7.3M Sep  4 10:21 chunk.5
-rw-r--r--. 1 root root 7.3M Sep  4 10:21 chunk.6
-rw-r--r--. 1 root root 7.3M Sep  4 10:21 chunk.7
-rw-r--r--. 1 root root 7.3M Sep  4 10:21 chunk.8
-rw-r--r--. 1 root root 7.3M Sep  4 10:21 chunk.9
-rw-r--r--. 1 root root  87M Sep  4 10:15 files.all

it just sits there:

I can find the process using ps -ef | grep rsync

# ps -ef | grep rsync
root      943480  943096  9 09:50 pts/1    00:05:53 /bin/bash ./prsync --parallel=10 --times --recursive --progress root@test:/data_nfs_new/test001/2019/01/ /data/rsync_data/
root      944861  943480  0 10:21 pts/1    00:00:04 /usr/bin/perl /usr/bin/parallel -j 10 -t --verbose --progress rsync --files-from={} --times --recursive --progress root@test:/data_nfs_new/test001/2019/01/ /data/rsync_data/
root      944875  944861 99 10:21 pts/1    00:28:56 rsync --files-from=/tmp/tmp.1RwTcdyHWc/chunk.0 --times --recursive --progress root@test:/data_nfs_new/test001/2019/01/ /data/rsync_data/
root      944876  944861 99 10:21 pts/1    00:28:55 rsync --files-from=/tmp/tmp.1RwTcdyHWc/chunk.1 --times --recursive --progress root@test:/data_nfs_new/test001/2019/01/ /data/rsync_data/
root      944877  944861 99 10:21 pts/1    00:28:56 rsync --files-from=/tmp/tmp.1RwTcdyHWc/chunk.2 --times --recursive --progress root@test:/data_nfs_new/test001/2019/01/ /data/rsync_data/
root      944878  944861 99 10:21 pts/1    00:28:56 rsync --files-from=/tmp/tmp.1RwTcdyHWc/chunk.3 --times --recursive --progress root@test:/data_nfs_new/test001/2019/01/ /data/rsync_data/
root      944879  944875  0 10:21 pts/1    00:00:00 ssh -l root test rsync --server --sender -trRe.iLsfxC --files-from=- --from0 . /data_nfs_new/test001/2019/01/
root      944880  944861 99 10:21 pts/1    00:28:56 rsync --files-from=/tmp/tmp.1RwTcdyHWc/chunk.4 --times --recursive --progress root@test:/data_nfs_new/test001/2019/01/ /data/rsync_data/
root      944881  944876  0 10:21 pts/1    00:00:00 ssh -l root test rsync --server --sender -trRe.iLsfxC --files-from=- --from0 . /data_nfs_new/test001/2019/01/
root      944882  944861 99 10:21 pts/1    00:28:55 rsync --files-from=/tmp/tmp.1RwTcdyHWc/chunk.5 --times --recursive --progress root@test:/data_nfs_new/test001/2019/01/ /data/rsync_data/
root      944883  944877  0 10:21 pts/1    00:00:00 ssh -l root test rsync --server --sender -trRe.iLsfxC --files-from=- --from0 . /data_nfs_new/test001/2019/01/
root      944884  944861 99 10:21 pts/1    00:28:56 rsync --files-from=/tmp/tmp.1RwTcdyHWc/chunk.6 --times --recursive --progress root@test:/data_nfs_new/test001/2019/01/ /data/rsync_data/
root      944885  944878  0 10:21 pts/1    00:00:00 ssh -l root test rsync --server --sender -trRe.iLsfxC --files-from=- --from0 . /data_nfs_new/test001/2019/01/
root      944886  944861 99 10:21 pts/1    00:28:56 rsync --files-from=/tmp/tmp.1RwTcdyHWc/chunk.7 --times --recursive --progress root@test:/data_nfs_new/test001/2019/01/ /data/rsync_data/
root      944887  944880  0 10:21 pts/1    00:00:00 ssh -l root test rsync --server --sender -trRe.iLsfxC --files-from=- --from0 . /data_nfs_new/test001/2019/01/
root      944888  944882  0 10:21 pts/1    00:00:00 ssh -l root test rsync --server --sender -trRe.iLsfxC --files-from=- --from0 . /data_nfs_new/test001/2019/01/
root      944889  944861 99 10:21 pts/1    00:28:56 rsync --files-from=/tmp/tmp.1RwTcdyHWc/chunk.8 --times --recursive --progress root@test:/data_nfs_new/test001/2019/01/ /data/rsync_data/
root      944890  944884  0 10:21 pts/1    00:00:00 ssh -l root test rsync --server --sender -trRe.iLsfxC --files-from=- --from0 . /data_nfs_new/test001/2019/01/
root      944891  944886  0 10:21 pts/1    00:00:00 ssh -l root test rsync --server --sender -trRe.iLsfxC --files-from=- --from0 . /data_nfs_new/test001/2019/01/
root      944892  944861 99 10:21 pts/1    00:28:56 rsync --files-from=/tmp/tmp.1RwTcdyHWc/chunk.9 --times --recursive --progress root@test:/data_nfs_new/test001/2019/01/ /data/rsync_data/
root      944894  944889  0 10:21 pts/1    00:00:00 ssh -l root test rsync --server --sender -trRe.iLsfxC --files-from=- --from0 . /data_nfs_new/test001/2019/01/
root      944896  944892  0 10:21 pts/1    00:00:00 ssh -l root test rsync --server --sender -trRe.iLsfxC --files-from=- --from0 . /data_nfs_new/test001/2019/01/
root      949456  943377  0 10:50 pts/3    00:00:00 grep --color=auto rsync

Any idears?

[boelle]

nope, i'm just a noob

i found this in my quest to run rsync in parallel, if i do not my internet connection to a remote site is not pushed to the limit

Right now it runs about 19 times slower

but maybe @nathanhaigh will give us a hint on how we can get a simple progress info like transfer speed and total job complete or something like that

[xuanyuanaosheng]

@boelle This error is because it need some time to handle this, I wait some time and it works. Just need more time

[boelle]

@xuanyuanaosheng so how does the progress report look like?

but i will try myself

[boelle]

i deleted one folder with backup of the system drive (OpenMediaVault) simple enough to recreate if this script would fail

i run the script and yes it finds the correct folder

now it just hangs there and i cant see what its doing.... some more "active" progress reporting would be nice....

lets see how it moves on

EDIT: 30 mins later and no progress

[boelle]

doing an ls on the offsite backup shows that its doing something

root@omvnap:/srv/mergerfs/Data/homedir/bo/Borgbackup/Backup-Bo-OMV_NAP/OS_Backup/omvbackup# ls -a
. .. backup-omv-2024-08-25_02-05-01.blkid .backup-omv-2024-08-25_02-05-01.ddfull.gz.95cbW1 .backup-omv-2024-08-25_02-05-01.ddfull.gz.JAX7m4

but if you ask me this should not be needed, the progress should be displayed at the other end just like a normal rsync would do

EDIT: data is being moved

root@omvnap:/srv/mergerfs/Data/homedir/bo/Borgbackup/Backup-Bo-OMV_NAP/OS_Backup/omvbackup# ls -lha
total 18G
drwxr-sr-x+ 2 root users 4.0K Sep 4 10:04 .
drwx--S---+ 3 root users 4.0K Dec 18 2022 ..
-rw-rw----+ 1 root users 863 Aug 25 01:05 backup-omv-2024-08-25_02-05-01.blkid
-rw-------+ 1 root users 8.7G Sep 4 10:55 .backup-omv-2024-08-25_02-05-01.ddfull.gz.95cbW1
-rw-------+ 1 root users 8.6G Sep 4 10:55 .backup-omv-2024-08-25_02-05-01.ddfull.gz.JAX7m4
root@omvnap:/srv/mergerfs/Data/homedir/bo/Borgbackup/Backup-Bo-OMV_NAP/OS_Backup/omvbackup# ls -lha
total 18G
drwxr-sr-x+ 2 root users 4.0K Sep 4 10:04 .
drwx--S---+ 3 root users 4.0K Dec 18 2022 ..
-rw-rw----+ 1 root users 863 Aug 25 01:05 backup-omv-2024-08-25_02-05-01.blkid
-rw-------+ 1 root users 8.7G Sep 4 10:55 .backup-omv-2024-08-25_02-05-01.ddfull.gz.95cbW1
-rw-------+ 1 root users 8.7G Sep 4 10:55 .backup-omv-2024-08-25_02-05-01.ddfull.gz.JAX7m4

roughly 700MB every 3 minutes

[nathanhaigh]

There is some limitations here since GNU parallel is used to run multiple rsync processes in parallel. As such, collecting and reporting the global progress from multiple processes is not straight forward. I think GNU parallel reports when one of its processes complete but if each process is transfering a large amount of files/data it could take a while for it's progress indicator to "update".
Implementing a global progress bar would be nice but would likely need a fair bit of effort/time which I don't have right now.

There are a few options you might like to explore:

• Use bmon to monitor the amount of network bandwidth being used - that way you know the transfer is happening
• It might be good for me to implement 2 arguments, one for the number of chunks to be used for dividing the file list and one for the maximum number of rsync/chunks to be transferred in parallel. That way the user can split things into smaller chunks and then GNU Parallel will report the completion of each chunk more quickly in its own progress status.

[boelle]

aha...

well it has now transfered 43G, but the output from the script has not changed

smaller chunks seems to be a good compromise, i'm not an expert in rsync but if it crashes will it start over or can it pick up from the last chunk sent?

[xuanyuanaosheng]

@boelle
It can not report the time of all transfer this is because the parallel process.
Please take a look at

1. https://gist.github.com/rcoup/5358786
2. https://gist.github.com/akorn/644855ddaa8065f564be
3. https://gist.github.com/reijovosu/fce3d808bed89d5021ade70223bfc4c3

[boelle]

well i have kind of given up on it as its above my head

and when the first sync is done not much change in my case