You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Discuss potential pitfalls of concurrent upload (currently set to 3) #4
at the moment, uploading a large file is split into chunks, which get pushed to nextcloud server 3 at a time; i've found that 3 connections are not enough for me to max out my network line and think it would be great if this value is configurable via occ
@realies I'm really interested in this discussion! I chose 3 kind of randomly.
Do you have some informations or insights on which amount should be better?
Is the files.chunked_upload.max_parallel_count how many chunks per file being uploaded? Such that with 3 files being uploaded in parallel its technically 15 chunks being uploaded (using default values)?
What are the benefits of increasing one vs the other (chunked upload vs # of files)?
The chunked_upload setting is a maximum value, how is the used value determined? I can know the uplink of the server but not the clients, will clients choose depending on their uplink speed, reliability (degree of packet loss) and latency? That would be the optimal as it has an large effect on the "efficiency" of the upload - but maybe most (all?) of such things are handled by the reverse proxy anyway? (relevant http2 discussion)
Does the files.chunked_upload.max_parallel_count even affect anything beyond the reverse proxy?
Is the files.chunked_upload.max_parallel_count how many chunks per file being uploaded? Such that with 3 files being uploaded in parallel its technically 15 chunks being uploaded (using default values)?
No it is how many parallel requests for uploads are allowed. If you set it to 3 and you do not have chunked uploads but 3 simple uploads then those 3 run in parallel.
If you have 2 upload with 2 chunks each then 3 of those 4 total chunks are uploaded.
Does the files.chunked_upload.max_parallel_count even affect anything beyond the reverse proxy?
This has nothing to do with a reverse proxy. It defined how many requests are spawned on the frontend.
And then it depends on how you configured your reverse proxy and your application server (how many handlers etc).
During a single file chunked upload, are the chunks uploaded strictly sequentially (on the nc side)?
Such that any parallelism is only handled by the number of files uploading (3 as seen above), and protocol level thinks such as the http2/3 stuff?
Activity
skjnldsv commentedon Feb 6, 2024
From nextcloud/server#43337
@realies I'm really interested in this discussion! I chose 3 kind of randomly.
Do you have some informations or insights on which amount should be better?
realies commentedon Feb 6, 2024
@skjnldsv, I would like to try with 10 concurrent chunks. What can I patch to try it out?
skjnldsv commentedon Feb 6, 2024
You cannot, why 10 ? Is it just randomly chosen?
What uplink are you using? 1Gbps?
realies commentedon Feb 6, 2024
why 'cannot'? 10, because iperf3 -c x.x.x.x -P 10 gave me optimal results; the link is 1Gbps
skjnldsv commentedon Feb 6, 2024
Because this is compiled libraries, you cannot patch our server bundles
MrRinkana commentedon Aug 29, 2024
If different uplinks perform optimally with different amount of chunks, shouldn't that number be dynamic rather than a compiled in constant?
susnux commentedon Mar 13, 2025
You can try this with setting different values to:
https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/config_sample_php_parameters.html#files-chunked-upload-max-parallel-count
MrRinkana commentedon Mar 13, 2025
Now I'm a bit confused how this works:
Is the
files.chunked_upload.max_parallel_counthow many chunks per file being uploaded? Such that with 3 files being uploaded in parallel its technically 15 chunks being uploaded (using default values)?What are the benefits of increasing one vs the other (chunked upload vs # of files)?
The chunked_upload setting is a maximum value, how is the used value determined? I can know the uplink of the server but not the clients, will clients choose depending on their uplink speed, reliability (degree of packet loss) and latency? That would be the optimal as it has an large effect on the "efficiency" of the upload - but maybe most (all?) of such things are handled by the reverse proxy anyway? (relevant http2 discussion)
Does the
files.chunked_upload.max_parallel_counteven affect anything beyond the reverse proxy?susnux commentedon Mar 13, 2025
No it is how many parallel requests for uploads are allowed. If you set it to 3 and you do not have chunked uploads but 3 simple uploads then those 3 run in parallel.
If you have 2 upload with 2 chunks each then 3 of those 4 total chunks are uploaded.
This has nothing to do with a reverse proxy. It defined how many requests are spawned on the frontend.
And then it depends on how you configured your reverse proxy and your application server (how many handlers etc).
MrRinkana commentedon Mar 13, 2025
Ah I see my mistake, thanks for clarifying!
During a single file chunked upload, are the chunks uploaded strictly sequentially (on the nc side)?
Such that any parallelism is only handled by the number of files uploading (3 as seen above), and protocol level thinks such as the http2/3 stuff?
susnux commentedon Mar 20, 2025
No the chunks can be received in any order