Advanced search

Message boards : Number crunching : Upload server sometimes awfully slow

Author Message
Erich56
Send message
Joined: 1 Jan 15
Posts: 831
Credit: 3,470,292,479
RAC: 1,400,320
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwat
Message 57626 - Posted: 16 Oct 2021 | 7:00:42 UTC

Within the past weeks, I have made the experience that while sometimes the finished WUs are uploaded with a good speed of nearly 1000 KB/sec, sometimes the speed is awfully low, like 60 KB/sec. So it takes forever to get a 500MB file uploaded, resulting in the loss of the <24 hours / <48 hours credit bonus.

How can this happen, with only about 80 users currently? I cannot remember this kind of problem at former times when there were 1000 or more users crunching and uploading their WUs.

Erich56
Send message
Joined: 1 Jan 15
Posts: 831
Credit: 3,470,292,479
RAC: 1,400,320
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwat
Message 57652 - Posted: 26 Oct 2021 | 5:33:40 UTC

No idea why the problems with the upload server obviously cannot be fixed :-(
Again, a finished task is uploading at ~60 kb/sec :-(

So anyone can figure out how long the upload lasts for a 499 MB size file.

Erich56
Send message
Joined: 1 Jan 15
Posts: 831
Credit: 3,470,292,479
RAC: 1,400,320
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwat
Message 57658 - Posted: 26 Oct 2021 | 12:42:55 UTC - in response to Message 57652.

So anyone can figure out how long the upload lasts for a 499 MB size file.

the latest upload took more than 2 hours.
Whereas my internet connection provides 3 Mbit/sec (which works in reality, I can see it when transferring files within the LHC project).

That's crazy :-(

Erich56
Send message
Joined: 1 Jan 15
Posts: 831
Credit: 3,470,292,479
RAC: 1,400,320
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwat
Message 57662 - Posted: 26 Oct 2021 | 18:43:21 UTC

now the next tasks which uploads with ~80 kb/sec :-(

Am I the only one who experiences this problem, or are there others, too?

Profile ServicEnginIC
Avatar
Send message
Joined: 24 Sep 10
Posts: 462
Credit: 2,137,165,494
RAC: 1,470,965
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 57664 - Posted: 26 Oct 2021 | 21:22:59 UTC - in response to Message 57662.

Am I the only one who experiences this problem, or are there others, too?

Maybe Gpugrid upload server is not the bottleneck.
I can bring an example of just the opposite.
I catched one of my hosts at the very end of uploading a 506,29 MB file.
It took 11 minutes and 15 seconds to upload, at rates above 700 KBps.
This host has a WiFi connection to an access point in my home network.
Internet access is through a fiber optic router.

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 813
Credit: 1,104,869,831
RAC: 2,478,776
Level
Met
Scientific publications
watwatwatwatwat
Message 57673 - Posted: 27 Oct 2021 | 6:39:45 UTC

All my uploads take 2+ hours because I have a miserable 1MBit upload pipe that only produces about 758KB in reality.

Oh how I wish I had a fiber connection. One can only dream I guess.

Erich56
Send message
Joined: 1 Jan 15
Posts: 831
Credit: 3,470,292,479
RAC: 1,400,320
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwat
Message 57675 - Posted: 27 Oct 2021 | 12:15:52 UTC - in response to Message 57664.

Maybe Gpugrid upload server is not the bottleneck.
I can bring an example of just the opposite.
I catched one of my hosts at the very end of uploading a 506,29 MB file.
It took 11 minutes and 15 seconds to upload, at rates above 700 KBps.
This host has a WiFi connection to an access point in my home network.
Internet access is through a fiber optic router.

As I wrote in my original posting, I also had cases where the upload speed for a GPUGRID task was about 1000 KB/sec.

At any rate, the bottleneck definitely cannot be within my own infrastructure, since I have a fiber glass connection with 30 Mbits upload / 300 Mbits download, which are tested/confirmed values.

Maybe the bottleneck lies somewhere inbetween my place and GPUGRID in Spain.

Still, it is somehow frustrating to see that some of the uploads go with 1000 KB/sec, and others with 60 KB/sec.

Ian&Steve C.
Avatar
Send message
Joined: 21 Feb 20
Posts: 546
Credit: 4,545,666,357
RAC: 7,461,901
Level
Arg
Scientific publications
wat
Message 57677 - Posted: 27 Oct 2021 | 13:27:11 UTC - in response to Message 57673.

All my uploads take 2+ hours because I have a miserable 1MBit upload pipe that only produces about 758KB in reality.

Oh how I wish I had a fiber connection. One can only dream I guess.


SpaceX Starlink?
____________

bozz4science
Send message
Joined: 22 May 20
Posts: 104
Credit: 25,049,591
RAC: 276,192
Level
Val
Scientific publications
wat
Message 57679 - Posted: 27 Oct 2021 | 14:22:54 UTC
Last modified: 27 Oct 2021 | 14:26:30 UTC

Great idea and awesome tech but comes at price. IMO it's a bit excessive even if you live in a remote area and are in dire need of a working alternative.

Just f.ex. current pricing for my service address in Germany would be 558€ plus 99€/month service fee. Nah

Erich56
Send message
Joined: 1 Jan 15
Posts: 831
Credit: 3,470,292,479
RAC: 1,400,320
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwat
Message 57681 - Posted: 27 Oct 2021 | 17:58:16 UTC

about 2 hours ago, a finished task was uploaded with ~ 980 KB/s, right now another task is uploading with ~ 80 KB/s.
There is never any value inbetween - either it's very fast, or very slow. This is rather strange, isn't it ?

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 813
Credit: 1,104,869,831
RAC: 2,478,776
Level
Met
Scientific publications
watwatwatwatwat
Message 57687 - Posted: 28 Oct 2021 | 17:42:53 UTC - in response to Message 57677.

I'm thinking on it. My brother has an order in for it. I think I will wait for him to test it and give me some feedback. He has an even slower connection than I do.

I'm a bit leery of using it for an always on connection since I have more data going upstream all the time compared to the intermittent topup downloads.

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 813
Credit: 1,104,869,831
RAC: 2,478,776
Level
Met
Scientific publications
watwatwatwatwat
Message 57688 - Posted: 28 Oct 2021 | 17:45:06 UTC - in response to Message 57681.

Probably not so strange considering only one host runs ALL the processes. Imagine the I/O gets bottlenecked regularly.

Erich56
Send message
Joined: 1 Jan 15
Posts: 831
Credit: 3,470,292,479
RAC: 1,400,320
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwat
Message 57689 - Posted: 28 Oct 2021 | 18:11:40 UTC - in response to Message 57688.

Probably not so strange considering only one host runs ALL the processes. Imagine the I/O gets bottlenecked regularly.

well, the fact is that whenever LHC/Atlas tasks are being uploaded, or downloaded (and this happens several times every hour), no speed problem at all. Never ever!

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 813
Credit: 1,104,869,831
RAC: 2,478,776
Level
Met
Scientific publications
watwatwatwatwat
Message 57690 - Posted: 28 Oct 2021 | 19:41:04 UTC - in response to Message 57689.

Strange indeed if the tasks go fast when another project is actively down/uploading.

Must be something to with the HTTP/S protocol handshakes. The get sorted out with another project in play but get bogged down when GPUGrid itself is uploading.

I still see the DDOS remediation from the project when several hosts from the same IP address are communicating with the project. I can wait several minutes and often the link is dropped when I try and view the website pages.

Profile ServicEnginIC
Avatar
Send message
Joined: 24 Sep 10
Posts: 462
Credit: 2,137,165,494
RAC: 1,470,965
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 57691 - Posted: 28 Oct 2021 | 21:49:10 UTC - in response to Message 57681.

There is never any value inbetween - either it's very fast, or very slow. This is rather strange, isn't it ?

Yes, it is...
From Windows Command Prompt you can order to trace the route from your host to the Gpugrid server, take screenshots, and compare between slow and fast episodes.
It might reveal some intermediate server that could be delaying communications.
You can enter a Command Prompt window by pressing Windows + R keys, and running CMD
From the Command Prompt window, type in tracert gpugrid.net
You'll get information about how many jumps it takes to access to Gpugrid server from your location, and the delay in milliseconds between every of them.

In the following example, from my location it took 14 jumps to reach the Gpugrid server (intermediate IPs have been hidden)



(Most of the times this is worthless, but it's cool ;-)

Post to thread

Message boards : Number crunching : Upload server sometimes awfully slow