Advanced search

Message boards : Server and website : upload problems

Author Message
davidBAM
Send message
Joined: 17 Sep 18
Posts: 11
Credit: 629,925,813
RAC: 476,235
Level
Lys
Scientific publications
wat
Message 53777 - Posted: 26 Feb 2020 | 23:53:31 UTC

Sorry, I am outta here. Taking WAY too much time to babysit uploads of completed WU. Especially since retry interval often exceeds run time of next WU so I get idle GPUs

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 508
Credit: 532,169,721
RAC: 1,716,928
Level
Lys
Scientific publications
wat
Message 53839 - Posted: 3 Mar 2020 | 20:37:26 UTC

I'm running into slow or stalled uploads the last day.

Need to try a new client that fixes that issue.

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 508
Credit: 532,169,721
RAC: 1,716,928
Level
Lys
Scientific publications
wat
Message 53912 - Posted: 13 Mar 2020 | 20:58:19 UTC

Still having upload issues. Was great this morning when I was able to clear all the backlog of stalled uploads.

But now having backoffs in uploads again. This causes my other projects to not be able to download work because of the hung GPUGrid scheduler connection.

Trotador
Send message
Joined: 25 Mar 12
Posts: 95
Credit: 1,668,759,890
RAC: 1,221,879
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 53978 - Posted: 22 Mar 2020 | 17:49:13 UTC

Seeing:
dom 22 mar 2020 18:43:18 CET | GPUGRID | [error] Error reported by file upload server: can't write file /home/ps3grid/projects/PS3GRID/upload/3d1/3nrnA01_450_2-TONI_MDADpr4sn-2-10-RND1908_0_0: No space left on server

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 508
Credit: 532,169,721
RAC: 1,716,928
Level
Lys
Scientific publications
wat
Message 53979 - Posted: 22 Mar 2020 | 18:09:16 UTC - in response to Message 53978.

Yep.
Sun 22 Mar 2020 11:08:17 AM PDT | GPUGRID | [error] Error reported by file upload server: can't write file /home/ps3grid/projects/PS3GRID/upload/1e5/3zs6A02_379_3-TONI_MDADpr4sz-2-10-RND8994_0_10: No space left on server

Killersocke
Send message
Joined: 18 Oct 13
Posts: 51
Credit: 333,404,147
RAC: 2,258
Level
Asp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwat
Message 53980 - Posted: 22 Mar 2020 | 18:12:28 UTC
Last modified: 22 Mar 2020 | 18:25:24 UTC

same here
upload issues - having backoffs in uploads

22.03.2020 19:24:43 | GPUGRID | [error] Error reported by file upload server: Server is out of disk space

Cartoonman
Send message
Joined: 16 Sep 08
Posts: 1
Credit: 75,029,782
RAC: 835,156
Level
Thr
Scientific publications
wat
Message 53982 - Posted: 22 Mar 2020 | 19:11:44 UTC

Yea seems like upload server ran out of space.

Erich56
Send message
Joined: 1 Jan 15
Posts: 697
Credit: 3,295,149,283
RAC: 329,689
Level
Arg
Scientific publications
watwatwatwatwatwat
Message 53984 - Posted: 22 Mar 2020 | 20:13:05 UTC - in response to Message 53982.

Yea seems like upload server ran out of space.

this is exactly what someone here in the forum was warning about when these 300.000 tasks were put in place for download and crunching.
Obviously, the warning was not taken serious enough :-(

Toni
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 9 Dec 08
Posts: 947
Credit: 4,353,973
RAC: 48
Level
Ala
Scientific publications
watwatwatwat
Message 53985 - Posted: 22 Mar 2020 | 20:25:28 UTC - in response to Message 53984.

The infinite-capacity disks were out of stock.

Profile ServicEnginIC
Avatar
Send message
Joined: 24 Sep 10
Posts: 199
Credit: 1,458,930,848
RAC: 816,218
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 53987 - Posted: 22 Mar 2020 | 20:51:38 UTC - in response to Message 53985.
Last modified: 22 Mar 2020 | 20:55:28 UTC

The infinite-capacity disks were out of stock.

Perhaps a Toni-replication tool should be investigated also...

Pop Piasa
Avatar
Send message
Joined: 8 Aug 19
Posts: 94
Credit: 134,511,679
RAC: 892,693
Level
Cys
Scientific publications
wat
Message 53993 - Posted: 23 Mar 2020 | 1:10:24 UTC

Bloody Sunday, it's full again!
How many 31/2" floppy drives does Grosso have, anyway?

rod4x4
Send message
Joined: 4 Aug 14
Posts: 164
Credit: 1,866,158,186
RAC: 1,221,523
Level
His
Scientific publications
watwatwatwatwatwatwat
Message 53995 - Posted: 23 Mar 2020 | 2:45:40 UTC - in response to Message 53985.

The infinite-capacity disks were out of stock.

dammit! must be with the toilet paper delivery....

Profile ServicEnginIC
Avatar
Send message
Joined: 24 Sep 10
Posts: 199
Credit: 1,458,930,848
RAC: 816,218
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 53997 - Posted: 23 Mar 2020 | 6:38:34 UTC
Last modified: 23 Mar 2020 | 6:43:09 UTC

If funds were required: donation form is still unavailable

Pop Piasa
Avatar
Send message
Joined: 8 Aug 19
Posts: 94
Credit: 134,511,679
RAC: 892,693
Level
Cys
Scientific publications
wat
Message 54004 - Posted: 23 Mar 2020 | 16:47:22 UTC - in response to Message 53997.

If funds were required: donation form is still unavailable

Ditto here. I also can't seem to create a profile. The bot screening picture doesn't load with the page so it rejects saving the profile. Maybe some HTML code errors?

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 508
Credit: 532,169,721
RAC: 1,716,928
Level
Lys
Scientific publications
wat
Message 54155 - Posted: 31 Mar 2020 | 3:06:16 UTC

Upload server is out of disk space again.

Darksider

33488 GPUGRID 3/30/2020 8:03:12 PM [error] Error reported by file upload server: can't write file /home/ps3grid/projects/PS3GRID/upload/d9/2zyrB03_320_1-TONI_MDADpr4sz-2-10-RND2326_0_2: No space left on server

Profile [PUGLIA] kidkidkid3
Avatar
Send message
Joined: 23 Feb 11
Posts: 77
Credit: 796,954,001
RAC: 373,664
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 54156 - Posted: 31 Mar 2020 | 4:50:59 UTC - in response to Message 54155.

Same problem for everyone, thanks in advance for the intervention, Toni
Good day
K.
____________
Dreams do not always come true. But not because they are too big or impossible. Why did we stop believing.
(Martin Luther King)

Erich56
Send message
Joined: 1 Jan 15
Posts: 697
Credit: 3,295,149,283
RAC: 329,689
Level
Arg
Scientific publications
watwatwatwatwatwat
Message 54157 - Posted: 31 Mar 2020 | 4:58:15 UTC

what I don't understand is that there is no estimation of when the disk will be full, while this should be rather easy by simply watching the hourly/daily upload volume.

Profile ServicEnginIC
Avatar
Send message
Joined: 24 Sep 10
Posts: 199
Credit: 1,458,930,848
RAC: 816,218
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 54158 - Posted: 31 Mar 2020 | 6:37:19 UTC

For information:
Tryin to mitigate Coronavirus impact, Spanish Government has hardened measures from March 30th to April 9th, both included.
All non-esencial activities are suspended, in the aim to limit workers movility, and thus reduce fast COVID-19 virus expansion.
There are some exclusions, but I'm afraid that Universitary activities are not.
GPUGrid depends on spanish Pompeu Fabra University (UPF)...
COVID-19 is drastically altering normal activity in many countries. It is a fact.

This also coincides with an outage I've noted today at Rosetta@home.

Toni
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 9 Dec 08
Posts: 947
Credit: 4,353,973
RAC: 48
Level
Ala
Scientific publications
watwatwatwat
Message 54159 - Posted: 31 Mar 2020 | 7:25:29 UTC - in response to Message 54158.

Suspending to try and fix the full disk.

The server's disk is just a buffer. It is emptied continuously. Disk full conditions happen when there is even a temporary imbalance between in (uploads) and out (moving to the main servers) rates. A few hours of imbalance are sufficient to fill it. At such high volumes there is no "easy fix".

Profile ServicEnginIC
Avatar
Send message
Joined: 24 Sep 10
Posts: 199
Credit: 1,458,930,848
RAC: 816,218
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 54160 - Posted: 31 Mar 2020 | 7:34:31 UTC - in response to Message 54159.

Happy to hear from you, Toni.
Thank you very much again (!)

Jim1348
Send message
Joined: 28 Jul 12
Posts: 733
Credit: 1,478,749,566
RAC: 96,002
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwat
Message 54162 - Posted: 31 Mar 2020 | 11:18:39 UTC - in response to Message 54158.

This also coincides with an outage I've noted today at Rosetta@home.

Rosetta is based in the U.S. (Seattle, Washington).
They just have a shortage of work at the moment. Their users have exploded four times since the virus started.

Erich56
Send message
Joined: 1 Jan 15
Posts: 697
Credit: 3,295,149,283
RAC: 329,689
Level
Arg
Scientific publications
watwatwatwatwatwat
Message 54163 - Posted: 31 Mar 2020 | 11:52:49 UTC - in response to Message 54159.

Suspending to try and fix the full disk.

The server's disk is just a buffer. It is emptied continuously. Disk full conditions happen when there is even a temporary imbalance between in (uploads) and out (moving to the main servers) rates. A few hours of imbalance are sufficient to fill it. At such high volumes there is no "easy fix".


thanks for explaining.

After uploads worked again for a few hours this late morning, they have stopped once more since some time ago :-(

Zhouanguo
Send message
Joined: 7 Mar 20
Posts: 1
Credit: 611,762
RAC: 2,977
Level
Gly
Scientific publications
wat
Message 54164 - Posted: 31 Mar 2020 | 12:54:35 UTC

Oh. I had suffer from it. It just started and turn to "Try again later" immediately in few seconds. Ohh... Please! fix it, quick is better.

Aurum
Avatar
Send message
Joined: 12 Jul 17
Posts: 252
Credit: 9,791,563,847
RAC: 2,648,925
Level
Tyr
Scientific publications
wat
Message 54177 - Posted: 31 Mar 2020 | 18:08:08 UTC - in response to Message 54159.

The server's disk is just a buffer. It is emptied continuously. Disk full conditions happen when there is even a temporary imbalance between in (uploads) and out (moving to the main servers) rates. A few hours of imbalance are sufficient to fill it. At such high volumes there is no "easy fix".

Now I understand the 2 WU per GPU limitation, you're trying to balance the goes inners and the goes outters.

Ian&Steve C.
Avatar
Send message
Joined: 21 Feb 20
Posts: 68
Credit: 952,402,490
RAC: 6,389,810
Level
Glu
Scientific publications
wat
Message 54182 - Posted: 1 Apr 2020 | 0:06:58 UTC - in response to Message 54159.

At such high volumes there is no "easy fix".


a larger disk buffer?
____________

ProDigit
Send message
Joined: 13 Nov 19
Posts: 6
Credit: 57,710,107
RAC: 353,664
Level
Thr
Scientific publications
wat
Message 54205 - Posted: 2 Apr 2020 | 16:24:23 UTC

Having the same issue. 6 GPU tasks refusing to upload.

*Edit: Ah, they just went through...

Post to thread

Message boards : Server and website : upload problems