Hi,

I'm seeing this happening with the last dowloaded units, wingmen also have the same error

"process exited with code 212 (0xd4, -44)"

Not sure but it could be only for linux WUs

Also for windows, error message

"(unknown error) - exit code -97 (0xffffff9f)"

Hi,

I'm seeing this happening with the last downloaded units, wingmen also have the same error

"process exited with code 212 (0xd4, -44)"

Not sure but it could be only for linux WUs

On my Windows 7 machine, (I7-3770, GTX 980) I currently had ~10 GERALD WU (more in the cue and still comming in) that were running @ less then %1 GPU usage (according to GPU-Z) while the progress in the BOINC manager appeared to be normal/a little slow (~15 hour estimation per WU). All these WU suddenly disappeared from the BOINC manager without any error massage and also without showing up in my results in my GPUGRID stats. Certainly there is something flawed with these WUs!

I missed the other WUs, but right now this happened to the WU:

e14s27_e9s23p1f368-GERARD_CXCL12_DIM_HEP_GLYCAM-0-1-RND5008

This WU was running @ <1% GPU usage but at close to normal progress speed, however it was restarting every ~10 hours or so. I now cancelled this WU, and the next one in my cue seems to work normally again (e13s16_e8s26p11f203-GERARD_CXCL12_DIMPROTO3-0-1-RND2849; estimated time ~12 hours, 82% GPU usage)

I have both kind of these WUs:

1. Erroring on all hosts, including mine.
https://www.gpugrid.net/workunit.php?wuid=11396918
https://www.gpugrid.net/workunit.php?wuid=11396911

2. Erroring on all hosts, except on mine:
https://www.gpugrid.net/workunit.php?wuid=11397526
https://www.gpugrid.net/workunit.php?wuid=11398513
https://www.gpugrid.net/workunit.php?wuid=11397102
https://www.gpugrid.net/workunit.php?wuid=11397012
https://www.gpugrid.net/workunit.php?wuid=11398161
https://www.gpugrid.net/workunit.php?wuid=11398515
https://www.gpugrid.net/workunit.php?wuid=11396116
https://www.gpugrid.net/workunit.php?wuid=11398187

I have both kind of these WUs:

1. Erroring on all hosts, including mine.
https://www.gpugrid.net/workunit.php?wuid=11396918
https://www.gpugrid.net/workunit.php?wuid=11396911

2. Erroring on all hosts, except on mine:
https://www.gpugrid.net/workunit.php?wuid=11397526
https://www.gpugrid.net/workunit.php?wuid=11398513
https://www.gpugrid.net/workunit.php?wuid=11397102
https://www.gpugrid.net/workunit.php?wuid=11397012
https://www.gpugrid.net/workunit.php?wuid=11398161
https://www.gpugrid.net/workunit.php?wuid=11398515
https://www.gpugrid.net/workunit.php?wuid=11396116
https://www.gpugrid.net/workunit.php?wuid=11398187

I've found the same behavior in my linux hosts, in WUs received since Jan-02-2016 past midday.
Consequently, statistics are getting worse, possibly due to those failing linux WUs...
This can be seen at the bottom of "Server status" page.

https://www.gpugrid.net/server_status.php

On Jan-02-2016 at 22:41 UTC, the medium error rate over the 25 kinds of WUs in progress was 20,9952 %
This has increased to 25,7552 % at 11:44 UTC on Jan-03-2016.

____________

So how many errors did you get recently? If it's a small number, you could attribute that to running into the occasional bad WU. If you have a lot more, than it's more than just a linux problem.

I haven't seen the problem yet on a pair of GTX 960s.
https://www.gpugrid.net/results.php?hostid=194224&offset=0&show_names=0&state=0&appid=

I had originally boosted the P2 memory clock as per ETA's suggestion (https://einstein.phys.uwm.edu/forum_thread.php?id=11044), but saw a few "simulation unstable" messages, though I don't think they led to actual errors at that point. But that was a little to close to the edge for me, so I removed that boost and the cards are back to factory default, which is not much of an overclock on these MSI 2GD5T OC cards. Maybe that keeps them stable on the most difficult work units.

14814161 11399908 286919 3 Jan 2016 | 16:38:22 UTC 3 Jan 2016 | 16:39:05 UTC Error while computing 0.00 0.00 --- Long runs

14814079 11399366 286919 3 Jan 2016 | 16:16:38 UTC 3 Jan 2016 | 16:32:43 UTC Error while computing 0.00 0.00 ---

14813534 11399465 286919 3 Jan 2016 | 13:04:05 UTC 3 Jan 2016 | 13:06:03 UTC Error while computing 0.00 0.00 ---

14801182 11384321 286919 29 Dec 2015 | 20:21:34 UTC 1 Jan 2016 | 9:50:05 UTC Completed and validated 212,450.23 4,110.21 135,300.00 Long runs

Same problem here with a valid run from dezember last year. Greets, Klaus

I have seen the same thing on my linux host - all work units since the one below error out the same way

Stderr output
<core_client_version>7.3.15</core_client_version>
<![CDATA[
<message>
process exited with code 212 (0xd4, -44)
</message>
<stderr_txt>

</stderr_txt>
]]>

14811283 11398815 176528 3 Jan 2016 | 0:28:00 UTC 3 Jan 2016 | 1:08:39 UTC Error while computing 0.00 0.00 --- Long runs (8-12 hours on fastest card) v8.46 (cuda65)

Hello , I'm using ubuntu 14.04 lts. Gerard-WU's stopped after 1 second and were uploaded. "Output file was absent" for four files at a time. I did some collatz conjecture earlier today but I guess that didn't mess up my computer as others are having problems too.

opr

Not sure if it's related, but I too just had an error with a Gerard unit, which is a rare thing to happen for me.

http://www.gpugrid.net/workunit.php?wuid=11389493
Exit status 194 (0xc2) EXIT_ABORTED_BY_CLIENT
(unknown error) - exit code 194 (0xc2)

Name e3s31_e2s25p1f424-GERARD_CXCL12_CHALC4_DIM1-0-1-RND7047_1
Workunit 11389493
Created 1 Jan 2016 | 22:45:42 UTC
Sent 1 Jan 2016 | 22:45:48 UTC
Received 3 Jan 2016 | 11:06:22 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0xc2) EXIT_ABORTED_BY_CLIENT
Computer ID 153764
Report deadline 6 Jan 2016 | 22:45:48 UTC
Run time 80,101.12
CPU time 11,903.64
Validate state Invalid
Credit 0.00
Application version Long runs (8-12 hours on fastest card) v8.47 (cuda65)
Stderr output

<core_client_version>7.6.22</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code 194 (0xc2)
</message>

Since 16:48 UTC on the second of January, my Linux host(206986) has failed all WU's it has received. My 2 Windows hosts are working as usual. A quick look through the task lists for the top 10 users shows the same pattern. Has anyone come up with a theory as to what is happening? In the meantime I have set that host to NNT to avoid causing any congestion at the server-side.

This issue continues ocurring in all my hosts (Linux).

Guess is that administrators are still in holidays, no claim, they deserve them.

Thanks Matt. Hope the update will improve it's performance

WU e15s19_e14s24p1f286-GERARD_CXCL12_DIMPROTO3-0-1-RND3500_2 has been stuck at '85% "progress" ' for some 12 hours now.
I only have a 640, so WUs take 40-60 hours generally.
This task has already run for 69:58.
is this part of a defective batch?
How many more hours should I sacrifice for this WU?
I am presuming that if I abort it, there will be zero credit for these 70 hours (even if it is a flawed WU?)

I Run Win 7 on my HP-1120, i7-2600. (I am NOT going to 'upgrade' to Win 10 for months, until (I hope) MS gets all the garbage in Win8-10 patched up.)

Please advise.

Meanwhile I have paused it and am putting my GPU to better use.

Thanks.

____________
I think ∴ I THINK I am
My thinking neither is the source of my being
NOR proves it to you
God Is Love, Jesus proves it! ∴ we are

I'd suggest restarting the PC. And if the problem still persists, then abort the task.

Hi,

I was wondering if others are still having this problem as the issue still persists on both my Linux boxes.

Yep, the GPUGRID tasks are still not processing on my Linux boxes.

But my backup project is getting quite a bit of work done.

Okay, thanks for the update. I guess I'll just add a backup project myself until the issue is resolved.

Cheers.

Same here, no joy for Linux hosts, five days in a row, we don't seem to be anything worthy for the project.

The tasks were doing OK on my XP box. I moved the cards to a Win7 box and now they all error out in 2 seconds. Looks like moving the cards was a mistake but I can't move them back ATM.

I managed to download 1 task that didn't error out in 2 seconds. *fingers crossed*

Still having issues getting tasks to download all the files needed to run. From event log.

4680 GPUGRID 1/9/2016 3:25:00 PM Temporarily failed download of e17s19_e13s27p1f405-GERARD_CXCL12_CHALC2_MON1-0-pdb_file: transient HTTP error

After 5 attempts and 30inutes the last file did finally download.
EDIT: Second task downloaded and is running. Stay tuned.

Same here on Linux. All WUs error out, even after a reset of the project.
____________

Team Belgium

Hi nanoprobe,

did you successfully complete the work unit that started?

Still not running under linux ...

Hi nanoprobe,

did you successfully complete the work unit that started?

One day more without Linux crunching and without status info...who cares?

One day more without Linux crunching and without status info...who cares?

One day more without Linux crunching and without status info...who cares?

Someone didn't get their nap today.

One day more without Linux crunching and without status info...who cares?

Someone didn't get their nap today.

No, don't be a jerk. This has been a known problem with a known cause for a week now and no one has bothered to fix it.

For many years there was a significant performance boost when crunching with Linux at this project. The developers actually recommended that you crunch with Linux. Many of us have dedicated Linux hosts to this project due to that fact. Now my Linux hosts are having to crunch mathematics crap and look for pulsars to keep my house warm.

Could someone please fix this?

That was weird. Triple post.?????

Can't explain the triple post.

Can't explain the triple post.

Guys! Matt is trying to fix it, see https://www.gpugrid.net/forum_thread.php?id=4235 . Apparently the solution must not be trivial. Please be patient!

Now I am getting this same "linux" error on my both my windows machines.


https://www.gpugrid.net/hosts_user.php?userid=19626


Also, when I downloaded a new unit, and I suspended a good unit to test the new unit. The new unit would crash, and when I resumed the previously good unit, it also crashed.

Can't explain the triple post.

You missed your nap? :)

Six errors in the last day on Windows, so has nothing to do with the Linux app.
Also the site is very slow at the moment.
____________
Greetings from TJ

New app 848.

Matt

Across 5 computers, all windows ranging from 7, 8.1, and 10, I have had about 130 errored out WUs in the past 24-30 hours. Over 100 of them are 0.00 second errors and the other 30 or so are with time put in. One of the computers crashed to an unrecoverable crash and needed manual assistance in the BIOS then the OS to get it back to a good state and running again. This caused aborted WUs or was caused by WUs. I didn't get a chance to check logs and drivers and recent updates and stuff, I just got it running again and walked away frustrated.

I did notice all my Windows machines have wanted or done reboots for several Windows Updates, some having to do with drivers such as power management, but most the general ominous "Update to Windows" or "Security Update for Windows" that you need to actual read KB articles to find out what they were. I am pretty sure they all wanted at least one if not two reboots in the past 7 days. I am not sure if this is significant to any issue with errored WUs, but I would suppose it would be more than just me if it was.

I did notice two of the tasks (GERARD_CXCL12_FXCHALC4_DIM and GERARD_CXCL12_FXCHALC4_MON) had a 100% error rate, so I manually aborted about 5 of them at less than an hour into them. They might have been fixed and actually finished, but it was a gut reaction. Before these recent errors I was hovering around 28-32 errors for all of the computers combined, so coming in to see 160 today was a shock. Then seeing they didn't even spend time to crunch eased my fears a bit that it was something totally on my end.

Hope this all works out and after all WUs are going to error out occasionally and bad bathes get released and fixed in every project since distributed computing began. But if there is something I can do on my end to help decrease these errors, let me know that too.
____________
1 Corinthians 9:16 "For though I preach the gospel, I have nothing to glory of: for necessity is laid upon me; yea, woe is unto me, if I preach not the gospel!"
Ephesians 6:18-20, please ;-)
http://tbc-pa.org

New app 848.

Matt

New app 848.

Matt

working OK in my host

Across 5 computers, all windows ranging from 7, 8.1, and 10, I have had about 130 errored out WUs in the past 24-30 hours. Over 100 of them are 0.00 second errors and the other 30 or so are with time put in. One of the computers crashed to an unrecoverable crash and needed manual assistance in the BIOS then the OS to get it back to a good state and running again. This caused aborted WUs or was caused by WUs. I didn't get a chance to check logs and drivers and recent updates and stuff, I just got it running again and walked away frustrated.

I did notice all my Windows machines have wanted or done reboots for several Windows Updates, some having to do with drivers such as power management, but most the general ominous "Update to Windows" or "Security Update for Windows" that you need to actual read KB articles to find out what they were. I am pretty sure they all wanted at least one if not two reboots in the past 7 days. I am not sure if this is significant to any issue with errored WUs, but I would suppose it would be more than just me if it was.

Across 5 computers, all windows ranging from 7, 8.1, and 10, I have had about 130 errored out WUs in the past 24-30 hours. Over 100 of them are 0.00 second errors and the other 30 or so are with time put in. One of the computers crashed to an unrecoverable crash and needed manual assistance in the BIOS then the OS to get it back to a good state and running again. This caused aborted WUs or was caused by WUs. I didn't get a chance to check logs and drivers and recent updates and stuff, I just got it running again and walked away frustrated.

I did notice all my Windows machines have wanted or done reboots for several Windows Updates, some having to do with drivers such as power management, but most the general ominous "Update to Windows" or "Security Update for Windows" that you need to actual read KB articles to find out what they were. I am pretty sure they all wanted at least one if not two reboots in the past 7 days. I am not sure if this is significant to any issue with errored WUs, but I would suppose it would be more than just me if it was.

Now I am the one who can't explain the double post. lol

Matt:
I'm still having the download issue. Anyone else?

I haven't had any issues since the new app was released.

I have on both machines running here. Can someone at least look into this and try to fix it?

I have on both machines running here. Can someone at least look into this and try to fix it?

I have on both machines running here. Can someone at least look into this and try to fix it?

Non stop successful work all month for me. I can't see your machines to see if there is a useful error message since you have them hidden.

Have you tried resetting the project? That sometimes clears up http errors.

I've seen this occasionally, but as far as I know have not had files stuck trying to download for hours as was mentioned in another thread. When I've seen it before it resolved in a few minutes. Tonight I observed a couple tasks stuck (probably around ten files in total), with a backoff time of one hour and 45 minutes. I manually tried downloading again and most of them completed, but a couple hung with the http transient error. I still have one file left which I guess will eventually finish. All my cards are busy so it's not a problem right now.

Interestingly I had no problems uploading a completed job while this was happening.
____________
Team USA forum | Team USA page
Join us and #crunchforcures. We are now also folding:join team ID 236370!

You are right. There is nothing I can see from your tasks that would explain the errors. All your task list shows are the successful tasks, not the ones that it could not download.

I can't offer any suggestions to find out where the issue is in the network. I can only tell you that everything in my path from my machine to GPUGrid does not exhibit that behavior any more. It did for a while a few weeks back, but it seems to have resolved itself with no action on my part.

Looked in the log. Can't find any recent stuck files that stick for hours thankfully, like I've got in the past. However to show that it's still happening, here's some recent ones that were stuck for a shorter time:

14/02/2016 10:33:03 AM | GPUGRID | Temporarily failed download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-vel_file: transient HTTP error
14/02/2016 10:33:03 AM | GPUGRID | Backing off 00:02:51 on download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-vel_file
14/02/2016 10:33:04 AM | | Project communication failed: attempting access to reference site
14/02/2016 10:33:05 AM | | Internet access OK - project servers may be temporarily down.
14/02/2016 10:33:06 AM | GPUGRID | Temporarily failed download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-pdb_file: transient HTTP error
14/02/2016 10:33:06 AM | GPUGRID | Backing off 00:02:24 on download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-pdb_file
14/02/2016 10:33:07 AM | | Project communication failed: attempting access to reference site
14/02/2016 10:33:08 AM | | Internet access OK - project servers may be temporarily down.
14/02/2016 10:35:31 AM | GPUGRID | Started download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-pdb_file
14/02/2016 10:35:55 AM | GPUGRID | Started download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-vel_file
14/02/2016 10:36:42 AM | GPUGRID | Temporarily failed download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-pdb_file: transient HTTP error
14/02/2016 10:36:42 AM | GPUGRID | Backing off 00:07:46 on download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-pdb_file
14/02/2016 10:36:43 AM | | Project communication failed: attempting access to reference site
14/02/2016 10:36:44 AM | | Internet access OK - project servers may be temporarily down.
14/02/2016 10:37:06 AM | GPUGRID | Temporarily failed download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-vel_file: transient HTTP error
14/02/2016 10:37:06 AM | GPUGRID | Backing off 00:04:55 on download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-vel_file
14/02/2016 10:37:07 AM | | Project communication failed: attempting access to reference site
14/02/2016 10:37:08 AM | | Internet access OK - project servers may be temporarily down.
14/02/2016 10:56:49 AM | GPUGRID | Started download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-vel_file
14/02/2016 10:56:49 AM | GPUGRID | Started download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-pdb_file
14/02/2016 10:56:55 AM | GPUGRID | Finished download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-vel_file
14/02/2016 10:57:50 AM | GPUGRID | Finished download of e5s20_e4s2p1f299-GERARD_CXCL12_CHLCPUBCHEM_chalcone4681-0-pdb_file

You are right. There is nothing I can see from your tasks that would explain the errors. All your task list shows are the successful tasks, not the ones that it could not download.

I can't offer any suggestions to find out where the issue is in the network. I can only tell you that everything in my path from my machine to GPUGrid does not exhibit that behavior any more. It did for a while a few weeks back, but it seems to have resolved itself with no action on my part.