Toni,

I have one of these that has been running for over 2 hours and so far looks like it is only using one CPU (thread). It has 4 threads allocated to each task.

There are also a number of warnings messages in the stderr.txt that look like this:

/var/lib/boinc-client/projects/www.gpugrid.net/miniconda/envs/qmml/lib/python3.6/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: '122'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
NaturalNameWarning)

Subj: Observation of QC CPU WU's Core Utilization

Regarding the last go around several weeks ago with the QC CPU WU's that ran successfully on my 8 and 4 core cpu's, the following was observed.

4-core (Phenom II 3 GHz) cpu: core utilization was consistantly near 100% all 4 cores. Work completed in about 40 minutes per WU calender time.

8-core (FX-8350 4 GHz) cpu's (2): 4-cores (alternatively) were utilized at 100% with the remaining cores at much lesser utilization. Work completed with about 20 minutes real time per WU.

With the most recent QC WU's (12/13/2017), one FX-8350 errors out consistantly and will not run the WU's (maybe it requires software or drivers not currently installed) and the other FX-8350 runs them but at a substanially less core utilization rate than the prior WU batches. So far, the first WU in progress is currently less than 40% complete with about 2 hours calender time invested. Not sure if these recent WU's are of the same length as the earlier ones however.

I have uploaded photos of ksysguard graphic cpu utilization that can be observed if interested with addresses below.

Basically, as I am sure that this is a work in progress and bugs need to be resolved but I would conclude so far that these WU's process efficiently on a 4-core system but that all 8-cores should be fully utilized to make it worth while sacrificing 8-cores to a single WU when 8 WU's from other projects use all 8 at 100% thereby being much more efficient. Haven't tried the suspend/resume yet but will when the opprotunity is available. The previous poster appears correct re thread utilization being one issue.

Screenshots:
http://members.toast.net/obc/computing/grid_computing/images/QC_4-core_cpu.png
http://members.toast.net/obc/computing/grid_computing/images/QC_8-core_cpu.png
http://members.toast.net/obc/computing/grid_computing/images/QC-12-13-2017.png

(Sorry, crude way to present photos but wanted to get them up for anyone interested)

Mine is also just a bit less than 1 thread with 7 available for CPU usage. Good thing I have another client available to keep the CPU busy. If the progress bar is correct it will take about 9.5 hours on 1950x at 3.75 GHz.

The Multiple threaded work units that were sent out last month worked fine for me with no issues.

These new ones however are all failing (so far on two different 64bit Linux machines) with this error

ERROR conda.core.link:_execute_actions(337): An error occurred while installing package 'psi4::gcc-5-5.2.0-1'.
LinkError: post-link script failed for package psi4::gcc-5-5.2.0-1
running your command again with `-v` will provide additional information
location of failed script: /home/Conan/BOINC/projects/www.gpugrid.net/miniconda/envs/qmml/bin/.gcc-5-post-link.sh

When checking this path I found that there is nothing in the /envs/ folder, which is probably where the job is failing.

Conan

Hmm 1st one completed for me and the 2nd one is at around 86%.

Toni said,

The next batch (QMML313a) should respect the number of threads requested by your client.

I am not able to get QC on my Ryzen 1700 machine running Ubuntu 17.10. I just get a "No tasks are available for Quantum Chemistry" message when I request them. However, I am able to get QC on my i7 3770 machine running Ubuntu 16.04 (both machines have BOINC 7.8.3). Both machines are set to the same profile (work), so they should be treated identically.

But I see that some people with AMD machines get work. Is this a bug or a feature?

These are called QMML, and rather experimental (more dependencies). Let's see how they work.

The reason why otherwise similar machines get/do not get work completely baffles me.

My 1950x also on 17.10 is getting tasks.

Were the old 3.13 tasks producing bad data as my processing time was just wasted by the server canceling them in the middle of processing them? Thats the absolute worst thing a project admin can do. Cancel ones not started but don't ever take a crap on donated resources.

Its still not working right. In 22.5min of the task running it has used 1:37min of CPU time when the task is limited to 3 cores. That's over 4 cores of CPU usage. And they just had a computation error.

At least 3.13 worked.

http://www.gpugrid.net/result.php?resultid=16767178

Prob a good thing tasks are being sent to some AMD CPUs. Damn seg fault.

Most of the Quantum Chemistry v3.14 (mt) fail on my AMD 1700x. v3.13 (mt) worked more or less.

As an example:
http://www.gpugrid.net/result.php?resultid=16767309

I use an app_config to limit the use to 4 cores for each work unit (WU) and runs 3 WUs in parallel. Two cores are reserved for the GPU.

I had to change the configuration to accept only GPU Work Unites as the computer crashed twice today.

Hope this helps.

Hey friends, in case you need some more machines for testing, I can set up another one that is Linux based. As I have seen in the below conversation, there might be some issues with the CPU type. So .. I have both brands available for testing. Which one would help you most, Intel or AMD?

If you want me to, I could even let you choose the generation. From older Sandy/Ivy Bridge to new Skylake to Ryzen. Lust let me know and I will make one available on short notice.

Edit: I can even contribute a very slow Celeron or Pentium, if that would give you some information on how older and slower systems will perform later... as there still are many older units out there.
____________
I would love to see HCF1 protein folding and interaction simulations to help my little boy... someday.

Quite a few errors on 3.14 so I stopped running QC. Seg faults on AMD and Intel machines.

So I did get some Tasks but it seems that only AMD Processors can run them.
http://www.gpugrid.net/show_host_detail.php?hostid=179948

A nice setting in the Preference for this Tasks would be to allow us to set the number of Cores;
Like on a 32 Core Host you could set 2 Tasks running 16 Cores Each.
Or even 2 Tasks running 8 Cores Each.
____________

Crunching@EVGA The Number One Team in the BOINC Community. Folding@EVGA The Number One Team in the Folding@Home Community.

I received ~60 new WUs yesterday, but I didn't see what happened with them. I was surprised when I went back to the computer an hour or so later and they had all disappeared.

I received another batch of ~60 WUs today, and this time I see that they all resulted in "Computation error".

Intel Xeon E5-2680 x 2 (ie. 32 hyperthreading cores).

I'm not getting any WUs, neither on my AMD, nor on my Intel CPUs.

The Multiple threaded work units that were sent out last month worked fine for me with no issues.

These new ones however are all failing (so far on two different 64bit Linux machines) with this error

ERROR conda.core.link:_execute_actions(337): An error occurred while installing package 'psi4::gcc-5-5.2.0-1'.
LinkError: post-link script failed for package psi4::gcc-5-5.2.0-1
running your command again with `-v` will provide additional information
location of failed script: /home/xxxxxxxx/BOINC/projects/www.gpugrid.net/miniconda/envs/qmml/bin/.gcc-5-post-link.sh

When checking this path I found that there is nothing in the /envs/ folder, which is probably where the job is failing.

Conan

@conan: do you have "gcc" installed in your system? If not, can you try to install it?

Another different problem is task distribution by the BOINC scheduler. First of all, as said above, some hosts are ignored for no reason I can fathom. Another is that some hosts are "soaking up" dozens of WUs, which means they are not available to others. I am hoping that both problems will sort out by themselves with a sufficiently large batch.

Final notes: (a) CPU maker is irrelevant. (b) disappeared WUs were tests which I cancelled from the server.

I got my first WU today. Unfortunately the WU needs 4,7 GB of ram. Can you optimise that?

@conan: do you have "gcc" installed in your system? If not, can you try to install it?

It was installed on on one computer with Fedora 25, but was not installed on the other two with Fedora 16 and Fedora 21, all 64 bit.
Have installed now and await to see what happens.
Versions range from 4.6.3-2 (Fedora 16), 4.9.2-6 (Fedora 21) to 6.4.1-1 (Fedora 25).

Thanks
Conan

I may have understood the problem of hosts not getting WUs. I was sending tasks at a high priority, which means they crossed the threshold to only go to "reliable hosts" -- a questionable heuristic.

100 tasks named "s*-QMML314long" I made at a lower priority seem to have been sent quickly.

@conan: do you have "gcc" installed in your system? If not, can you try to install it?

It was installed on on one computer with Fedora 25, but was not installed on the other two with Fedora 16 and Fedora 21, all 64 bit.
Have installed now and await to see what happens.
Versions range from 4.6.3-2 (Fedora 16), 4.9.2-6 (Fedora 21) to 6.4.1-1 (Fedora 25).

Thanks
Conan

The Fedora 16 host still has the same error, but the Fedora 21 host is processing a work unit now for the last 8 hours 21 minutes and 68% done, so it looks good at this point.
My Fedora 25 host has not received any work yet so can't say about that one.

My WU is using 1.5 GB of RAM.

Thanks
Conan

https://drive.google.com/file/d/1bKmSXT4IAVTR8b-fpiGdC6Gm4szduk0X/view?usp=sharing

Running one now. 1950x, Linux 17.10.
Average time taken is two hours, fifteen minutes per task.

At the time of the screenshot, the work unit is around fourty percent done. I'm watching my CPU usage hit 100%, stay there for a while, then...waves.
I don't think it's thermal throttling. It's not overclocked and WCG tasks only make those patterns when tasks are starting/finishing.

If it's working as intended, ok.

What are the requirements for the running of Psi4?

My Fedora 16 computer after installing "gcc" is still getting the same error that it failed to install, but the Fedora 21 computer is now running fine.

Is there a certain "glibc", "gcc" or Linux kernel that is required to install this programme?
My older Fedora 16 install may not meet the requirements perhaps?

I still can't get any work on my Intel Xeon running Fedora 25, keeps saying that there is no work available when in fact there is, but that is another issue.

Thanks
Conan

My WU is using 1.5 GB of RAM.

@dayle - oscillating CPU% is expected and due to the parts of the calculation which are not parallelized. Thermal throttling is unlikely imho (and I imagine it would manifests as a decrease in CPU clock, not CPU%).

Did tasks just get aborted by the system?

Name s51-TONI_QMML314long-0-1-RND1523_1
Workunit 12930407
Exit status 202 (0xca) EXIT_ABORTED_BY_PROJECT

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<message>
aborted by project - no longer usable</message>
<stderr_txt>

Did tasks just get aborted by the system?

All I can tell is that the one task I had running was a couple hours from completion the last I looked.

Then I checked the task list and saw the cancellation:

16776352 12930407 457243
18 Dec 2017 | 6:16:48 UTC 18 Dec 2017 | 14:17:56 UTC
Cancelled by server 28,813.53 238,856.00 ---
Quantum Chemistry v3.14 (mt)

Can you please confirm that those WUs were cancelled while running and not just while waiting to start?

By the way: these WUs should not run 10+ hours on modern CPUs. That's strange.

It's true that running tasks are being killed. This is not what I expected.

Can you please confirm that those WUs were cancelled while running and not just while waiting to start?

I have still problems with task miniconda-installer reached time limit 360. Tried 4 tasks today with same result (other 2 task I cancelled). Have standard Fedora 26, nothing special.
I don't think the problem is in firewall or slow connection (as suggested in another thread). Is miniconda-installer really downloading something? I rather think, that there is something wrong with installation of files already downloaded on hard-drive.
I think I will now wait some time, and come back later (maybe month or two). Hopefully, it will be resolved.

I downloaded 3 wus 3.14 on my vbox linux.
They don't start....."Waiting to run". No message on boinc manager.

I have still problems with task miniconda-installer reached time limit 360. Tried 4 tasks today with same result (other 2 task I cancelled). Have standard Fedora 26, nothing special.
I don't think the problem is in firewall or slow connection (as suggested in another thread). Is miniconda-installer really downloading something? I rather think, that there is something wrong with installation of files already downloaded on hard-drive.
I think I will now wait some time, and come back later (maybe month or two). Hopefully, it will be resolved.

@ Toni, when these work units run do they get to a certain point then just idle along on a single core for hours on end?

I finally got a work unit to download to my Fedora 25 host, and it ran fine up to about 6 hours or so run time and 78.698% completed.
After this it has been running on a single core for almost 8 hours now and the progress is still locked at 78.698%.
The time to completion has increased from 1 hour 39 minutes to 3 hours 44 minutes and counting.

What happened to the Multi-Threading I thought these work units were supposed to do?

Run time is now approaching 14 hours and the % done has not moved, this is on a 16 core computer.

Conan

Run time is now approaching 14 hours and the % done has not moved, this is on a 16 core computer.

Run time is now approaching 14 hours and the % done has not moved, this is on a 16 core computer.

I seem to recall problems on LHC/ATLAS when running on more than 8 cores, though I was not involved with the problem myself as I run only 7 cores there anyway. But you could try an app_config.xml to limit it to 8 cores.

@Toni, Conan: Thanks to both of you. Looks like the SELINUX was blocking it. It's kind of black box for me. I have found this procedure, which I applied on my system. I hope, that I didn't open the Pandora's box instead :).
But after this, the wu started to download additional pkgs and now the computation is running. I will see, if it will succeed, but so far all 6 cores (12 threads) runs at full speed (with some small slowdowns from time to time).
So again thx for help.

@ Toni, when these work units run do they get to a certain point then just idle along on a single core for hours on end?

I finally got a work unit to download to my Fedora 25 host, and it ran fine up to about 6 hours or so run time and 78.698% completed.
After this it has been running on a single core for almost 8 hours now and the progress is still locked at 78.698%.
The time to completion has increased from 1 hour 39 minutes to 3 hours 44 minutes and counting.

What happened to the Multi-Threading I thought these work units were supposed to do?

Run time is now approaching 14 hours and the % done has not moved, this is on a 16 core computer.

Conan

It is now running on 8 cpus instead of 16 which had stopped other work for a day.
Will now see what happens.

If these workunits are gonna take an average of Fifty five hours of CPU time, they really shouldn't crash when I reboot the system.

https://www.gpugrid.net/workunit.php?wuid=12932734

Needed to apply updates. Waited until one finished uploading before risking it. Glad I waited. This one was active for less than five minutes.


Select Language​▼

Twitter
Facebook
Follow us on:
|
Server status
|
Dayle Diamond [log out]

logo
About Science Volunteers Performance Forum Join us Donate

Name c448-TONI_QMML314rst-0-1-RND2050_0
Workunit 12932734
Created 18 Dec 2017 | 17:32:38 UTC
Sent 18 Dec 2017 | 21:17:59 UTC
Received 20 Dec 2017 | 1:26:41 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 195 (0xc3) EXIT_CHILD_FAILED
Computer ID 453935
Report deadline 23 Dec 2017 | 21:17:59 UTC
Run time 10.36
CPU time 6.37
Validate state Invalid
Credit 0.00
Application version Quantum Chemistry v3.14 (mt)
Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
17:21:56 (70558): wrapper (7.7.26016): starting
17:21:56 (70558): wrapper (7.7.26016): starting
17:21:56 (70558): wrapper: running ../../projects/www.gpugrid.net/Miniconda3-4.3.30-Linux-x86_64.sh (-b -f -p /var/lib/boinc-client/projects/www.gpugrid.net/miniconda)
Python 3.6.3 :: Anaconda, Inc.
17:22:04 (70558): miniconda-installer exited; CPU time 6.370986
17:22:04 (70558): wrapper: running /var/lib/boinc-client/projects/www.gpugrid.net/miniconda/bin/python (pre_script.py)

CondaValueError: prefix already exists: /var/lib/boinc-client/projects/www.gpugrid.net/miniconda/envs/qmml

forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libpcm.so.1 00007F3B35B1D725 Unknown Unknown Unknown
libpcm.so.1 00007F3B35B1B347 Unknown Unknown Unknown
libpcm.so.1 00007F3B35A32AA2 Unknown Unknown Unknown
libpcm.so.1 00007F3B35A328F6 Unknown Unknown Unknown
libpcm.so.1 00007F3B35A00EFD Unknown Unknown Unknown
libpcm.so.1 00007F3B35A04298 Unknown Unknown Unknown
libpthread.so.0 00007F3B48D8E150 Unknown Unknown Unknown
libmkl_def.so 00007F3B2293D916 Unknown Unknown Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpcm.so.1 00007F3B35A04B9A Unknown Unknown Unknown
libpthread.so.0 00007F3B48D8E150 Unknown Unknown Unknown

Stack trace terminated abnormally.
SIGSEGV: segmentation violation
Stack trace (11 frames):
../../projects/www.gpugrid.net/wrapper_26198_x86_64-pc-linux-gnu[0x457672]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13150)[0x7fa252998150]
../../projects/www.gpugrid.net/wrapper_26198_x86_64-pc-linux-gnu[0x494313]
../../projects/www.gpugrid.net/wrapper_26198_x86_64-pc-linux-gnu[0x4905b5]
/lib/x86_64-linux-gnu/libc.so.6(+0x37140)[0x7fa2525dc140]
/lib/x86_64-linux-gnu/libc.so.6(nanosleep+0x58)[0x7fa25267db98]
/lib/x86_64-linux-gnu/libc.so.6(usleep+0x44)[0x7fa2526b0134]
../../projects/www.gpugrid.net/wrapper_26198_x86_64-pc-linux-gnu[0x467f2f]
../../projects/www.gpugrid.net/wrapper_26198_x86_64-pc-linux-gnu[0x40b1a1]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fa2525c61c1]
../../projects/www.gpugrid.net/wrapper_26198_x86_64-pc-linux-gnu[0x407ca2]

Exiting...
17:24:57 (1324): wrapper (7.7.26016): starting
17:24:57 (1324): wrapper (7.7.26016): starting
17:24:57 (1324): wrapper: running /var/lib/boinc-client/projects/www.gpugrid.net/miniconda/bin/python (pre_script.py)

CondaValueError: prefix already exists: /var/lib/boinc-client/projects/www.gpugrid.net/miniconda/envs/qmml


CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.continuum.io/pkgs/main/linux-64/repodata.json.bz2>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
ConnectionError(MaxRetryError("HTTPSConnectionPool(host='repo.continuum.io', port=443): Max retries exceeded with url: /pkgs/main/linux-64/repodata.json.bz2 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f19ff5e8898>: Failed to establish a new connection: [Errno -2] Name or service not known',))",),)


Traceback (most recent call last):
File "pre_script.py", line 13, in <module>
raise Exception("Error installing h5py")
Exception: Error installing h5py
17:24:58 (1324): $PROJECT_DIR/miniconda/bin/python exited; CPU time 0.257581
17:24:58 (1324): app exit status: 0x1
17:24:58 (1324): called boinc_finish(195)

</stderr_txt>
]]>

About
Science
Volunteers
Performance
Forum
Join us
Contact

Google+ Facebook Twitter
© 2017 Universitat Pompeu Fabra

I finally finished a task so I can post now.

Can someone explain what the QC app shows for Status in the BOINC Manager. I had a app_config.xml loaded to limit the number of cpu cores it was supposed to use to 4. However in the Status column it showed 16C for the number of cores allotted.

Is that normal? Is that just how it describes itself to BOINC or was it really using all 16 cores?

This was my app_confg.xml

I finally finished a task so I can post now.

Can someone explain what the QC app shows for Status in the BOINC Manager. I had a app_config.xml loaded to limit the number of cpu cores it was supposed to use to 4. However in the Status column it showed 16C for the number of cores allotted.

Is that normal? Is that just how it describes itself to BOINC or was it really using all 16 cores?

This was my app_confg.xml

<app_config>
<app>
<name>acemdlong</name>
<max_concurrent>1</max_concurrent>
<gpu_versions>
<gpu_usage>1</gpu_usage>
<cpu_usage>1</cpu_usage>
</gpu_versions>
</app>
<app>
<name>acemdshort</name>
<max_concurrent>1</max_concurrent>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1</cpu_usage>
</gpu_versions>
</app>
<app>
<name>QC</name>
<max_concurrent>1</max_concurrent>
</app>
<app_version>
<app_name>QC</app_name>
<plan_class>mt</plan_class>
<avg_ncpus>4</avg_ncpus>
<cmdline>--nthreads 4</cmdline>
</app_version>
</app_config>

Does anyone see anything wrong with the app_config?

Can someone explain what the QC app shows for Status in the BOINC Manager. I had a app_config.xml loaded to limit the number of cpu cores it was supposed to use to 4. However in the Status column it showed 16C for the number of cores allotted.

Is that normal? Is that just how it describes itself to BOINC or was it really using all 16 cores?

After about 10h and reaching 69.568% app started to use only one core. What's worst it stays in that state for another 10h and perf is indicating that it's in OMP spinlock:

83.49% python libiomp5.so [.] __kmp_wait_yield_4
6.76% python libiomp5.so [.] __kmp_eq_4
5.74% python libiomp5.so [.] __kmp_yield
0.66% python [kernel.vmlinux] [k] entry_SYSCALL_64
...

Edit: On second machine it looks similar but after 6h and 78.698% it stays in that state for about 11h now. Perf:

84.60% python libiomp5.so [.] __kmp_wait_yield_4
6.80% python libiomp5.so [.] __kmp_eq_4
5.77% python libiomp5.so [.] __kmp_yield
0.59% python [kernel.vmlinux] [k] entry_SYSCALL_64
0.37% python [kernel.vmlinux] [k] __schedule
...

Even on a single core, they should not take longer than 20h maximum (on a relatively modern PC).

Two have just completed on my Ryzen 1700 (4 cores each). The elapsed time shows as 4 hours 10 minutes, but the CPU time is over two days.
http://www.gpugrid.net/results.php?hostid=452287

Can someone explain what the QC app shows for Status in the BOINC Manager. I had a app_config.xml loaded to limit the number of cpu cores it was supposed to use to 4. However in the Status column it showed 16C for the number of cores allotted.

Is that normal? Is that just how it describes itself to BOINC or was it really using all 16 cores?

The app_config looks the same as mine. Did you reboot in order to activate it?

In some of these multi-core projects, the Status is not updated until the next group of work units comes in after you have set the app_config. But a reboot usually fixes it.

But I could swear I had the app_config in place before I finally snagged my first two tasks. Will wait and see when I can get my next task.

The official BOINC Manager has an option to reread config files as well. I use BOINC Tasks and a new/updated file is picked up without a reboot or client restart.

There is also an option in the BOINC code that allows for the number of cpus that you want to use per host.
Each host can have a different setting.

Ask over at Amicable Numbers as they have done that there.
Then you wont need app_config.xml files at all.

Conan

Setting CPU % in BOINC is system and project wide. Not very good for fine tuning per project. The app_config was specifically introduced for specific project tuning and is the preferred method to control gpu and cpu usage per application.

I have the cpu cores limited in my app_config for both the ACEMD and QC apps. I was wondering why it wasn't picked up after all config files were re-read.

There is also an option in the BOINC code that allows for the number of cpus that you want to use per host.
Each host can have a different setting.

Ask over at Amicable Numbers as they have done that there.
Then you wont need app_config.xml files at all.

Conan

Setting CPU % in BOINC is system and project wide. Not very good for fine tuning per project. The app_config was specifically introduced for specific project tuning and is the preferred method to control gpu and cpu usage per application.

I have the cpu cores limited in my app_config for both the ACEMD and QC apps. I was wondering why it wasn't picked up after all config files were re-read.
Keith Myers

I am not referring to the Boinc Client on your personal computer.
My comments were aimed at the BOINC Server Code and therefore are relevant to fine tuning per project.
The option I am referring to is meant for Multiple Threading, so you can set the number of cores that you want to run a MT work unit on.

I understood what you are saying. The only way that works is if you set up different venues for different projects. I work primarily at SETI and Einstein. The venue mechanism does not work correctly and will likely never be updated. Very low chance that any major rework of the BOINC server code happens in the future with the lack of developers.

I am very comfortable with writing and editing app_info and app_config. Been doing it for a very long while. App_config is the simplest way to tune for individual projects as long as you are using a later version of the Client.

I also run more than one project simultaneously which makes your solution unworkable.

I just had to restart the BOINC client and the QMML work unit started back from 0% "fraction done" even though it had a checkpoint time of ~130000 and was at ~65%. Boo.

name: c457-TONI_QMML314rst-0-1-RND3080_4
WU name: c457-TONI_QMML314rst-0-1-RND3080
project URL: http://www.gpugrid.net/
received: Thu Dec 21 01:43:45 2017
report deadline: Tue Dec 26 01:43:44 2017
ready to report: no
got server ack: no
final CPU time: 0.000000
state: downloaded
scheduler state: scheduled
exit_status: 0
signal: 0
suspended via GUI: no
active_task_state: EXECUTING
app version num: 314
checkpoint CPU time: 135211.300000
current CPU time: 138876.400000
fraction done: 0.010989
swap size: 1236 MB
working set size: 304 MB
estimated CPU time remaining: 747697.676876

If another one would be made available, I could try. I only ever see one task ready to be snagged. Just got one. Happy to report the change in allowed cores limit was properly applied after I changed 4 to 4.0000.

Why is there such a big credit difference?



16794335 12932706 25 Dec 2017 | 10:46:18 UTC 26 Dec 2017 | 5:21:03 UTC Fertig und Bestätigt 66,830.38 199,504.30 440.38 Quantum Chemistry v3.14 (mt)

16792051 12932673 24 Dec 2017 | 10:37:41 UTC 25 Dec 2017 | 5:22:21 UTC Fertig und Bestätigt 67,429.09 200,424.90 1,627.77 Quantum Chemistry v3.14 (mt)

I just grabbed 2 QC tasks and I will attempt to run them simultaneously tomorrow during the SETI outage.

These two WU's ran concurrently on one of my FX8350 using 4 cores each. They were started within about twenty minutes of each other and finished about 4 minutes apart.

16795026 12932509 426610 25 Dec 2017 | 15:47:53 UTC 27 Dec 2017 | 12:23:12 UTC Completed and validated 53,395.91 188,995.80 3,034.10 Quantum Chemistry v3.14 (mt)

16795025 12932492 426610 25 Dec 2017 | 15:47:53 UTC 27 Dec 2017 | 12:19:03 UTC Completed and validated 53,957.92 193,510.10 3,066.04 Quantum Chemistry v3.14 (mt)

I just grabbed 2 QC tasks and I will attempt to run them simultaneously tomorrow during the SETI outage.

Toni, well there's your answer if you discount the low population sample. I doubt that two successful runs were due to the brand of cpu, but could be wrong. As long as you are using a Client later than 7.0.40 you can use an app_config.xml file to tune the number of cores you allow the task to run on.

My Ryzen 1700 machine has certainly done better than my two i7-3770 PCs (all machines on Ubuntu, run 24/7 and otherwise set up the same):

Ryzen 1700: http://www.gpugrid.net/results.php?hostid=452287

i7-3770:
http://www.gpugrid.net/results.php?hostid=433866
http://www.gpugrid.net/results.php?hostid=448995

EDIT: My i7-4770 PC also tended to hang or otherwise fail.
http://www.gpugrid.net/results.php?hostid=357332

I have often gotten hung work units on the Intel machines, but seldom or never on the AMD. And looking around at the other users who fail the work units, they seem to be predominantly Intel, while the ones that succeed seem to be AMD, though I have not done a count myself. Presumably Toni can get those figures.

Thanks for the post. Interesting. I could discount the fact that the Ryzen's have actual 8 physical cores so on paper a good head start over the Intel 4 core cpu's. But the FX-9350 earlier in the thread had a good result too with only 4 physical cores too and much more handicapped FFT registers in its modules compared to Ryzen and Intel. Need a lot more samples to definitively clarify I think.

Need a lot more samples to definitively clarify I think.

Good experiment. I was just thinking that spreading the compute load over 4 cores is less hard compared to 100% workload over 4 cores on Intel. If the test on 2 and one cores is equally stable, just slower, it might suggest something bothersome on Intel architecture. The different OS platform could have a big effect too.

It appears one cannot start two or more of these type of WU's simultaneously. One of the two errored as shown below.

16803962 12952984 426610 29 Dec 2017 | 16:16:51 UTC 29 Dec 2017 | 17:15:51 UTC Completed and validated 916.51 6,109.03 35.20 Quantum Chemistry v3.14 (mt)
16803949 12952971 426610 29 Dec 2017 | 16:16:51 UTC 29 Dec 2017 | 17:26:30 UTC Error while computing 5.04 0.00 --- Quantum Chemistry v3.14 (mt)

So I started two more but with a five second delay and they both are now happily processing together on one of my FX8350's 4 cores each. With that in mind, the boinc client may not be relied upon to run these cpu jobs unattended using a split cpu core configuration to allow multiple WU processing lest two or more were to start simultaneously causing a possible failure on at least one WU.

Edit: Will try a simultaneous start with my last two WU's to see if this is repeatable.

Edit2: Yep, happened again. Will copy the pair to this post when the one in progress finishes.

These are the last two, one of which errored:

16804044 12953066 426610 29 Dec 2017 | 17:28:46 UTC 29 Dec 2017 | 21:02:26 UTC Error while computing 6.06 0.00 --- Quantum Chemistry v3.14 (mt)
16804026 12953048 426610 29 Dec 2017 | 17:28:46 UTC 29 Dec 2017 | 21:24:44 UTC Completed and validated 1,393.35 5,255.89 60.62 Quantum Chemistry v3.14 (mt)

It appears one cannot start two or more of these type of WU's simultaneously.

Looks like that multiple WUs together is ok, but starting exactly at the same time is not. I hope it is a relatively rare occurrence. In principle I could put a locking mechanism, but I am not enthusiastic because that would be inviting more failure modes (e.g. stale locks) to solve a relatively rare case.

Looks like that multiple WUs together is ok, but starting exactly at the same time is not. I hope it is a relatively rare occurrence. In principle I could put a locking mechanism, but I am not enthusiastic because that would be inviting more failure modes (e.g. stale locks) to solve a relatively rare case.

So every new member crunching these will get an error on their 1st task. Genius. It absolutely should be fixed.

I would speculate that the probability of two or more WU's starting at exactly the same time in an unattended environment would be low however, those running multiple projects competing for the same GPU/CPU resources are usually switched at a specified time interval to give each project process time. With that in mind, it is possible for two QC jobs to finish and the boinc client nearing the end of a switch app interval, change to the other project and then when it came time to switch back to run the cpu gpugrid WU's with a queue full of QC jobs start two or more simultaneously.

The only way I can see avoiding the possibility of such an error completely would be to keep all cores dedicated to one QC job on unattended machines (headless crunchers in my case). When the next batch is available, I will cut my switch app time down to say 5 minutes and see if I can get a feel for how the client handles things and the possibility of simultaneous starts with the QC jobs. The two jobs that failed due to simultaneous starts both ended with exit code 195 EXIT_CHILD_FAILED. Not sure if that means the app failed to spawn a child thread or close one for one of the two WU's.

You are right that BOINC is not really a random environment, and if it happens in a more-or-less predictable manner, it should be possible to prevent it. We will see how often that is.

You are right that BOINC is not really a random environment, and if it happens in a more-or-less predictable manner, it should be possible to prevent it. We will see how often that is.

When there is work or the GPUGrid servers sent out work they come in batches. Two tasks end up starting at once since they have short deadlines. I also said new members so again multiple tasks starting at once. Not everyone runs the same project all the time so that tasks have a chance to get off sequence.

I have changed one of the i7-3770 machines (GTX-1070-PC) from Ubuntu 17.10 (and BOINC 7.8.3) back to Ubuntu 16.04 (and BOINC 7.6.31).
I doubt that it will make much difference, but I will let it run for a couple of weeks. If I continue to get more errors on Intel, I think I will go with just the Ryzen PC.

I just ran a DOMINIK QC task and it ran very fast.
Task 16815024

Anybody else run one of these yet? I see that there are a TON of them available.

Server Status

50,000 WUs... Holy ****. If only those were GPU WUs

50,000 WUs... Holy ****. If only those were GPU WUs

So far from what I have learned starting more than one multiple cpu job at a time in a split core senerio is that they need not be started at exactly the same time for one to error. Given not an exact start at the same time, the one started first always errors and the second one started processes to successful completion with the next WU in the queue started to also complete successfully. Four of four tries, near simultaneous starts the one started first ended up failing. Secondly, controlling the time to when the second WU is allowed to start following the first start time is up to 5 seconds as tested so far. Third, when the boinc client switches between projects, the QC WU's so far observed are completed in pairs leaving no single job left in progress (suspended) to stagger start the times. This unfortunate characteristic means "simultaneous (or nearly so) starts" cause an error whenever the client switches back to the gpugrid cpu jobs with a queue larger than one WU. Guess the only way to prevent this behavior is to not split cores, especially for unattended clients.

Edit: correct my lousy spelling, stupid keyboard :)

I didn't have that experience with the two TONI tasks I started simultaneously. Or within the 5 second window you described. Both completed successfully. I am limiting core usage to four with an app_config file. Limiting the max_concurrent to 1 now since I also crunch SETI cpu tasks on that computer. I ran the two concurrent jobs when Toni requested users to try that experiment.

Wow, the credit awarded is all over the place for these DOMINIK tasks. Obviously NOT tied to computation time or resources used for compute.

Task 16862848
3715 seconds CPU time Credit awarded 21

Task 16815159
3687 seconds CPU time Credit awarded 161

I just got a task and it finished on a dual e5-2450l 32g ram server fine.
resultid=16815237
Keep developing this, please. This is quite nice and I would really like to see this as part of GPUGRID permanently.

resultid=16815470

Hello Keith,

Wow, the credit awarded is all over the place for these DOMINIK tasks. Obviously NOT tied to computation time or resources used for compute.

All tasks were erroring out in 2 minutes due to the app using gcc5.5

This got it to go to farther and start being multithreaded. We'll see if it actually completes.

Yup, it completed.
http://www.gpugrid.net/result.php?resultid=16817115

Great! Thank you very much

@klepel - can you try installing gcc (if not already there)?

tks

These are the two WU's that were started about 5 seconds apart on an AMD FX-8350 with the first one started failing.

Stdoutdea.txt:
03-Jan-2018 17:20:14 [GPUGRID] [css] running e113s22_e86s4p0f123-PABLO_p53_PHEX10P_IDP-0-1-RND2720_0 (0.987 CPUs + 1 NVIDIA GPU)
03-Jan-2018 17:20:14 [GPUGRID] Starting task c00000_00024-DOMINIK_QMML2_m0000000055-0-1-RND3244_0
03-Jan-2018 17:20:14 [GPUGRID] [cpu_sched] Starting task c00000_00024-DOMINIK_QMML2_m0000000055-0-1-RND3244_0 using QC version 314 (mt) in slot 9
03-Jan-2018 17:20:14 [GPUGRID] [css] running c00000_00024-DOMINIK_QMML2_m0000000055-0-1-RND3244_0 (4 CPUs)
03-Jan-2018 17:20:20 [GPUGRID] task c06475_06499-DOMINIK_QMML2_m0000000054-0-1-RND3067_0 resumed by user
03-Jan-2018 17:20:21 [GPUGRID] [css] running e113s22_e86s4p0f123-PABLO_p53_PHEX10P_IDP-0-1-RND2720_0 (0.987 CPUs + 1 NVIDIA GPU)
03-Jan-2018 17:20:21 [GPUGRID] [css] running c00000_00024-DOMINIK_QMML2_m0000000055-0-1-RND3244_0 (4 CPUs)
03-Jan-2018 17:20:21 [GPUGRID] Starting task c06475_06499-DOMINIK_QMML2_m0000000054-0-1-RND3067_0
03-Jan-2018 17:20:21 [GPUGRID] [cpu_sched] Starting task c06475_06499-DOMINIK_QMML2_m0000000054-0-1-RND3067_0 using QC version 314 (mt) in slot 10
03-Jan-2018 17:20:21 [GPUGRID] [css] running c06475_06499-DOMINIK_QMML2_m0000000054-0-1-RND3067_0 (4 CPUs)
03-Jan-2018 17:20:25 [GPUGRID] [sched_op] Deferring communication for 00:01:39
03-Jan-2018 17:20:25 [GPUGRID] [sched_op] Reason: Unrecoverable error for task c00000_00024-DOMINIK_QMML2_m0000000055-0-1-RND3244_0
03-Jan-2018 17:20:25 [GPUGRID] Computation for task c00000_00024-DOMINIK_QMML2_m0000000055-0-1-RND3244_0 finished
03-Jan-2018 17:20:25 [GPUGRID] [css] running e113s22_e86s4p0f123-PABLO_p53_PHEX10P_IDP-0-1-RND2720_0 (0.987 CPUs + 1 NVIDIA GPU)
03-Jan-2018 17:20:25 [GPUGRID] [css] running c06475_06499-DOMINIK_QMML2_m0000000054-0-1-RND3067_0 (4 CPUs)

I didn't have that experience with the two TONI tasks I started simultaneously. Or within the 5 second window you described. Both completed successfully. I am limiting core usage to four with an app_config file. Limiting the max_concurrent to 1 now since I also crunch SETI cpu tasks on that computer. I ran the two concurrent jobs when Toni requested users to try that experiment.

Hello Keith,

Wow, the credit awarded is all over the place for these DOMINIK tasks. Obviously NOT tied to computation time or resources used for compute.

Did you observe this behavior multiple times? Really strange to be honest.


Thanks for helping out everyone!

16862848 12962574 456812 3 Jan 2018 | 23:38:10 UTC 4 Jan 2018 | 1:20:46 UTC Completed and validated 1,000.69 3,715.60 21.12 Quantum Chemistry v3.14 (mt)
16815333 12963093 456812 4 Jan 2018 | 1:00:00 UTC 4 Jan 2018 | 2:57:45 UTC Completed and validated 1,020.00 3,812.63 27.38 Quantum Chemistry v3.14 (mt)
16815332 12963092 456812 4 Jan 2018 | 0:59:23 UTC 4 Jan 2018 | 2:40:47 UTC Completed and validated 990.38 3,697.44 25.78 Quantum Chemistry v3.14 (mt)
16815320 12963080 456812 4 Jan 2018 | 1:00:37 UTC 4 Jan 2018 | 3:14:28 UTC Completed and validated 997.66 3,731.61 27.28 Quantum Chemistry v3.14 (mt)
16815307 12963067 456812 4 Jan 2018 | 1:07:01 UTC 4 Jan 2018 | 4:04:01 UTC Completed and validated 1,009.28 3,642.48 26.40 Quantum Chemistry v3.14 (mt)
16815275 12963035 456812 4 Jan 2018 | 1:07:38 UTC 4 Jan 2018 | 4:21:09 UTC Completed and validated 1,033.63 3,668.31 26.52 Quantum Chemistry v3.14 (mt)
16815264 12963024 456812 4 Jan 2018 | 1:06:24 UTC 4 Jan 2018 | 3:46:54 UTC Completed and validated 969.15 3,592.69 25.72 Quantum Chemistry v3.14 (mt)
16815248 12963008 456812 4 Jan 2018 | 0:58:46 UTC 4 Jan 2018 | 2:24:17 UTC Completed and validated 935.31 3,503.52 23.59 Quantum Chemistry v3.14 (mt)
16815234 12962994 456812 4 Jan 2018 | 1:05:48 UTC 4 Jan 2018 | 3:30:44 UTC Completed and validated 981.30 3,616.45 26.43 Quantum Chemistry v3.14 (mt)
16815171 12962931 456812 3 Jan 2018 | 23:37:35 UTC 4 Jan 2018 | 1:04:10 UTC Completed and validated 929.82 3,523.44 18.45 Quantum Chemistry v3.14 (mt)
16815159 12962919 456812 3 Jan 2018 | 23:35:43 UTC 4 Jan 2018 | 0:16:22 UTC Completed and validated 986.88 3,687.96 161.36 Quantum Chemistry v3.14 (mt)

I have to report back on the AMD Ryzen 1700x Computer: http://www.gpugrid.net/results.php?hostid=420971

If I run 3 instances, the WUs crashes after about 200 seconds, and after that the computer crashes completely. If I am running one (01) instance (WU), the computer runs without any problem.

However, as BOINC downloads several of this Quantum Chemistry v3.14 (mt) WUs, BOINC thinks my CPU cache is full and refuses to download additional CPU WUs from PRIMGRID. So after a while the CPU is only loaded with one QC WU (4 threads) and the rest of the cores are idle - Not very efficient.

Sorry, Dominik unter diesen Umständen kann ich keine weiteren QC WUs für diesen Computer herunterladen. Komme aber gerne zurück, wenn wir ohne Probleme mehrere MultiCores WUs gleichzeitig bearbeiten können.

However, as BOINC downloads several of this Quantum Chemistry v3.14 (mt) WUs, BOINC thinks my CPU cache is full and refuses to download additional CPU WUs from PRIMGRID. So after a while the CPU is only loaded with one QC WU (4 threads) and the rest of the cores are idle - Not very efficient.

Seti@home has been down all day so ran out of work. Decided to give the QC Dominik tasks another try. Thought that possibly the last batch were unique or one-offs or something different about the computer from when I first ran them.

Nope. Even worse credit for the batch I ran this afternoon. Credits awarded = 6.

If you expect to get anybody to want to run these, you are going to have to make them more appealing, credit-wise at least. For me, not worth the electricity to run them. Would rather let the computer go cold and give my power bill a temporary reprieve.

Yeah credit took a dump on the last two I completed.

I got 5.97 points for 32 minutes calculation on my fx 6100.

That's ridiculous!

If you care to learn about why the low credit or why certain tasks get hi-middle-low credit, read my post here

The project is using the "old" BOINC credit algorithm. If I remember how it works, the very first few tasks on a new application that BOINC sees gives very high credit. Then when more tasks are returned the algorithm tunes the credit downwards. The APR for the application stabilizes after 11 valid tasks have been returned. This project utilizes the application APR function of BOINC. It is up to each project to decide whether to follow BOINC standards or write their own.

So far I have seen credit awards of 160, 21 and 7. That must be because those were 3 different molecules.

Seems to me that credit should be based on CPU time only. One WU should be worth a specific amount of credit regardless of how fast or slowly processed. Probably require a large bureaucratic committee to attempt to quantify such a value but that's my 2 cents and maybe it would be worth it if we get an answer within a few years :).

Edit: Added last sentence, sorry, couldn't resist.

Seems to me that credit should be based on CPU time only. One WU should be worth a specific amount of credit regardless of how fast or slowly processed. Probably require a large bureaucratic committee to attempt to quantify such a value but that's my 2 cents and maybe it would be worth it if we get an answer within a few years :).

Edit: Added last sentence, sorry, couldn't resist.

Just posting to say that I have finally gotten my 6 core Phenom CPU working on this project.
It has taken me about 3 months of errors and an OS upgrade from Fedora 16 to 21, before the error message changed, saying that now qmml/bin/python was not there.

So after comparing another working system and a huge number of updates, I noticed that a number of symlinks were missing from my computer.

I installed 7 symlinks, all python related.

Existing files........Missing files

pydoc3.6 >>>>>>>>>>> pydoc
pydoc3.6 >>>>>>>>>>> pydoc3
python3.6 >>>>>>>>>> python
python3.6 >>>>>>>>>> python3
pyvenv-3.6 >>>>>>>>> pyvenv
python3.6m-config >> python3.6-config
python3.6m-config >> python3-config

After installing all of these links it now has begun to work.

So Toni, hopefully no more errors.

Conan