Ready for action ;)

First 330 task completed and validated. The second one is waiting for memory alongside a GPU task, which is however using only 4% of the total 8 GB of RAM.
Tullio

@ Toni, @ Stefan

There's an error report in Number Crunching (This computer has finished a daily quota of 31 tasks) which suggests that the maximum upload size for a batch of QC tasks has been set too low.

Task name is 6955_1_15_16_18_dd130713_n00001-SDOERR_SELE2-0-1-RND2528

@ Toni, @ Stefan

There's an error report in Number Crunching (This computer has finished a daily quota of 31 tasks) which suggests that the maximum upload size for a batch of QC tasks has been set too low.

Task name is 6955_1_15_16_18_dd130713_n00001-SDOERR_SELE2-0-1-RND2528

Just curious, is there someplace that tells how well the QC apps are running like on the server status page. I can look at my own machines and see the errors there but overall is there someplace like for the GPU apps?

I think the actual error is a segmentation fault, which leaves large temporary files behind, and their transfer is attempted. Let's see if the situation improves with the new version.

May well be restricted to a few machines.

My main Linux box is crunching SELE6 on its Opteron 1210. If necessary, I have a Windows 10 PC with an AMD A10-6700 and 22 GB RAM.
Tullio
____________

again

<message>
upload failure: <file_xfer_error>
<file_name>5516_14_15_18_19_8125b500_n00001-SDOERR_SELE6-0-1-RND8343_0_1</file_name>
<error_code>-131 (file size too big)</error_code>
</file_xfer_error>

CPU apps not working very well. Lots of idle time. It maybe because of the 48 threads... some pythons use 4 cores, the rest just 1 and not all the time. It's gotta be an I/O issue (?).
____________

again

<message>
upload failure: <file_xfer_error>
<file_name>5516_14_15_18_19_8125b500_n00001-SDOERR_SELE6-0-1-RND8343_0_1</file_name>
<error_code>-131 (file size too big)</error_code>
</file_xfer_error>

Have i had to reset the project?

I have had good luck for the past day running QC on my i7-4770 (Ubuntu 16.04). That doesn't prove much, except that there is no fatal flaw in all the work units.

And I am limiting them to two cores per work unit, and three work units at a time. That gives me essentially the same output as four cores on two work units at a time, but leaves over a little more CPU support for my GTX 1070 on Folding. All in all, it seems to be working fine.

(boboviz - I wouldn't draw conclusions from virtual machines. Even LHC has a hard time with their own stuff.)

(and the wu re-starts)

Most LHC users are Windows users and they run Scientific Linux research programs from CERN (not BOINC programs) using Virtual Machines.
Tullio

This is interesting. Twice overnight my PC crashed. Shut down. Didn't run.
But each time, I saw no errors in the BoincTasks log, or in the Folding log either. The Folding work unit just continued from where it left off.

But each time, a new set of (three) QC tasks started after starting up the PC.

And now I see the same old error message in the stderr.txt file:

</stderr_txt>
<message>
upload failure: <file_xfer_error>
<file_name>2360_16_18_20_21_e0c95459_n00001-SDOERR_SELE6-0-1-RND3935_0_1</file_name>
<error_code>-131 (file size too big)</error_code>
</file_xfer_error>
</message>

(boboviz - I wouldn't draw conclusions from virtual machines. Even LHC has a hard time with their own stuff.)

(boboviz - I wouldn't draw conclusions from virtual machines. Even LHC has a hard time with their own stuff.)

Do you think i have to use a physical linux machine?

I am running GPUGRID,both CPU an GPU, on a SuSE Linux box with a GTX 750 Ti GPU board. I am running Atlas@home of LHC, on a Windows 10 PC with 4 cores (but Task Manager says two cores and 4 logical processors on a AMD A10-6700 CPU). It has a GTX 1050 Ti GPU board, but GPUGRID overheats it to 80 C and it crashes, so I am running Atlas (no GPU but VirtuaBox 5.2.18), Einstein@home and SETI@home, both CPU and GPU.
Tullio
Atlas native runs only on Ubuntu Linux,it does not run on my SuSE Linux nor on Windows.
____________

If it does, is the problem solved enabling the "keep application in memory" option?

So I'm curious. When we see the results for a work unit for QC it list both a run time and a cpu time. Since we are so used to run time and cpu time beling a linear measurement, I was wondering if CPU time for the QC is actually a combined sum total for all CPU threads being used.

I've calculated out how long the CPU time is and it's much higher than the actual run time I'm seeing on the machine. What does make sense is if it's the sum total of 4 threads all running at the time time for a set amount of time.

CPU time = N threads x actual run time

Is this correct? Mostly for my own curiosity.
____________

Since the utilization of the CPU is so low on these WUs I presume it doesn't count most of the run time as CPU time because the CPU is waiting for the hard drive.

I thought I would try my Ryzen 1700 on QC (Ubuntu 18.04), to see if it would behave differently than my Intel machines (i7-4770, i7-8700).
The first work unit errored:

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.anaconda.com/pkgs/r/noarch/repodata.json.bz2>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

If your current network has https://www.anaconda.com blocked, please file
a support request with your network engineering team.

ConnectionError(MaxRetryError("HTTPSConnectionPool(host='repo.anaconda.com', port=443): Max retries exceeded with url: /pkgs/r/noarch/repodata.json.bz2 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f60af4b15c0>: Failed to establish a new connection: [Errno -2] Name or service not known',))",),)

A reportable application error has occurred. Conda has prepared the above report.
Upload did not complete.10:07:33 (1830): /usr/bin/flock exited; CPU time 11.796829
10:07:33 (1830): app exit status: 0x1
10:07:33 (1830): called boinc_finish(195)

So I'm curious. When we see the results for a work unit for QC it list both a run time and a cpu time. Since we are so used to run time and cpu time beling a linear measurement, I was wondering if CPU time for the QC is actually a combined sum total for all CPU threads being used.

I've calculated out how long the CPU time is and it's much higher than the actual run time I'm seeing on the machine. What does make sense is if it's the sum total of 4 threads all running at the time time for a set amount of time.

CPU time = N threads x actual run time

Is this correct? Mostly for my own curiosity.

The QC jobs should use 4GB of RAM each. If you are swapping just don't run as many in parallel. You will never finish them anyway if you end up swapping.

Is it possible that the QC app is returning most but not all memory it uses back to the memory pool as calc's are completed?

That is an interesting question, and could explain some of the random errors I have been getting. But QC is running OK now on an i7-3770, running four work units at a time with 2 cores per WU. I see memory usage up around 4 GB per work unit though, so it is fortunate I have 32 GB. That has not prevented the errors in the past on comparable machines, but I have found that when it works, don't touch it.

While running 4 WU's 2 threads each, have you noticed any swap file usage with the 32 G RAM?

I should have gone full 32 GiB when I last upgraded RAM. Darn, I went 4 x 2 initially and later added 4 x 2 more so now I have to buy all 32 G (8 x 4) rather than just add 16 G more, to the tune of around 250 USD each FX box. My rule has been 2 G per thread/core but in this situation 4 G /thread appears minimum, in fact, all my ATX can take.

Just leave it at the default 4 cores per work unit.

Would be interesting to find out if the QC app is faithfully returning all memory used back to the system after each of the calc's.

(It is getting hard to post again, due to all the browser timeouts. It is a wonder I am able to get work at all.)

On the swap file issue on your machine, try swapoff -a as root or sudo. That for sure disables swapping. I use it to flush the swap. Use swapon -a to re-enable swap.

I have a GPU task and a CPU task running on my two cores Opteron 1210, 8 GB DDR2 RAM, GTX 750 Ti at 1202 MHz clock, 61 C. OS is SuSE Leap 42.3.Swap space is used at 37% of 2 GB. My HP Linux laptop, also running SuSE Leap 15.0 has a 7 GB swap space, not used. It is not running GPUGRID tasks because BOINC space is only 30 GB instead of the 760 GB of my main Linux host, a 2008 vintage SUN workstation, running 24/7 since January 2008. Hats off to SUN!
Tullio

Very good. I did sudo swapoff -a, and now swap shows as zero. We will see how it goes.

I have a GPU task and a CPU task running on my two cores Opteron 1210, 8 GB DDR2 RAM, GTX 750 Ti at 1202 MHz clock, 61 C. OS is SuSE Leap 42.3.Swap space is used at 37% of 2 GB. My HP Linux laptop, also running SuSE Leap 15.0 has a 7 GB swap space, not used.

I am using this computer to read my mail, to navigate the WWW, read the newspapers including the NYTimes which leaves me ten free articles/month and I feel no delay. I have a 30 Mbit/s mixed fiber/copper connection by Telecom Italy, which means the the fiber reaches a cabinet that is not far from my home, then I have a copper connection. My router is also connected by WiFi to a Windows 10 PC, a printer and a decoder which gives me SKY TV on the TV set which is also the monitor of the Windows PC. I just had a Microsoft update on the Windows 10 PC and it restarted with its two Einstein@home tasks.
Tullio
I have also a smartphone running Android 7.1.1 on its 8 cores 64ARM CPU, connected by WiFi to the router. It is running SETI@HOMe and Einstein@home CPU tsks.
____________

Doesn't happen alot but i still am getting these occasionally that cause errors.

</stderr_txt>
<message>
finish file present too long
</message>
]]>

Doesn't happen alot but i still am getting these occasionally that cause errors.

</stderr_txt>
<message>
finish file present too long
</message>
]]>

Toni asked:

General question to windows users: do you see "black windows" like a command prompt coming up when running QC apps?

Toni asked:

Regarding threads - the python app is indeed QC. Are you running multiple WUs simultaneously, or just one WU occupied all 16 threads?

Just had 8 tasks error out with the following error code.

SafetyError: The package for hdf5 located at /var/lib/boinc-client/projects/www.gpugrid.net/miniconda/pkgs/hdf5-1.10.2-hba1933b_1
appears to be corrupted. The path 'lib/libhdf5.so.101.1.0'
has a sha256 mismatch.
reported sha256: d8628337423317dafe2d7f1f5029bfbb5cd22428fbf97e81678cc0db0e93c2c2
actual sha256: 6ef2e91ed97113943149adc62ad53c6d831db045c015176b13d3f7fd8f9e1c0f

Looks like we have made headway on these work units. Just over a thousand left. Will be interesting to see if they bring back the other set.
____________

Will be interesting to see if they bring back the other set.

No worries, it was more of a comment than a compliant. Enjoy your conference, we aren't going anyplace hahaha...
____________

Are you give us any comment about results of this batch? Was it usefull?

Ok, Thanks Stefan. Happy New Years
____________

I everything goes smooth I ought to have the last workunits out by tomorrow, or latest the day after.

That is perfectly OK. It is much better for projects to tell us what is going on than to leave us hanging. You never know if they are dead or alive. At least you are alive.

which machine do you use for calculating the WU? Just in case some of us are willing to donate some money for hardware upgrades at your institute.
____________
I would love to see HCF1 protein folding and interaction simulations to help my little boy... someday.

Does QC still have a problem when multiple WUs start at the same time???

I loaded Rosetta jobs and watched them in System Monitor. They appeared to load one by one. The first CPU ramped up to 100% and then a minute or so later the second ramped up... I wonder if they have a solution they'd share.

I'm thinking of a script that could replicate that behavior.
QC WUs DL. The app_config.xml file could be iteratively modified:

Aurum asked:

Does QC still have a problem when multiple WUs start at the same time???

Great. How much RAM do I need per QC WU???
Rosetta needs between 750 MB to 1 GB.

Is there an L3 Cache requirement for QC WUs???

E.g. at WCG the MIP project require 4-5s MB per WU. It runs with less but speed drops fast if overloaded.

Wow! This is a major RAM hog. I just watched while 6 4C WUs consumed 16 GB and crashed a computer. Then another had 3 4C WUs running and it stopped one them as Suspended:Wating for memory, then a second one and third is still using 12 of 16 GB. Then they all 3 come back on and use all the memory. Then start failing for computation error.

We may need to reserve 3 GB per C, i.e. run only one 4C WU per 16 GB RAM.
So I'm adding this to my app_config:

I got two of them. One took 2.5 GB, and the other 3.5 GB. Send more. I need to use the 32 GB of memory on my i7-4770.
And longer ones are OK; these took only 12.75 minutes. I have no idea about disk usage though.

I have some TONI WUs that have very low CPU usage and very low RAM usage.

Another thing I noticed is my GPU usage, normally at 100% running GPUGrid is now 5-15% and the CPU is not anywhere near 100% usage.

Update:

I just suspended and resumed the GPU WU and now it's back at 100% usage

Today I'll upload a new batch. I have no real clue on the technical requirements though. It will be try-and-see because these are some last large molecules I need to run. After these large ones I'll send some last small and fast ones again.

No joy with my computers in getting any work.

Can you set it so that each Host gets one beta before a single Host gets 87 ???

No joy with my computers in getting any work.

Do you have "Run test applications?" selected in your preferences? I am getting them fairly regularly now.

Each BOINC slot with a QC job is about 29 GB, so plan accordingly. I am running two now, which uses all eight cores and still leaves 29 GB free, so I think I am safe for now.

I will need to raise the limit (the production QC app is 60 GB).

Two tasks have "restarted" themself and started run from th beginning when they achieved about 40%. Just before it my CPU has heated suddenly to over 80 Celsius degrees, fortunately only for a moment.

The default BOINC settings for disk memory are far too low for your experiment. The RAM requirements are all over the map. You're sending these jobs to Xeons with 24 to 44 logical CPUs that can only run one or two at a time.

Why don't these WUs respect the client's resources??? E.g., if I have 16 GB RAM you'll start 5 or 6 WUs and consume all RAM and the computer freezes. If I have an app_config.xml set limiting QC or QC_beta to one WU (4C needs 16 GB RAM) then you want 60 GB of disk memory which is fine if I've set BOINC high enough.

Did you think this through before you launched this mayhem??? You should be able to tell us what these requirements are before launching. 4 GB per CPU thread per WU = 16 GB RAM is 4 times higher than Rosetta and by far the highest I've seen.

This is the first time in 5 hours I've been able to access this website.

The default BOINC settings for disk memory are far too low for your experiment.

My windows machine is doing QC WUs, is this supposed to happen?

PappaLito said:

My windows machine is doing QC WUs, is this supposed to happen?

So about half are failing the other half are finishing. They are using all of the threads on my CPUs. I had a max concurrent in my app_config but I think these are a different name so that limitation is being ignored. I should have tried a <project_max_concurrent> but didn't have time to play with it to see if it would restrict the number running. Guess that will have to wait until Saturday when I can re-access those machines.
____________

So about half are failing the other half are finishing. They are using all of the threads on my CPUs.

I changed my mind about submitting the other monsters, I'll try to run them on our local cluster first.

After some times i restart my linux box and.... 226 mb of upload of a "Toni" quantum chemistry wu!!
Ouch

The QC per core RAM load has dropped. What's today's requirement???

What are the requirements to receive Regular Non BETA QC Wu's ??? I have never received 1 even though I have been calling for them all along. I can get the BETA QC Wu's so one would think I should be able get the Regular QC Wu's. Running Windows 10 Pro with 64gb's Memory on a Intel i7 7770k CPU.

When my Box calls for Wu's I get this:

PBOYZTOYNP9873 13027 GPUGRID 1/12/2019 4:35:13 AM No tasks are available for Quantum Chemistry


Server Status shows Quantum Chemistry 125,158 Unsent ... Thanks
____________
STE\/E

STEVE, Have you gone to BOINCmgr/Options/Computing Preferences/Disk and Memory??? The combination of those 3 checkboxes under Disk must yield over 60 GB of free disk space. May be 60 GB per QC WU.

In my case I was getting QCs but they seem to have stopped over night. No idea why.

How many months will it take to clear 125,000 WUs???

The combination of those 3 checkboxes under Disk must yield over 60 GB of free disk space. May be 60 GB per QC WU.

STEVE, Have you gone to BOINCmgr/Options/Computing Preferences/Disk and Memory??? The combination of those 3 checkboxes under Disk must yield over 60 GB of free disk space. May be 60 GB per QC WU.

In my case I was getting QCs but they seem to have stopped over night. No idea why.

How many months will it take to clear 125,000 WUs???

STEVE asked,

What are the requirements to receive Regular Non BETA QC Wu's ???

STEVE asked,

What are the requirements to receive Regular Non BETA QC Wu's ???

As far as I know, regular non BETA QC WU's are still Linux only. I am getting plenty on my Linux box, none on a Windows box.

STEVE,

I prefer this line of thinking, general business users usually run Windows, graphics artist and publishers usually run Macs, and scientists usually run Linux. Almost all of the top 500 super-computers in the world run on Linux. I'm guessing that most of the scientific research applications natively run on Linux. It was easier for the GPUGRID scientist to get the QC CPU app running on Linux PC's so that was the initial release.

They tried to get the QC CPU to run on Windows a few months ago using the Windows Subsystem for Linux (WSL) but ran into problems with inconsistency between Windows releases. This is their next attempt to get the QC app to run on Windows. Looks like this one might work when they get the parameters set up for the size of tasks involved.

Well I tried to install a <project_max_concurrent> into the app_config but the beta work units just ignore that restriction. They were utilizing all 20 threads on the computer and starving the GPUs for resources. So I've stopped the beta for now on the machines. I have to rebuild a computer anyway this week so I think I will strip out the GPUs and leave the 16 thread CPU and install a 4 TB HDD. That should be enough I think for the beta work units and I will leave that machine only for those work units.
____________

Well I tried to install a <project_max_concurrent> into the app_config but the beta work units just ignore that restriction.

That has worked for me in the past.

Using <project_max_concurrent> is a problem since it counts all GG WUs. Also, they ran out of GPU WUs yesterday. The control of CPU & GPU projects should be completely separate.

Well I tried to install a <project_max_concurrent> into the app_config but the beta work units just ignore that restriction.

You could try:

<app_config>
<app>
<name>QC_beta</name>
<max_concurrent>4</max_concurrent>
</app>
</app_config>

That has worked for me in the past.

Two tasks have been frozen at 1,098% despite they ahievied it very quickly. I dont't think the app is good, yet...

Two tasks have been frozen at 1,098% despite they ahievied it very quickly. I dont't think the app is good, yet...

Two tasks have been frozen at 1,098% despite they ahievied it very quickly. I dont't think the app is good, yet...

Off to a rough start but think I got the kinks worked out. Got my 16 thread i7 on an Alphacool 360mm Radiator. 3 work units at a time with 4 threads, holding around 61-62C. Will see how it runs over the next 24 hours.
____________

So this is the biggest one that I've crunched so far.


____________

Starting to see lot of big ones lately but still none of those Giant one that we had a few months ago.

20239542 15931887 13 Feb 2019 | 4:52:27 UTC 13 Feb 2019 | 10:28:09 UTC

Completed and validated 5,123.56 19,627.41 1,235.32 Quantum Chemistry v3.31 (mt)
____________

Starting to see lot of big ones lately but still none of those Giant one that we had a few months ago.

Starting to see lot of big ones lately but still none of those Giant one that we had a few months ago.

It is not as much fun working on the small ones. Maybe when they get the Windows app developed, they will have enough users that they can offer "small" and "large" work units, as defined by disk requirements (or even main memory).

Are they working on a Windows app?

Current test WUs are limited to 30GB, but many are failing because the limit is hit. I will need to raise the limit (the production QC app is 60 GB).

Wow, these things are going to be rough on SSD drive life. How large are the uploads and downloads?

Current test WUs are limited to 30GB, but many are failing because the limit is hit. I will need to raise the limit (the production QC app is 60 GB).

Wow, these things are going to be rough on SSD drive life. How large are the uploads and downloads?

I've had a few "error while computing" for QC tasks on linux recently. I had 4 on Feb 14th and 3 on Feb 13th but none today yet.

Yeah I got a bunch of those as well. I think it's having an issue either downloading or connecting to get the required data. Doesn't happen often but when does, it usually results in numerous errors.
____________

I am running QC on a HP Laptop with E-450 CPU, 8 GB RAM, a 1 TB hard disk which has a SSD partition of 8 GB. No problem so far after two SSD disks on the same laptop failed one after the other running BOINC tasks.
Tullio
____________

A remote server error occurred when trying to retrieve this URL.

A 500-type error (e.g. 500, 501, 502, 503, etc.) indicates the server failed to
fulfill a valid request. The problem may be spurious, and will resolve itself if you
try your request again. If the problem persists, consider notifying the maintainer
of the remote server.

Upgrade my main CPU cruncher to 420 mm Rad with push pull and 4 TB HDD. Need to tweek it just a bit but seems stable at 5 work units with 4 threads apiece. Would include a pic but would take up a lot of space.
____________