Welcome!

Let's start with some good news. I picked up one of your test tasks a couple of days ago.

T0_1-QUICO_TEST_ATM-0-1-RND8922_0

It ran right through without raising any red flags, and validated at the end. A good start.

Thanks for creating an official topic on these types of tasks.

The latest problem observed recently was upload hangs due to a file size too big. it didnt cause an error, but it just never uploaded because the file size exceeded the size limt of your apache server. the only resolution for the user was to abort the transfer and hope it didnt get marked as an error.

have you already addressed this issue? either by adjusting the apache server file size, or adjusting the tasks to not create such large result files.
____________

Welcome and thanks for info Quico

I did notice on past batch the upload got halted by server. It got rejected to download result.
I did a check on client_state file and it was below max_nbyte but still it didn´t allow to upload.

File size in past history that max allowed have been 700mb and these have been around 713-730mb in so something else control this cap and a change maybe help but i don´t see where issue would be.

event log for TL9_72-RAIMIS_TEST_ATM did say

File size in past history that max allowed have been 700mb

I have one which is looking a bit poorly. It's 'running' of host 132158l (Linux Mint 21.1, GTX 1660 super, 64 GB RAM), but it's only showing 3% progress after 18 hours.


(image from remote monitoring on a Windows computer)

Are there any files I can examine, or which would be useful to you for debugging - or should I simply abort it?

I am trying to upload one, but can't get it to do the transfer:
Computer: MSI-B550-A-Pro
Project GPUGRID

Name TL9_82-RAIMIS_TEST_ATM-0-1-RND3943_1

Application ATM: Free energy calculations of protein-ligand binding 1.13 (cuda1121)
Workunit name TL9_82-RAIMIS_TEST_ATM-0-1-RND3943
State Uploading
Received 3/1/2023 4:46:17 PM
Report deadline 3/6/2023 4:46:16 PM
Estimated app speed 16.548,99 GFLOPs/sec
Estimated task size 1.000.000.000 GFLOPs
Resources 0,949 CPUs + 1 NVIDIA GPU
CPU time at last checkpoint 00:00:00
CPU time 05:27:34
Elapsed time 05:28:51
Estimated time remaining 00:00:00
Fraction done 100%
Virtual memory size 0,00 MB
Working set size 0,00 MB

Debug State: 4 - Scheduler: 0

I think mine is a failure. Nothing has been written to stderr.txt since 14:22:59 UTC yesterday, and the final entries are:

This task didn't want to upload, but neither would GPUGrid update when I aborted the upload.
Only got 24h time-outs.
____________
- - - - - - - - - -
Greetings, Jens

I just aborted 1 ATM Wu https://www.gpugrid.net/result.php?resultid=33338739 that had been running for over 7 Days, it sat at 75% done the whole time. Got another one & it immediately jumped to 75% done. Probably just abort it & deselect any new ATM Wu's ...
____________
STE\/E

Some still running, many failing.
Does ATM really just need one CPU?
I think I saw a new 1.1 GB executable DLing. Maybe the failures tried to run on the older version?
What are the VRAM and RAM minimum requirements for ATM?

Server Status shows both ATM and ATMbeta tasks but Tasks shows them all as ATM.
Strange, all my previously completed ATM WUs have vanished from my Tasks list?

Thanks for the papers, I'll read them later.

Three successive errors on host 132158

All with "python: can't open file '/hdd/boinc-client/slots/2/Scripts/rbfe_explicit_sync.py': [Errno 2] No such file or directory"

I let some computers run off all other WUs so they were just running 2 ATM WUs. It appears they do only use one CPU each but that may just be a consequence of specifying a single CPU in the client_state.xml file. Might your ATM project benefit from using multiple CPUs?

Is there a way to tell if an ATM WU is progressing? I have had only one succeed so far over the last several weeks. However, all of the failures so far were one of two types: either a failure to upload (and the download aborted by me) or a simple "Error while computing", which happened very quickly.

However, I now have an ATM WU which has been processing for over seven hours. Looking at the WU properties, it shows the CPU time nearly equal to the elapsed time. The GPU shows processing spikes up to 99%, and the 'down' periods are short.

As others have reported, the Progress shows 75% steadily.

I am inclined to keep letting it compute, but want to know what behavior others have seen on successful ATM WUs.

let me explain something about the 75% since it seems many don't understand what's happening here. the 75% is in no way an indication of how much the task has progressed. it is totally a function of how BOINC acts with the wrapper when the tasks are setup in the way that they are.

the wrapper uses a jobs.xml file to instruct BOINC on different "subtasks" to perform over the course of the run of a single task from the project. in the <task> element there is an option to add a <weight> argument. this would tell boinc how much "weight" in percentage of total task completion that this subtask is worth. weight of 1 is equal to 1% and so on. if this weight argument is not defined, each subtask gets equal weight.

in the case of the ATM tasks, the job.xml file has four subtasks, and no weights defined. the first 3 tasks are just quick extractions and unpacking and complete quickly. which is why the tasks jump to 75% straight away. if it's staying at 75% indefinitely then that's pretty indicative that the task is stuck and probably wont make more progress.

by comparison, the PythonGPU tasks have 2 sub tasks, but the first extraction task has a weight of 1 and the second run.py task has weight of 99 which is why it doesnt have this kind of behavior. and the acemd3 tasks only have one subtask in the file so it doesnt need a weight at all and progress is pretty linear.
____________

I have one that's running (?) much the same. I think I've found a way to confirm it's still alive.

I looked at the task properties to see which slot directory it was running in (slot 2, in my case). Then I found the relevant directory, and poked about a bit.

I found our usual touchstone (stderr.txt) to be useless - it hadn't been touched in hours. But another file - run.log - is currently active. The most recent entries are current:

Thanks for the idea. Sure enough, that file is showing activity (On sample 324, replica 3 for me.) OK. Just going to sit and wait.

Ian&Steve, thanks for the explanation. Just one thought: what if the fourth item is just "do everything else"? Couldn't that mean going straight from 75% to 100% at some point (assuming it is progressing)?

At least in the runs I sent recently we are expecting 341 samples.

Can you see a cntxt_0 folder or several r0-r21 folders? These should be some of the outputs that the run generates, and also the ones I'm getting from the succesful runs.

you should also add weights to the <tasks> element in the jobs.xml file that's being used as well as adding some kind of progress reporting for the main script. jumping to 75% at the start and staying there for 12-24hrs until it jumps to 100% at the end is counterintuitive for most users and causes confusion about if the task is doing anything or not.
____________

Next time I'll prepare them so they run for shorter amounts of time and finish over next submissions. Is there an aprox time you suggest per task?

Well, here it is:



BOINC sees that as 500.28 MB (Linux counts in 1000s, BOINC counts in 1024s) - wish me luck!

Edit - phew, it got through. But that's very, very close to the old limit. Task 33344733

nothing but errors from the current ATM batch. run.sh is missing or misnamed/misreferenced.
____________

I vaguely recall GG had a rule something like a computer can only DL 200 WUs a day. If it's still in place it would be absurd since the overriding rule is that a computer can only hold 2 WUs at a time.
At the rate ATM WUs are failing I could hit that limit, so I halted GG DLs.
Please delete all your WUs until you fix the bug.

Today's tasks are running OK - the run.sh script problem has been cured.

I'm running one that the previous user aborted before it even started - no need for that any more (WU 27426736).

i wouldnt say "cured". but newer tasks seem to be fine. I'm still getting a good number of resends with the same problem. i guess they'll make their way through the meat grinder before defaulting out.

example: http://www.gpugrid.net/result.php?resultid=33357435
____________

My point was: if you get one of these, let it run - it may be going to produce useful science. If it's one of the faulty ones, you waste about 20 seconds, and move on.

Quico/GDF, GPU utilization is low so I'd like to test running 3 and 4 ATM WUs simultaneously.
Sadly GG chokes off work at 2 WUs per computer so that's presently impossible.

My host 508381 (GTX 1660 Ti) has finished a couple overnight, in about 9 hours. The last one finished just as I was reading your message, and I saw the upload size - 114 MB. Another failed with 'Energy is NaN', but that's another question.

The size and time figures are comfortable for me, but others will post their own views.

It would be helpful to work on the intermediate progress reports and checkpointing - at the moment, neither are reported to BOINC. This host (Linux Mint 20.3) spends the entire run reporting 75% progress: my other machine (Linux Mint 21.1) is stuck at 3%. Both run exactly the same build of BOINC.

My observations show the GPU switching from periods of high utilization (~96-98%) to periods of idle (0%). About every minute or two.

i think the current size of the ATM are pretty good. about 4hrs on a 3080Ti and about 5hrs on a 2080Ti.

I'll second Richards's comment that you should put some effort into checkpointing about fixing the completion reporting (add weights to the job.xml file)
____________

Quico/GDF, GPU utilization is low so I'd like to test running 3 and 4 ATM WUs simultaneously.
Sadly GG chokes off work at 2 WUs per computer so that's presently impossible.

How low is it? It really shouldn't be the case at least taking into account the tests we performed internally.

I noticed Free energy calculations of protein ligand binding in WUProp. For example, today's time is 0.03 hours. I checked, and i've 68 of these with a total of minimal time. So i checked, and they all get "Error while computing". I looked at a recent work unit, 27429650 T_CDK2_new_2_edit_1oiu_26_T2_2A_1-QUICO_TEST_ATM-0-1-RND4575_0
The log has this:

+ python -m pip install git+https://github.com/raimis/AToM-OpenMM.git@5d7eac55295e8c6e777505c3ca7c998f1c85987d
Running command git clone --filter=blob:none --quiet https://github.com/raimis/AToM-OpenMM.git /t/boinclib/boinc-client/slots/8/tmp/pip-req-build-3qm67lb1
Running command git rev-parse -q --verify 'sha^5d7eac55295e8c6e777505c3ca7c998f1c85987d'
Running command git fetch -q https://github.com/raimis/AToM-OpenMM.git 5d7eac55295e8c6e777505c3ca7c998f1c85987d
Running command git checkout -q 5d7eac55295e8c6e777505c3ca7c998f1c85987d
error: subprocess-exited-with-error

&#195;&#151; python setup.py egg_info did not run successfully.
&#226;&#148;&#130; exit code: -4


I'm running Linux Mint 19 (a bit out of date), git is git version 2.17.1
/usr/bin/python is Python 2.7.17 and /usr/bin/python3 is Python 3.6.9 -- this was common until recently
uname -a
Linux berfon 5.4.0-104-generic #118~18.04.1-Ubuntu SMP Thu Mar 3 13:53:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
My machine has a gtx-950, so cuda tasks are OK.
It's having an issue writing to /t/boinclib/boinc-client/slots/8/tmp

sudo ls -ld /t/boinclib/boinc-client/slots/8/
drwxrwx--x 2 boinc boinc 4096 Mar 15 10:24 /t/boinclib/boinc-client/slots/8/
So it doesn't look like a permissions issue. The disk drive this is on has over 1 TB space free. It looks to me like git failed, and this is what is happening on all the work units.
My machine is running "New version of ACEMD" routinely.
My preferences for GPUGrid is to run everything. I'm not sure which category this is in, but it must be one of the beta apps.

I hope this helps.

GPUgrid is set to only DL 2 WUs per computer.

ACEMD WUs take around 12ish hours and have approxiamtely 50% GPU utilization

I just started using nvitop for Linux and it gives a very different image of GPU utilization while running ATM: https://github.com/XuehaiPan/nvitop

i would probably give more trust to nvidia's own tools.

I'm running Linux Mint 19 (a bit out of date)

My machine has a gtx-950, so cuda tasks are OK.

Is there a minimum requirement for CUDA and Compute Capability for ATM WUs?
https://www.techpowerup.com/gpu-specs/geforce-gtx-950.c2747 says CUDA 5.2 and https://developer.nvidia.com/cuda-gpus says 5.2.

GPUgrid is set to only DL 2 WUs per computer.

it's actually 2 per GPU, for up to 8 GPUs. 16 per computer/host.

ACEMD WUs take around 12ish hours and have approxiamtely 50% GPU utilization

maybe not much slower, but also not faster.
____________

i would probably give more trust to nvidia's own tools.

watch -n 1 nvidia-smi

maybe not much slower, but also not faster.

until your 12h task turns into two 25hr tasks running two and you get robbed anyway. robbed of the bonus for two tasks instead of just one.

you can set your machine to not download excess tasks by setting a smaller cache size or playing with resource share. that way it wont download the second task until the first one is nearly finished. there are lots of options you can tweak to get the desired behavior.
____________

Picked up another ATM task but not holding much hope that it will run correctly based on the previous wingmen output files. Looks like the configuration is not correct again.

Had hope since the task mentions new in the name.

T_CDK2_new_2_edit_26_1h1q_T4_2_1-QUICO_TEST_ATM-0-1-RND2833_2

[Errno 2] No such file or directory

openmm.OpenMMException: Illegal value for DeviceIndex: 1

Guess I will be the next guinea pig.

Does the ATM app work with RTX 4000 series?
____________
Reno, NV
Team: SETI.USA

Does the ATM app work with RTX 4000 series?

I'm running Linux Mint 19 (a bit out of date)

I just retired my last Linux Mint 19 computer yesterday and it had been running ATM, ACEMD & Python WUs on a 2080 Ti (12/7.5) fine. BTW, I tried the LM 21.1 upgrade from LM 20.3 and can't do things like open BOINC folder as admin. I can't see any advantage to 21.1 so I'm going to do a fresh install and revert back to 20.3.

My machine has a gtx-950, so cuda tasks are OK.

Is there a minimum requirement for CUDA and Compute Capability for ATM WUs?
https://www.techpowerup.com/gpu-specs/geforce-gtx-950.c2747 says CUDA 5.2 and https://developer.nvidia.com/cuda-gpus says 5.2.

Got my first ATM Beta. Completed and validated.

...it seems that the Windows app will available soon!

I see that there is a windows app for ATM. But I have never received an app on any of my win machines, even with an updater. And yes, I have all the right project preferences set (everything checked). So, has anyone received an ATM task on a windows machine?

I see that there is a windows app for ATM. But I have never received an app on any of my win machines, even with an updater. And yes, I have all the right project preferences set (everything checked). So, has anyone received an ATM task on a windows machine?

I see that there is a windows app for ATM. But I have never received an app on any of my win machines, even with an updater. And yes, I have all the right project preferences set (everything checked). So, has anyone received an ATM task on a windows machine?

do you have allow beta/test applications checked?

I see that there is a windows app for ATM. But I have never received an app on any of my win machines, even with an updater. And yes, I have all the right project preferences set (everything checked). So, has anyone received an ATM task on a windows machine?

do you have allow beta/test applications checked?

Yep. Are you saying that you have received windows tasks for ATM?

I see that there is a windows app for ATM. But I have never received an app on any of my win machines, even with an updater. And yes, I have all the right project preferences set (everything checked). So, has anyone received an ATM task on a windows machine?

I see that there is a windows app for ATM. But I have never received an app on any of my win machines, even with an updater. And yes, I have all the right project preferences set (everything checked). So, has anyone received an ATM task on a windows machine?

do you have allow beta/test applications checked?

Yep. Are you saying that you have received windows tasks for ATM?

no I don't run windows. i was just asking if you had the beta box selected because that's necessary.

but looking at the server, some people did get them. someone else earlier in this thread reported that they got and processed one also. very few went out, so unless your system asked when they were available, it would be easy to miss. you can setup a script to ask for them regularly, BOINC will stop asking after so many requests with no tasks sent.

KAMasud got one on his Windows system. maybe he can share his settings.
____________

Quico, Do you have some cryptic requirements specified for your Win ATMbeta WUs?

I've even had my Win computer set to only request ATMbeta WUs and still got nothing.

KAMasud got one on his Windows system. maybe he can share his settings.

Still no checkpoints. Hopefully this is top of your priority list.

BTW, highlight your URL and click URL above and it'll be linkable:
https://arxiv.org/abs/2303.11065

There ate two different ATM apps on the server stats page, and also on the apps.php page. But in project preferences, there is only one ATM app listed. We need a way to select both/either in our project preferences.
____________
Reno, NV
Team: SETI.USA

Let it be. It is more fun this way. Never know what you will get next and adjust.

My new WU behaves differently but I don't think checkpointing is working. It reported the first checkpoint after a minute and after an hour has yet to report a second one. Progress is stuck at 0.2 but time remaining has decreased from 1222 days to 22 days.

The Windoze WUs are all failing.

I have started to get these ATM tasks on my windoze hosts.

All are failing like this:

(unknown error) - exit code 195 (0xc3)</message>
<stderr_txt>
11:28:53 (11872): wrapper (7.9.26016): starting
11:28:53 (11872): wrapper: running python.exe (bin/conda-unpack)
11:28:54 (11872): python.exe exited; CPU time 0.000000
11:28:54 (11872): wrapper: running Library/usr/bin/tar.exe (xjvf input.tar.bz2)
analyze.sh
cntxt_0/
cntxt_0/PTP1B_new-23486-23479
p-0.dat
p-10.dat
p-11.dat
p-12.dat
p-13.dat
p-14.dat
p-15.dat
p-16.dat
p-17.dat
p-18.dat
p-19.dat
p-1.dat
p-20.dat
p-21.dat
p-2.dat
p-3.dat
p-4.dat
p-5.dat
p-6.dat
p-7.dat
p-8.dat
p-9.dat
PTP1B_new-23486-23479_0.xml
PTP1B_new-23486-23479_asyncre.cntl
PTP1B_new-23486-23479.inpcrd
PTP1B_new-23486-23479.prmtop
r0/
r0/PTP1B_new-23486-23479.dcd
r0/PTP1B_new-23486-23479_ckpt.xml
r0/PTP1B_new-23486-23479.out
r1/
r1/PTP1B_new-23486-23479.dcd
r1/PTP1B_new-23486-23479_ckpt.xml
r1/PTP1B_new-23486-23479.out
r10/
r10/PTP1B_new-23486-23479.dcd
r10/PTP1B_new-23486-23479_ckpt.xml
r10/PTP1B_new-23486-23479.out
r11/
r11/PTP1B_new-23486-23479.dcd
r11/PTP1B_new-23486-23479_ckpt.xml
r11/PTP1B_new-23486-23479.out
r12/
r12/PTP1B_new-23486-23479.dcd
r12/PTP1B_new-23486-23479_ckpt.xml
r12/PTP1B_new-23486-23479.out
r13/
r13/PTP1B_new-23486-23479.dcd
r13/PTP1B_new-23486-23479_ckpt.xml
r13/PTP1B_new-23486-23479.out
r14/
r14/PTP1B_new-23486-23479.dcd
r14/PTP1B_new-23486-23479_ckpt.xml
r14/PTP1B_new-23486-23479.out
r15/
r15/PTP1B_new-23486-23479.dcd
r15/PTP1B_new-23486-23479_ckpt.xml
r15/PTP1B_new-23486-23479.out
r16/
r16/PTP1B_new-23486-23479.dcd
r16/PTP1B_new-23486-23479_ckpt.xml
r16/PTP1B_new-23486-23479.out
r17/
r17/PTP1B_new-23486-23479.dcd
r17/PTP1B_new-23486-23479_ckpt.xml
r17/PTP1B_new-23486-23479.out
r18/
r18/PTP1B_new-23486-23479.dcd
r18/PTP1B_new-23486-23479_ckpt.xml
r18/PTP1B_new-23486-23479.out
r19/
r19/PTP1B_new-23486-23479.dcd
r19/PTP1B_new-23486-23479_ckpt.xml
r19/PTP1B_new-23486-23479.out
r2/
r2/PTP1B_new-23486-23479.dcd
r2/PTP1B_new-23486-23479_ckpt.xml
r2/PTP1B_new-23486-23479.out
r20/
r20/PTP1B_new-23486-23479.dcd
r20/PTP1B_new-23486-23479_ckpt.xml
r20/PTP1B_new-23486-23479.out
r21/
r21/PTP1B_new-23486-23479.dcd
r21/PTP1B_new-23486-23479_ckpt.xml
r21/PTP1B_new-23486-23479.out
r3/
r3/PTP1B_new-23486-23479.dcd
r3/PTP1B_new-23486-23479_ckpt.xml
r3/PTP1B_new-23486-23479.out
r4/
r4/PTP1B_new-23486-23479.dcd
r4/PTP1B_new-23486-23479_ckpt.xml
r4/PTP1B_new-23486-23479.out
r5/
r5/PTP1B_new-23486-23479.dcd
r5/PTP1B_new-23486-23479_ckpt.xml
r5/PTP1B_new-23486-23479.out
r6/
r6/PTP1B_new-23486-23479.dcd
r6/PTP1B_new-23486-23479_ckpt.xml
r6/PTP1B_new-23486-23479.out
r7/
r7/PTP1B_new-23486-23479.dcd
r7/PTP1B_new-23486-23479_ckpt.xml
r7/PTP1B_new-23486-23479.out
r8/
r8/PTP1B_new-23486-23479.dcd
r8/PTP1B_new-23486-23479_ckpt.xml
r8/PTP1B_new-23486-23479.out
r9/
r9/PTP1B_new-23486-23479.dcd
r9/PTP1B_new-23486-23479_ckpt.xml
r9/PTP1B_new-23486-23479.out
Rplots.pdf
run.log
run.sh
uwham_analysis.R
uwham_analysis.Rout
11:29:23 (11872): Library/usr/bin/tar.exe exited; CPU time 0.796875
11:29:23 (11872): wrapper: running C:/Windows/system32/cmd.exe (/c call run.bat)
'run.bat' is not recognized as an internal or external command,
operable program or batch file.
11:29:24 (11872): C:/Windows/system32/cmd.exe exited; CPU time 0.000000
11:29:24 (11872): app exit status: 0x1
11:29:24 (11872): called boinc_finish(195)

FWIW, this morning my windows machines started getting ATM tasks. Most of these tasks are erroring out. For these tasks, they have been issued many times over too many and failed every time. Looks like a problem with the tasks and not the clients running them. They will eventually work their way out of the system. But a few of the windows tasks I received today are actually working. Here is a successful example:

http://www.gpugrid.net/result.php?resultid=33375372

So there is hope.
____________
Reno, NV
Team: SETI.USA

FWIW, this morning my windows machines started getting ATM tasks. Most of these tasks are erroring out. For these tasks, they have been issued many times over too many and failed every time. Looks like a problem with the tasks and not the clients running them. They will eventually work their way out of the system. But a few of the windows tasks I received today are actually working. Here is a successful example:

http://www.gpugrid.net/result.php?resultid=33375372

So there is hope.

All openifs tasks are already sent.

...But a few of the windows tasks I received today are actually working.

app_config>
<app>
<name>PythonGPU</name>
<max_concurrent>1</max_concurrent>
<fraction_done_exact/>
<gpu_versions>
<gpu_usage>1</gpu_usage>
<cpu_usage>1</cpu_usage>
</gpu_versions>
</app>
<app>
<name>acemd3</name>
<max_concurrent>1</max_concurrent>
<fraction_done_exact/>
<gpu_versions>
<gpu_usage>1</gpu_usage>
<cpu_usage>1</cpu_usage>
</gpu_versions>
</app>
<app>
<name>ATM</name>
<max_concurrent>1</max_concurrent>
<fraction_done_exact/>
<gpu_versions>
<gpu_usage>1</gpu_usage>
<cpu_usage>1</cpu_usage>
</gpu_versions>
</app>
<project_max_concurrent>1</project_max_concurrent>
<report_results_immediately/>
</app_config>

Why not
<app>
<name>PythonGPU</name>
<max_concurrent>1</max_concurrent>
<fraction_done_exact/>
<gpu_versions>
<gpu_usage>1</gpu_usage>
<cpu_usage>4</cpu_usage>
</gpu_versions>
</app>

?

So far, 2 WUs successfully completed, another one running.

https://www.gpugrid.net/workunit.php?wuid=27438037

https://www.gpugrid.net/workunit.php?wuid=27438416

https://www.gpugrid.net/workunit.php?wuid=27438497

it still can't run run.bat
http://www.gpugrid.net/result.php?resultid=33377536

progress reporting is still not working.

instead of halting progress at 75%, it now halts at 0.19%. the weights help prevent the task from jumping to 75%, but there is still something missing.

Python tasks are able to jump to about 1% after the extraction phase due to the weights, and then slowly creeps up over time as the task progresses. 2%, 3%, 4%, etc until it hits 100% in a natural and linear way. The ATM tasks do not do this at all. they sit at 0.19% for hours and hours with no indication of when they will complete. is it 4hrs? is it 20hrs? there's no feedback to the user. when it's done it just jumps to 100% without warning.

makes it very difficult to tell is a task is stuck or working.

-Edit-

The "BACE" tasks do seem to be reporting progress now. but the earlier tasks from yesterday ("T_p38") do not.
____________

Yes, BACE looks good.

But something wrong with CDK2_new. It jumped to 100% but is still running.
____________

Hello Quico and everyone. Thank you for trying AToM-OpenMM on GPUGRID.

I am unsure if it is relevant to this issue, but AToM implements full checkpointing. Each replica's status is stored in a .xml file in the replica directory. We usually checkpoint every 10 mins, but this interval can be changed in the control file with the CHECKPOINT_TIME parameter (in seconds). Checkpointing is also triggered by SIGTERM or SIGINT signals sent to the main AToM process.

Launching the AToM job from the same folder reads the checkpoints and should restart the simulation as if it had kept running.

The python task must tell the boinc client how many ticks are to calculate (MAX_SAMPLES = 341 from *_asyncre.cntl times 22 replica) and the end of each tick.

In addition, the elapsed time used starts counting again at 0 after each restart. I don't know what the current situation is.

If the progress indicator is now ok, forgot my reply

The ATM tasks also record that a task has checkpointed in the job.log file in the slot directory (or did so, a few debug iterations ago - see message 60046).

That file can be viewed while a task is running, but not after it's finished. It's written (I think) by the science app, but messages are passed to BOINC by the wrapper: that's probably where the problem is.

Edit: OK, I've downloaded a BACE task (resend _4) and a T_PTP1B_new task (resend _3). I'll watch them when the current pair of Abouh tasks have finished.

The GPUGRID version of AToM:

https://github.com/Gallicchio-Lab/AToM-OpenMM/blob/master/sync/atm.py

has this:

OK, the BACE task is running, and after 7 minutes or so, I see:

Actually, it is unclear if AToM's GPUGRID version checkpoints after catching termination signals. I'll ask Raimondas. Termination without checkpointing is usually okay, but progress since the checkpoint would be lost, and the number of samples recorded in the checkpoint file would not reflect the actual number of samples recorded.

Does anyone know if BOINC sends specific signals to terminate an app? Would the app pass the signal to the main AToM's python process?

The app seems to be both checkpointing, and updating progress, at the end of each sample. That will make re-alignment after a pause easier, but there's always some over-run, and data lost on restart. It's up to the application itself to record the data point reached, and to be used for the restart, as an integral part of the checkpointing process.

I can't answer immediately on the termination question, but it's all open-source and I can look through it. In this case, it's more complicated, because BOINC will talk to the wrapper, and the wrapper will talk to the science app.

But the basic idea is that BOINC will send a request to terminate over the API, and wait for the application to close itself down as it sees fit. Actual signals will only be used to force termination in the case of an unconditional quit, such as an operating system closedown.

Seriously? Only 14 tasks a day?

Seriously? Only 14 tasks a day?

The T_PTP1B_new task, on the other hand, is not reporting progress, even though it's logging checkpoints in the run.log

A file is maintained in the slot folder, called 'boinc_task_state.xml' (it's probably written by the wrapper, though I'm not certain of that).

The current contents are:

task 27438680
Completed and validated. While the following task had a failure after a re-start.
task 27438865

task 27438680
Completed and validated. While the following task had a failure after a re-start.
task 27438865

My BACE task 33378091 finished successfully after 5 hours, under Linux Mint 21.1 with a GTX 1660 Super.

Four previous attempts failed, two of them under Windows with a 0xc0000135 error in Python.exe - that's a missing DLL.

Task 27438853
Completed and validated. Short one though.

I can't answer immediately on the termination question, but it's all open-source and I can look through it. In this case, it's more complicated, because BOINC will talk to the wrapper, and the wrapper will talk to the science app.

But the basic idea is that BOINC will send a request to terminate over the API, and wait for the application to close itself down as it sees fit. Actual signals will only be used to force termination in the case of an unconditional quit, such as an operating system closedown.

Hi, i have some "new_2" ATMs that run for 14h+ yet. Should i abort them?
Running linux with rtx3070 cards
____________

The wrapper you're using at the moment is called "wrapper_26198_x86_64-pc-linux-gnu" (I haven't tried ATM under Windows yet, but can and will do so when I get a moment).

That wrapper name looks as if it was prepared from BOINC code dating to around February 2017. At that time, BOINC was working on versions of the wrapper specifically intended for use with VirtualBox.

BOINC makes pre-compiled versions of the wrapper available for projects to use "as is", but some projects customise the source code to suit their own needs. I don't know which path GPUGrid has taken.

Edit - I just looked at the file name the first time. In stderr.txt, I see

Hi, i have some "new_2" ATMs that run for 14h+ yet. Should i abort them?

Hi, i have some "new_2" ATMs that run for 14h+ yet. Should i abort them?

I have one at the moment which has been running for 17.5 hours. The same machine completed one yesterday (task 33374928) which ran for 19 hours. I wouldn't abort it just yet.

And completed.

Seriously? Only 14 tasks a day?

The quota adjusts dynamically - it goes up if you report successful tasks, and goes down if you report errors.

WUs failing on Linux computers:

Any ideas why WUs are failing on a linux ubuntu machine with gtx1070?

<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
14:01:49 (3551): wrapper (7.7.26016): starting
14:02:12 (3551): wrapper (7.7.26016): starting
14:02:12 (3551): wrapper: running bin/python (bin/conda-unpack)
14:02:13 (3551): bin/python exited; CPU time 0.280413
14:02:13 (3551): wrapper: running bin/tar (xjvf input.tar.bz2)
14:02:14 (3551): bin/tar exited; CPU time 0.840912
14:02:14 (3551): wrapper: running bin/bash (run.sh)
+ echo 'Setup environment'
+ source bin/activate
++ _conda_pack_activate
++ local _CONDA_SHELL_FLAVOR
++ '[' -n x ']'
++ _CONDA_SHELL_FLAVOR=bash
++ local script_dir
++ case "$_CONDA_SHELL_FLAVOR" in
+++ dirname bin/activate
++ script_dir=bin
+++ cd bin
+++ pwd
++ local full_path_script_dir=/var/lib/boinc-client/slots/7/bin
+++ dirname /var/lib/boinc-client/slots/7/bin
++ local full_path_env=/var/lib/boinc-client/slots/7
+++ basename /var/lib/boinc-client/slots/7
++ local env_name=7
++ '[' -n '' ']'
++ export CONDA_PREFIX=/var/lib/boinc-client/slots/7
++ CONDA_PREFIX=/var/lib/boinc-client/slots/7
++ export _CONDA_PACK_OLD_PS1=
++ _CONDA_PACK_OLD_PS1=
++ PATH=/var/lib/boinc-client/slots/7/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
++ PS1='(7) '
++ case "$_CONDA_SHELL_FLAVOR" in
++ hash -r
++ local _script_dir=/var/lib/boinc-client/slots/7/etc/conda/activate.d
++ '[' -d /var/lib/boinc-client/slots/7/etc/conda/activate.d ']'
+++ ls -A /var/lib/boinc-client/slots/7/etc/conda/activate.d
++ '[' -n ocl-icd_activate.sh ']'
++ local _path
++ for _path in "$_script_dir"/*.sh
++ . /var/lib/boinc-client/slots/7/etc/conda/activate.d/ocl-icd_activate.sh
+++ conda_ocl_icd_activate
++++ ls /var/lib/boinc-client/slots/7/etc/OpenCL/vendors/
+++ [[ -z ocl-icd-system ]]
+ export PATH=/var/lib/boinc-client/slots/7:/var/lib/boinc-client/slots/7/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ PATH=/var/lib/boinc-client/slots/7:/var/lib/boinc-client/slots/7/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ echo 'Create a temporary directory'
+ export TMP=/var/lib/boinc-client/slots/7/tmp
+ TMP=/var/lib/boinc-client/slots/7/tmp
+ mkdir -p /var/lib/boinc-client/slots/7/tmp
+ echo 'Install AToM'
+ REPO_URL=git+https://github.com/raimis/AToM-OpenMM.git@172e6db924567cd0af1312d33f05b156b53e3d1c
+ python -m pip install git+https://github.com/raimis/AToM-OpenMM.git@172e6db924567cd0af1312d33f05b156b53e3d1c
Running command git clone --filter=blob:none --quiet https://github.com/raimis/AToM-OpenMM.git /var/lib/boinc-client/slots/7/tmp/pip-req-build-0qwsbkqo
Running command git rev-parse -q --verify 'sha^172e6db924567cd0af1312d33f05b156b53e3d1c'
Running command git fetch -q https://github.com/raimis/AToM-OpenMM.git 172e6db924567cd0af1312d33f05b156b53e3d1c
Running command git checkout -q 172e6db924567cd0af1312d33f05b156b53e3d1c
error: subprocess-exited-with-error

&#195;&#151; python setup.py egg_info did not run successfully.
&#226;&#148;&#130; exit code: -4
&#226;&#149;&#176;&#226;&#148;&#128;> [0 lines of output]
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

&#195;&#151; Encountered error while generating package metadata.
&#226;&#149;&#176;&#226;&#148;&#128;> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
14:02:22 (3551): bin/bash exited; CPU time 2.696428
14:02:22 (3551): app exit status: 0x1
14:02:22 (3551): called boinc_finish(195)

</stderr_txt>

(I haven't tried ATM under Windows yet, but can and will do so when I get a moment).

First reports from Dependency Walker:

"Error opening file: The system cannot find the file specified" for
API-MS-WIN-CORE-PATH-L1-1-0.DLL
API-MS-WIN-CORE-WINRT-ERROR-L1-1-0.DLL
API-MS-WIN-CORE-WINRT-L1-1-0.DLL
API-MS-WIN-CORE-WINRT-ROBUFFER-L1-1-0.DLL
API-MS-WIN-CORE-WINRT-STRING-L1-1-0.DLL
DCOMP.DLL
IESHIMS.DLL

The API-MS-WIN group and IESHIMS.DLL usually resolve when delay-load files are loaded during the run. But I can't find DCOMP.DLL in either the unpacked libraries, or the Windows system disk.

DCOMP.DLL seems to be called from MSHTML.DLL, which is a Windows system file. But I still can't find it from there.

Enough for now - my head is spinning!

Edit - DCOMP.DLL is present on my Windows 10 - now Windows 11 - laptop. Another fine example of Microsoft version control.

Just a note of warning: one of my machines is running a JNK1 task - been running for 13 hours.

It's running fine - the run log has reached sample 287, and progress has reached 1.2654867256637168

But that's over 100%, and the BOINC display has reached (and is pegged at) 100% - probably has been for several hours. Ignore it.

Edit: It's reached sample 298. And I've found a [task name].cntl file, which contains the line

I also noticed this latest round of BACE tasks have become much longer to run on my GPUs. Some are hitting > 24 hrs. I am going to stop taking new ones unless the # samples/task is trimmed down.

I had this one running for about 8 hours, but then i had to shut down my computer.
Unfortunately, it couldn't restart from the app checkpoint, and since there is no boinc checkpoint, it crashed and reported no run time.

Forget about a re-start, these WUs cannot even take a suspension. I suspended my computer and this WU collapsed.
task 27438865

i'm a bit surprised right now, i looked at the resend, it was successfully completed in just over 2 minutes, how come? the computer has more WUs that were successfully completed in such a short time. Am I doing something wrong?

I also noticed this latest round of BACE tasks have become much longer to run on my GPUs. Some are hitting > 24 hrs. I am going to stop taking new ones unless the # samples/task is trimmed down.