I've ended up with two phantom WUs, and with one running dont want to detatch just yet. I am sole cruncher for both, so would be good if they could be reissued to prevent any delays. They are:

http://www.gpugrid.net/workunit.php?wuid=738974

and

http://www.gpugrid.net/workunit.php?wuid=740737

(currently crunching 742779)

Regards
Zy

Phantom WUs - they are a common occurance in BOINC due to its design. I get a few from time to time, most people do. Generally it gets ignored and the WU times out to be reallocated by the server, which is fine, but wastes much time - eg CPDN with timeouts of 12months + :)

This recently cropped up at Aqua, where the admins were not aware of the nature of Phantoms. Richard kindly helps out there as they are going through a steep learning curve now their apps are getting complex. He posted the extract below to explain to them, and I have copied it here, as its a good suscinct summary of the issue and what to do about it.

============= Richards Aqua Post - Phantoms ==========

Time for another little tutorial, I think.

This problem of "ghost", "phantom" or just plain "missing" tasks happens on all projects from time to time. The assumption is that a user's computer requests work: the server hears that request, selects a task, marks it as 'in progress' in the database, and sends a message back to the user with all the details. But for some reason the user's computer misses that message - there's a glitch on the internet, the computer is switched off or rebooted at the critical moment, or something like that. Unlike most BOINC communications, these messages don't seem to require an active acknowledgement, so if they get missed, they stay missed.

Straight out of the box (which as we know is the way boinc_admin and neo like to run things round here), that's the end of the story. So there will be an unknown number of ghosts in the database, and unless somebody reports them here so they can be exorcised manually, they'll hang around haunting the place until they reach their deadline (early October, for recent tasks). Then they'll get re-issued: not good for draining this batch of tasks out of the system. [insert by Zy: Aqua are currently running down the queue ready for a new design of WU]

Fortunately, as with most things, BOINC provides a solution in the standard code: all it requires is configuration. [Now where have I heard that before?].

The magic incantation is <resend_lost_results>1</resend_lost_results> in the server's config.xml file. As noted in the Project configuration documentation, this causes extra work for the database server, but I can't imagine the server is overloaded at the moment: it would a good moment to learn about this setting (and whether the AQUA server can cope with the extra load), so that the ghosts can be exorcised automatically by the machine, and not take up valuable staff time cancelling each one manually.

===============

In my case at the time of download I was having issues with a nasty piece of spyware that locked my whole system. Hence I fell into the classic scenario Richard described - you think I got them, in fact I didnt, they show in my task list, but do not appear in BAM at my end. Similar can happen if a user has a disk crash and looses held WUs on reformat - they are marked as held in the Db your end - but gone "poof" at the user end.

Most times however Phantoms occur due to the incomplete way the BOINC server deals with the RPC commands as Richard describes.

A Classic way to treat Phantoms, is to run down a Projects queue in BAM until its dry for a PC, then detach & reattach the PC. This fires off BOINC at your end to reallocate any uncrunched tasks, which of course will include the Phantom entries. Right now, I have some GPUGRID ones running, so dont want to detach/reattach as I would end up killing off a running WU.

Its this detach/reattach solution used by lots of BOINC'rs that deals with the issue - but can mask that the issue exists in the first place. Richards explanation shows how a Project can pro-actively deal with it, not just leave it to the Cruncher - as many of the latter will not bother, and just leave them to time out.

Regards
Zy

Sorry forgot the last bit .....

So, at the end of the day, whats in it for GPUGRID? Those phantom WUs - and I guarantee they will be there in unknown numbers if this has not been addressed before - will lurk until timeout. You think they are issued - they have not been. Here that means they stay there in [whoever's] task list until timeout - currently 5 days.

You want WUs back, ideally, in 24 hrs, there are lots in there that will never come back until timeout 5 days later. Fix this, and you will increase the rate WUs are returned because you stop the five day delay of zero crunching, and increase the number of available workpieces to scientists in a shorter space of time.

Bottom line - you'll get more done in less time.

Regards
Zy

The devil is in the detail - that "extra work for the database server".

Every time a BOINC client contacts a BOINC server, it sends - almost as an afterthought - a complete list of every 'task in progress' on the client. The process invoked by resend_lost_results involves checking that comprehensive list against the comprehensive list in the database of all tasks issued to the host.

For AQUA, that's ~3x3: for GPUGrid, ~6x6. Neither should cause much angst.

But for projects like SETI, with short CUDA tasks and generous deadlines, that can be several thousand x several thousand. The SETI servers are under stress at the best of times: resend_lost_results has been tried, and it breaks them.

So it's not unreasonable that the resend_lost_results code exists, but is disabled by default: server admins are at liberty to enable it as and when they are confident that their server hardware is up to the task.

It all goes to prove that RTFM applies to admins as much as punters!

You'll need to run that one past David A or Rom W.

I was involved in a similar discussion at SETI about three years ago, and (partly) as a result, Rom wrote a much-quoted blog The evils of 'Returning Results Immediately': the point being the amount of time it takes the scheduler DB process to open the database and make the necessary transactions on each client contact, with the emphasis being on the 'opening'.

Your process involves three separate server database interactions: because of the asynchronous timing, that possibly involves three separate 'open' phases, which would fall foul of Rom's concern about overheads.

Since then, I've heard much less about database loading as the primary BOINC constraint: since then it's mostly been disk loading, network throughput and (most recently) log volume latency. Maybe modern hardware or improvements in MySQL have supplanted the DB overhead, and your scheme could be implemented now. But you're right: it's anomalous that this crucial transaction is one of the few that isn't protected by a robust transactional handshake.

Guys - just had a thought [ happens from time to time rofl:) ]

I well appreciate the transactional process you have described above, and possible solutions - the latter being predicated on making sure the transaction goes ahead correctly in the first place, a natural and proper thought process.

How about attacking it from another angle - and maybe - this could be implimented at Project level outside the BOINC server code. Its not a "Solution", more a workaround, but I acknowledge the problem of getting the server code to be all things to all men when so many different needs from disparate Projects are being served by the same Infrastructure.

In the classic transaction solutions being discussed, the critical issue always comes back to processing the information on the server database and the loading it causes. Flip the principle, put the db loading on the distributed client. Pick a quiet(ish) time, all Projects have a time of day where there is usually a downturn in activity, and send off a list of [alledgedly] held WUs to Clients, then get BAM to search and respond if it agrees. Whilst there will be an element of db activity at the server end compiling the queeries, its far far less than [at SETI for example thousands x thousands scenario] running what is in effect a Many to Many Join SQL queery, its "just" a straightforward "Select From .... For [user]" far less intensive on the server, even acknowledging issues from Projects like SETI with huge user numbers.

That way the bottom line problem of validation and db search loading is handed to the Client and far less load on the server ? As you are aware, I dont have the lower level background you guys do .... so there is probably a "no no" lurking in there somewhere :) - however I cant get the thought out of my skull, so I thought I would let it go and see if it flys.

Regards
Zy

The servers know the client IP address at last contact (it's shown on your host details page, though not visible if you're not logged in on the associated user account) - and very useful that is too, for tracing stolen laptops.

But it's not enough. I have seven - belay that, six (I lost a fan on my faithful Celeron 400MHz MMX, and retired it after 10 years) - hosts behind a single DSL router, so all with the same IP address. I'm not opening the firewall on that router for anybody, not even BOINC. So servers "calling out" is a complete no-no.

But all is not lost. This project already uses the "call me again in a few hours, to confirm you're still alive" protocol. Normally, that just results in NOP, or occasionally "BTW, I've finished a task - you may as well have the report while we're chatting". That would be a good opportunity fo Zy's extension: "Things are quiet round here at the moment. Let's compare notes about what we each think you're up to at the moment."

Come to think of it, that's exactly what resend_lost_results does at the moment: the trouble is, it's either done on every single contact, crashing the server, or NEVER. Maybe if, when RLR was enabled, it did a full check every (randomised) ~nth scheduler contact (n configurable from 100% to 0.01%, to suit project needs and capabilities), there would be a periodic 'catchup' without massive overheads?

One of the fundamental problems is that though the design intent is for small projects with low budgets much of the implementation revolves about SETI@Home's issues and constraints.

This is just one more example of how the need to shave corners for SaH causes issues for other projects.

The defaults should be for robust protocols with options for turning down and turning off options if and as they might be needed for performance reasons.

THough I will note that much of the performance problems of the BOINC database design are self inflicted but that all suggestions to rectify those issues are routinely ignored.