I have been crunching on Test4Theory for just over 2 years. It is a project that does work for CERN and the large hadron collider.

The theory is that by using a virtual machine, they are only required to get their application to work on one operating system.

The setup requires a host operating system (either Windows, OSX or Linux), Virtualbox, a guest operating system (CERN uses their own flavor of LINUX), BOINC, and the scientific application. And that my friend is an awful lot of moving parts to try to keep working together.

Any time there is an upgrade in any of the components, there is a chance of compatibility problems. Most of the time, problems occur when Oracle "improves" Virtualbox and the "improvements" break something in the BOINC -> Virtualbox -> guest O/S interface.

Another area of problems is getting permissions set up correctly so that Virtualbox and the guest O/S have the permissions that they need to function properly, especially in Linux. That may have improved some with the latest release of the Virtualbox wrapper, but caution is still advised.

If you cruise on over to the Test4Theory web site, you will see that they have a relatively low level of participation. It is probably because it is challenging to set up the BOINC/Virtualbox environment and keep up with all the upgrades and problem resolutions.

You might want to spend some time browsing the message boards at Test4Theory and get a feel for they challenges that they face.

If you want more information from me, let me know.

Hope that helps,
CaptianJack

The Test4Theory project has been one of the projects at the forefront of developing the "app in a VM" approach. Working closely with Rom Walton and Oracle, they've eliminated most of the technical problems though I think they haven't quite finished recompiling and bundling all the fixes together and releasing a new vboxwrapper. Jacob Klein is more familiar with that than I am so I'll leave that topic for him. I would rather describe to you what I think has been the source of past problems and may well be the source of future problems.

In the traditional model BOINC communicates with and controls the project app directly. By "app control" I mean suspending, resuming, stopping, starting the app. When the app is in a VM the traditional BOINC <-> app comms lines don't exist because the app is sandboxed in the VM. App control is achieved through the vboxwrapper developed mostly by Rom Walton. Vboxwrapper calls vbox-manager (a convenience app Oracle ships with the VBox package) with appropriate parms, vbox-manager controls the VM, and that is mostly where past problems have occurred. Bugs in either vbox-manager or the VM itself or perhaps Rom's use of vbox-manager, have caused sporadic comms "disconnects" between BOINC client and vbox-manager which have resulted in, among other things, the VM not stopping/suspending when it should. As I mentioned above, that problem as well as others appear to have been eliminated recently. There may be other problems I am not aware of.

Today the main deficiency, though it may never cause a problem that becomes apparent to the user, is that controlling the VM via vbox-manager is like controlling BOINC client via boinccmd. It works but it's clunky, for lack of a technical term. Oracle has developed an API for the control interface which vboxwrapper could use (but does not) to control the VM directly. Perhaps one day BOINC client itself will use the API and eliminate the wrapper.

____________
BOINC <<--- credit whores, pedants, alien hunters

Here is another factor to consider, not a problem per se, but just something to think about. When running a BOINC task under a VM, each task that runs requires a virtual machine, a copy of the guest O/S, a copy of the scientific application and all the data that is required to perform all the calculations. That all amounts to quite a bit of memory that is required to run a task.

The Test4Theory task that I am running right now in Linux is using 348MiB of memory.

Test4Theory is currently set up to only run 1 task per computer. If a project/person wants to run more than one task per machine, memory requirements will go up proportionally.

Problems include the VM not being removed when a WU fails (would use up hard drive space).

I'll chime in.

I've only been doing VM tasks for about 3-4 weeks. I wish I had been doing it earlier, since I'm a BOINC alpha tester (and could have helped find/fix the problems), but alas, for whatever reason, I didn't. But now, I am doing work for Test4Theory, RNA World, and Climate@Home (when they have work).

The main problems I've seen have been:

1) Oracle problems resuming from snapshots. 4.2.16 was great, and didn't have problems, but 4.2.18 introduced a bug, 4.3.0 fixed it but introduced another, and 4.3.2 [current] fixed it but introduced another. The most recent unfixed bug does not affect RNA World, but does affect T4T. Details can be found at these 2 links:
http://lhcathome2.cern.ch/test4theory/forum_thread.php?id=1364&postid=15560
https://www.virtualbox.org/ticket/12291
I expect Oracle to get this resolved in the next maintenance release, so VirtualBox itself doesn't fail BOINC tasks like it does now when resuming T4T tasks.

2) Any BOINC applications that use the VM, go through that vboxwrapper. It too has had some bugs that I've worked with Rom to solve. Most recently, we found a heartbeat bug where tasks for applications compiled with API code within the past few months, when ran on Windows service installs, would simply fail due to a problem with the heartbeat check. It's fixed now, but it requires a newer vboxwrapper, which means a newer application version. Also, there were some issues with trickle messages not working properly, but I believe that's fixed too. Long story short: The vboxwrapper is not quite matured yet, but it is maturing, and I'd recommend any application that uses it to have a team that is "nimble" enough to easily update to newer application versions.

3) Communication/control doesn't always work. It works almost all the time, but during BOINC upgrades and a few other scenarios, the VMs are not paused appropriately. Then when BOINC later tries to "attain a lock" to control it, the lock can fail, and it causes problems. Rom is trying to get better control on this with the wrapper, but it is an ongoing issue that does occur.

4) VirtualBox being installed. This is a problem for some. For me, I encounter network problems during LAN parties, whenever VirtualBox is installed (because it does something with some "VirtualBox Host-Only Network" network adapter that can cause problems when gaming or sharing files.) In general, it works just fine, but it does monkey with the network, and that right there is a reason that I don't install it on my wife's laptop, because I cannot risk her having any sort of crazy network issue when she's away at the office doing work. And so that's a "showstopper" for some machines/users.

5) VM Management. If you are in a non-service-install, it's neat that you can open the "VirtualBox Manager" program to see BOINC managing (pausing, snapshotting, resuming, shutting off) your VMs. However, for service-installs, you cannot. So, you lose out on seeing that info. Not a big deal, but it is a problem, in the sense that the user can't ever manually see/control/cleanup the VMs. Note: T4T actually uses the "Remote Display" capabilities of VirtualBox, and the BOINC client has support for it too, such that in BOINC I can click a "Show VM Console" button, that uses Window Remote Desktop to actually connect to the VM and show me some text. It's neat. :)

6) Cleanup. Most times, when a task fails, BOINC uses the vboxwrapper to clean the slot up. But sometimes it doesn't do that. (I haven't thoroughly tested) But Rom told me that, the next time that application runs in that slot, it'll do a cleanup before copying data into the slot. I find that a bit sloppy, especially since we're dealing with big files, but it is what it is.

7) Memory. The RNA World uses a 64-bit VM with a Base Memory of 4 GB. So, BOINC will make sure your system is allowed to use 4 GB before giving you the task. But, while that task is running, because Windows doesn't have an effective way to see how much memory the VM is using, and because the VirtualBox manual states that the host shall grant the Base Memory amount as free memory and not expect to use any of it... We recently put in a client fix (made it into 7.2.28) where, when any VM app is running, the <rsc_memory_bound> value will be used as the "this amount of memory is allocated for this running app" value. Non-VM apps still query the Windows processing list to get that information. So... If you want to run an RNA World app, BOINC [correctly] considers 4 GB memory as "used", even if the client OS is using a fraction of that. Thus, it's crucial to set "Base Memory" to an appropriate value, and to make sure <rsc_memory_bound> matches it, and for those projects that "might use x amount", BOINC will consider "x amount used" while the task is running.

8) Host responsiveness. I have found that, when running 2 VMs (that use the "Enable VT-x/ADM-v" hardware acceleration option) at the same time, my music sometimes skips, my mouse cursor's acceleration isn't always registered properly, so my cursor skips around when I move it, the UI sometimes refreshes panes/grids slowly, and my machine is difficult to control. We're still looking into this issue, but I'm pretty sure it's related to that hardware acceleration option, and perhaps Intel doesn't handle multiple VMs too well on my 2009 i7 965XE CPU. But it is an issue. So, I make sure to only run 1 VM at a time, by manually suspending tasks, until I go to bed, and then I run multiples.

Whew, I'm beat. I'm sure there's more "problems" out there, but that's all I can think of off the top of my head. I'm still cutting my teeth on this VM stuff, but am pressing T4T admins, RNA World admins, and Oracle devs, to all fix the bugs so it all runs smoother. You'll see my name on a lot of posts, mainly because I care. Eventually, BOINC devs may ONLY offer the "BOINC+VirtualBox" installer; they want users to be capable of running these new VirtualBox tasks. And I get that. But, at the same time, we're still working out the kinks.

By the way, I should mention that, for RNA World, their VirtualBox application is a blessing! I restart my machines often, and so for the longest time (years?), I refused to run their tasks, because they could take 2-60 days, with no checkpointing! But that issue is solved! So, right now, I've got one of their VM tasks that is estimated to take 60 days, I've been working on it for 3 weeks, I'm on "CPU-processing-day 11", and so far it is working well! I've restarted numerous times, and it resumes form snapshot just fine. Here's a little more info on how it works: I believe that, if the user has "save to disk" at a value that is less than 10 minutes, vboxwrapper takes a snapshot every 10 minutes (since a lot of data may need to be saved to store the snapshot which is a "dynamic difference disk"). So, any time I need to restart my PC, I only lose up-to-10 minutes of work. That's AWESOME! That means I can actually finish this task that is estimated to take 8 weeks! (I'm excited).

Hope you find this useful!
- Jacob

8) Host responsiveness. I have found that, when running 2 VMs (that use the "Enable VT-x/ADM-v" hardware acceleration option) at the same time, my music sometimes skips, my mouse cursor's acceleration isn't always registered properly, so my cursor skips around when I move it, the UI sometimes refreshes panes/grids slowly, and my machine is difficult to control. We're still looking into this issue, but I'm pretty sure it's related to that hardware acceleration option, and perhaps Intel doesn't handle multiple VMs too well on my 2009 i7 965XE CPU. But it is an issue. So, I make sure to only run 1 VM at a time, by manually suspending tasks, until I go to bed, and then I run multiples.

Just out of curiousity, when you were running multiple VMs, are you ABSOLUTELY POSITIVE that multiple running VMs had the "Enable VT-x/ADM-v" option checked? The reason I ask is that, right now, T4T uses a version of the wrapper where it leaves that option unchecked. (T4T uses a single-CPU 32-bit VM, and the old version of the wrapper left the option unchecked in that scenario, and on side note, it suffers from the server-install heartbeat problem as well. T4T admins are looking to upgrade to latest wrapper in beta, sometime soon.)

... and I've only noticed the OS responsiveness issues when multiple tasks with "Enable VT-x/ADM-v" were running. (For T4T, I'm currently using anonymous platform with the newest version of the wrapper, for testing purposes, which checks/enables that option even for single-CPU 32-bit tasks, which is why I'm having the issue currently.)

So, again... are you sure your T4T tasks were using it?

So, again... are you sure your T4T tasks were using it?

am I right to understand that the VM image is supplied in the same way that the application is for a standard BOINC project? That's to say that it is downloaded once and re-used and shared for all WUs?

Jacob,

Thanks very much for taking the time to write such a comprehensive summary. It's very useful!
One question - am I right to understand that the VM image is supplied in the same way that the application is for a standard BOINC project? That's to say that it is downloaded once and re-used and shared for all WUs?

Matt

Ah, thanks, I must have missed that.

Last time I tried to get the drivers to work for my GPU's I forgot I didn't have the monitor plugged in, and ended up at a black screen with an X for an arrow. It might be easier to go headless but I might have to build an image outside of the VM and then import it, some how. Next time I try the drivers I'll remember to clone first and plug in a monitor!

I wanted to mention one more thing that occurred to me, regarding VMs. There are sort of 2 different ways of handling the downloading/uploading of the work.

Method 1:
The way it was intended by the BOINC devs (used by RNA World): BOINC downloads the VM and the work, and the VM is fired up to run that 1 task. A "shared folder" is setup within the slot directory, where BOINC deposits input files (used by the VM), and later grabs output files (given by the VM). When the task is done, the VM is fired down and the result it uploaded by BOINC.

Method 2:
The way where downloading/uploading of work is managed within the VM (used by Test4Theory): BOINC downloads the VM, but the VM does some network connections to get 1-or-more tasks, or even get new tasks in serial after the prior task completes, and the VM runs indefinitely or it runs until it has been configured to shutdown.

Test4Theory uses method 2, where they run a VM for 24 hours, and in that time frame, the VM uses networking to get a task, run it, report it, get another, etc., several times within that 24 hours.

I've personally had some problems with the way Test4Theory does this. There are times when their servers (called "copilot"?) aren't working, and then an entire CPU is completely wasted, because BOINC "allocated/gave" that task the CPU, and the VM doesn't utilize it. That was a bitter pill for me to swallow, since I desperately desire to keep all of my resources fully-utilized.

Method 1 is preferred, so far as I know.

Regards,
Jacob

Your concerns regarding numerous VMs, one from each of several projects, is valid. If that trend continues it won't turn out well for any project. A better approach is one (maybe two) VMs that can run any and all projects that need to run in a VM.

Regarding Tesla and Quadro working in a VM, try GeForce GTX780 to Tesla K20 . There are links for GK104 too, gonna try it on my 660Ti soonest. I'll be buying as many 780 as I can and keeping copies of recent drivers because nVidia will put a stop to this somehow.

____________
BOINC <<--- credit whores, pedants, alien hunters

I have solved my "Host responsiveness" issue that I reported as #8 in my list.
Basically, Oracle VirtualBox has had a bug since v4.2, and the bug still affects the current release, v4.3.2. Read on for a solution.

The bug manifests itself as huge DPC Latencies in the host environment, possibly only on x64 Windows hosts. For me, this meant that the mouse cursor acceleration was misinterpreted, my mouse cursor skipped around irregularly, my music sometimes skipped, and my windows/controls sometimes lagged when being painted/refreshed/moved. My workaround had been to only allow 1 VM to run at a time, since the problem dissipated with just 1 VM running (problem still existed, just not as bad). Keep reading for that solution.

The bug is documented a bit in this thread:
https://forums.virtualbox.org/viewtopic.php?f=6&t=58030&sid=029eb906ea896f79a218b8d33ed399cf

The VirtualBox devs believe they've fixed the problem, and they gave a link to a 4.3.3 Test version at the top of page 2 in that thread.

That 4.3.3 Test version does indeed resolve my host responsiveness issue. I can now run 3 VMs, plus 4 other non-VM CPU tasks, plus 3 GPU tasks, while listening to music, while playing a game.... all without a single problem. Life is good again.

I expect the 4.3.3 official release (no ETA yet) will include this bug fix.

Regards,
Jacob