Very long run times, poor performance

Message boards : Number crunching : Very long run times, poor performance

Author	Message
RaymondFO* Send message Joined: 22 Nov 12 Posts: 72 Credit: 14,040,706,346 RAC: 0 Level Scientific publications	Message 27438 - Posted: 27 Nov 2012 \| 4:02:45 UTC
	Hello. I am new to this site and was testing my video cards to see how well they would perform with the GPUGRID work units. The computers with Windows 7 with EVGA video card GTX 570 and 580 performed very well. However when I run GPUGRID work units under Ubuntu 12.10 64 bit with a ZOTAC GTX 580 the run times are very poor. Currently I am running ACEMD2:GPU molecular dynamics 6.16 (cuda 42) and after approximately 17 hours runtime with a projected 19 hours to run this compares poorly to the Windows 7 64 bit GTX 570 EVGA video card with approximately 6900 seconds average runtime for comparable ACEMD2 work units. The video performance is terrible as well, yet the Windows version had a minor performance issue. The Unbuntu has the NVIDIA driver 304.43 installed, with over 200gigs hard drive space available, 8 gigs of RAM, the "Use GPU while computer is in use box checked", BOINC manager 7.0.27, Intel i7-2600k CPU @3.4 GHz. Any ideas what is going on?
	ID: 27438 \| Rating: 0 \| rate: / Reply Quote

skgiven Volunteer moderator Volunteer tester Send message Joined: 23 Apr 09 Posts: 3968 Credit: 1,995,359,260 RAC: 0 Level Scientific publications	Message 27440 - Posted: 27 Nov 2012 \| 5:35:55 UTC - in response to Message 27438. Last modified: 27 Nov 2012 \| 5:37:29 UTC
	You probably need to set Prefer Maximum Performance. See this post ____________ FAQ's HOW TO: - Opt out of Beta Tests - Ask for Help
	ID: 27440 \| Rating: 0 \| rate: / Reply Quote

RaymondFO* Send message Joined: 22 Nov 12 Posts: 72 Credit: 14,040,706,346 RAC: 0 Level Scientific publications	Message 27443 - Posted: 27 Nov 2012 \| 13:49:48 UTC - in response to Message 27440.
	The NVIDA driver was changed over a day ago to maximum performance with no change.
	ID: 27443 \| Rating: 0 \| rate: / Reply Quote

skgiven Volunteer moderator Volunteer tester Send message Joined: 23 Apr 09 Posts: 3968 Credit: 1,995,359,260 RAC: 0 Level Scientific publications	Message 27445 - Posted: 27 Nov 2012 \| 14:43:45 UTC - in response to Message 27443. Last modified: 27 Nov 2012 \| 14:53:02 UTC
	Note that you have to set Prefer Maximum Performance again after a reboot. I cant see your system (it's hidden) and I don't know enough about your setup. So a bit of guesswork: It could be downclocking - Do you know what your GPU temps and clocks are? Are you using fan speed control? If not turn this on and set the fan speed to prevent downclocking as a result of overheating. Could be CPU saturation - Are you using all your CPU threads for CPU projects? Free up a thread (or two), and you might want to try setting the niceness. I guess it could also be a driver related issue; maybe it downclocks when the display is off. Posting the first page or your Boinc log files might help. ____________ FAQ's HOW TO: - Opt out of Beta Tests - Ask for Help
	ID: 27445 \| Rating: 0 \| rate: / Reply Quote

RaymondFO* Send message Joined: 22 Nov 12 Posts: 72 Credit: 14,040,706,346 RAC: 0 Level Scientific publications	Message 27446 - Posted: 27 Nov 2012 \| 18:55:26 UTC - in response to Message 27445.
	I made the computer available for viewing for your reference. I am not using any fan speed or any voltage controls, I did briefly suspend other CPU WU's however that did not appear to change anything as the projected run times usually continued their slow net-upward climb (the WU was eventually completed earlier today). I again will try to free up a thread or two later tonight after I download another ACEMD2 work unit. I do not believe it down clocks when the display is off as I had the setting set for "Never," but I will check again. The video card driver software stated Core clock: 815, Shader Clock 1630 MHz, and Memory Clock 2048(?)(Not sure if that is was the exact number but it is close). Thank you for your assistance in this matter. Ray
	ID: 27446 \| Rating: 0 \| rate: / Reply Quote

skgiven Volunteer moderator Volunteer tester Send message Joined: 23 Apr 09 Posts: 3968 Credit: 1,995,359,260 RAC: 0 Level Scientific publications	Message 27449 - Posted: 28 Nov 2012 \| 0:12:31 UTC - in response to Message 27446. Last modified: 28 Nov 2012 \| 0:13:48 UTC
	That task should take less than a tenth of that time on a GTX580. Close to 6K sec, rather than 90K sec. To me it looks like it's both downclocking and starved of CPU resources. CPU usage was very low on the 4.2 task, but this is normal for Fermi's unless you are using SWAN_SYNC=0. The runtime and bad video suggest downclocking, however I noticed that you swapped out a GTX560Ti for a GTX580. This normally requires reinstalling the NVidia driver, and several restarts. Don't know if you did that after installing the GPU or not? I strongly suggest enabling fan control and setting the fan speed to something solid (around 70%). You will have to do this after each restart, just like setting Prefer Maximum Performance, but it solves many issues. Definitely don't use all the CPU cores, or you will literally starve the GPU of CPU support. If you can get it to work you could try using SWAN_SYNC=0, but most people struggle with this and I don't think it's necessary any more. Setting the nice value might also help, but the extent of necessity for doing this depends on what and how much you are crunching on the CPU. What sort of install was it; daemon or manual? Per chance is the system on a modified Preference (Work, School, Home) such as the one WCG employs where it stops crunching so the system can go to sleep after not being used for 30min or whatever it is? Check in Boinc Activities, that Run Always is selected. ____________ FAQ's HOW TO: - Opt out of Beta Tests - Ask for Help
	ID: 27449 \| Rating: 0 \| rate: / Reply Quote

RaymondFO* Send message Joined: 22 Nov 12 Posts: 72 Credit: 14,040,706,346 RAC: 0 Level Scientific publications	Message 27451 - Posted: 28 Nov 2012 \| 4:45:13 UTC - in response to Message 27449.
	Thank you for all your comments. I decided to test for CPU resources being starved and possible down clocking so I shut down all CPU tasks for several minuets, downloaded new GPUGRID work units, and let them run. After several minuets the GPUGRID work units started to behave similar to how a similar video card worked under Windows 7. Over a period of time I slowly released each suspended work unit and continued this process until all work units were running. I observed that the other work units and the GPU now appear to be responding appropriately. I did swap out the video cards under the assumption that maybe the card was possibly defective and unusable. The fan speed is currently at 51% and variable with the video card at 66 C. Not sure under Ubuntu how to change the video card speed. The install was an upgrade from 10.04, then 11.10, and now 12.10. I have checked to make sure this crunching process never goes to sleep. Thank you for all your help and comments. I will definitely keep these comments around to help troubleshoot potential future issues. Thanks again, it is greatly appreciated. Ray
	ID: 27451 \| Rating: 0 \| rate: / Reply Quote

Post to thread

Message boards : Number crunching : Very long run times, poor performance

	About	Science	Volunteers	Performance	Forum	Join us	Donate