Advanced search

Message boards : Number crunching : How to run 4 Nvidia GTX295 on Asrock X58 (tutorial)

Author Message
Profile Hydropower
Avatar
Send message
Joined: 3 Apr 09
Posts: 70
Credit: 6,003,024
RAC: 0
Level
Ser
Scientific publications
watwatwatwatwatwat
Message 10119 - Posted: 24 May 2009 | 1:19:21 UTC

This explains some of the pitfalls you will run into when trying to run 4 GTX295 cards in an Asrock X58 motherboard. I have built such a system for Bletchley Park (www.bletchleypark.com /.nl ) as a study and here are some of the findings.

1. You need to make sure you upgrade the bios to a version higher than 2.0

2. Do NOT use the Nvidia 18x drivers. Use 178.13 instead.

3. A tweak is needed for the drivers to recognize the card. I will post this as an additional message.

4. You will need a 64 bit operating system, I can only recommend Windows XP64 bit. I tried several 32 and 64 bit OSes and none worked properly with the 4 cards combined. With 8 GPUs you will run out of system RAM under 32 bit windows because each card needs a piece of address space to work, whatever is left is given to the system as addressable RAM regardless of how much you installed.

5. You need to make sure the motherboard is fed enough amps @ 12 volt to feed the cards over the pcie bus. This means that the powersupply must provide for a maximum of 34 A @ 12 volt on ONE rail for the motherboard. (4 x 6 A for each GTX on the pcie slots and 10 A for the rest of the motherboard components). In addition to that each card should be provided with enough power to absorb peak performance. This means that each card preferably should have 40 A @ 12 V split over the 6 and 8 pin connector. In practice this means you will need 3 power supplies to feed all components and provide for headroom. there are people running it fine on 2 powersupplies but this does not allow for peak draws. (all of this is debatable ofcourse). I used four 700 W / 50 A powersupplies.

You may be encouraged to know it took me a month to figure this all out and to get it running. If you appreciate these instructions, please join team Bletchley Park.

____________
Join team Bletchley Park, the innovators.

Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist
Send message
Joined: 14 Mar 07
Posts: 1947
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 10122 - Posted: 24 May 2009 | 9:29:45 UTC - in response to Message 10119.

The problem of the drivers is quite annoying.
Is GPUGRID working on those drivers? We had to upgrade to 180.xx on our machines.
gdf

Profile Hydropower
Avatar
Send message
Joined: 3 Apr 09
Posts: 70
Credit: 6,003,024
RAC: 0
Level
Ser
Scientific publications
watwatwatwatwatwat
Message 10125 - Posted: 24 May 2009 | 10:24:10 UTC - in response to Message 10122.

Yes it does.

There is the occasional glitch as you can see from my statistics. Use of BOINC is a delicate process. It requires a clean machine, with Precision tool run and terminated before BOINC is run. Running Precision (the Evga clocking tuner) with BOINC active is asking for trouble. Suspending tasks is another delicate process. I usually make a backup before BOINC is run. I stop BOINC with network communication off so that a restart with errors will not communicate to the server and waste my previous work. On this system GPU 3 is sensitive, almost all computer errors you will find in my stats are from GPU 3, so I have underclocked its RAM now by 100 MHz with Precision.
____________
Join team Bletchley Park, the innovators.

Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist
Send message
Joined: 14 Mar 07
Posts: 1947
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 10174 - Posted: 25 May 2009 | 22:11:42 UTC - in response to Message 10125.

I think that the new applications will require 180 at least.

gdf

Profile Hydropower
Avatar
Send message
Joined: 3 Apr 09
Posts: 70
Credit: 6,003,024
RAC: 0
Level
Ser
Scientific publications
watwatwatwatwatwat
Message 10320 - Posted: 30 May 2009 | 8:23:49 UTC - in response to Message 10174.

If we are currently running the new applications that means they can run on the 178.13 driver. If I adhere to underclocking the RAM on GPU 3 (which is a hardware issue) then I have no problems. What version of the application are you referring to ? I run 6.6.4.
____________
Join team Bletchley Park, the innovators.

Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist
Send message
Joined: 14 Mar 07
Posts: 1947
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 10332 - Posted: 30 May 2009 | 19:30:44 UTC - in response to Message 10320.

From now on, new applications will be compiled with CUDA2.1, so they will require updated drivers. Nothing to worry yet.

It would be interesting to have a setup under Linux 64, as we do not have Windows hosts.

gdf

Profile skgiven
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 14004 - Posted: 20 Dec 2009 | 15:56:05 UTC - in response to Message 10332.

I think they are all 2.3 now.

Post to thread

Message boards : Number crunching : How to run 4 Nvidia GTX295 on Asrock X58 (tutorial)

//