One of the biggest problems with Macs (at least as far as gaming is concerned) is the limited availability for graphics cards. Apple doesn’t really provide any way to upgrade the graphics card beyond what it ships with, which is a real shame. Fortunately, the folks over at netkas.org (amongst others) have made some good progress in to making high-end cards from ATI and nVidia work on Mac Pros. It takes a bit of tinkering, but the results are well worth it.

Disclaimer

Some of the steps involve messing about with core functionality of the operating system. If you don’t feel comfortable doing this, then please do not proceed. You should of course, have a complete and up-to-date backup before proceeding in any case.

As the Nvidia cards don’t have an EFI Rom, you’ll be unable to see the boot screen, which means you can’t boot into recovery mode. For this reason, you should hang on to your original graphics card to be able to (temporarily) swap them back as needed.

Mountain Lion

For OS 10.8 (and possibly 10.7.5 which shares the same drivers, but we haven’t tested it) onwards, the process is greatly simplified. Just install the hardware and you’re good to go.

What you’ll need – hardware

At Mac Gamer, we use the very solid Nvidia GTX 570 (specifically, the [amazon_link id=”B004XVMAKU” target=”_blank” container=”” container_class=”” ]MSI Twin FROZR III[/amazon_link], not least of all because it runs the quietest of similar cards). Other cards may work, check the netkas.org and macrumors forums.

You can also use a “Kepler” (6XX) series card too, see our guide for more information.

There are two main issues that make this process tricky: power and software support. The power issue is less of a concern, particularly if you pick the right graphics card. Basically, almost every Mac Pro ships with a graphics card that uses a single 6-pin PCI-E power cable (that runs from the motherboard to the card). However, every Mac Pro motherboard actually has two sockets available in order to power more demanding graphics cards- you just need an additional cable. You can usually find 6-pin to 6-pin PCI-E power cables for Macs on eBay, though there are probably other places to get them. The only thing to make sure of is that the graphics card you intend to use requires 2 x 6-pin power connectors (the [amazon_link id=”B004XVMAKU” target=”_blank” container=”” container_class=”” ]MSI Twin FROZR III[/amazon_link] we use does).

Finally, if you can get your hands on an additional Mac-compatible graphics card that doesn’t require external power (such as the GT 120), this will act as a “helper” card, allowing you to see the boot screen and boot OS X even if something isn’t configured correctly. You can occasionally find these on eBay as well, but make sure you get one appropriate to your Mac Pro model (Mac Pro 1,1 require a different card to the others). This step is optional however, if you don’t get the helper card, you can still resolve problems by swapping the original graphics card back in.

What you’ll need – software

You should be running OS 10.7.2 or higher. (OS 10.6 will not work very well). You’ll also need to be logged into an account with administrative privileges. Download the following:

• Official nvidia OS X driver (there’s an issue with the current version of this driver)
• ATY_Init kext (for Mac Pro 1,1 get this version instead)
• Kext Helper

(For OS 10.8, none of these are needed).

It’s also recommended (but not necessary) to download the Mac CUDA driver, which will add an icon to the System Preferences application and will also notify you about any driver updates (it also provides a handy way to verify which driver version you’ve got).

Installing the hardware

In terms of actually installing the hardware, it’s merely a matter of removing the original graphics board (note that newer Mac Pro’s have a locking bar to hold the graphics card in place that must be swung out of the way first), disconnecting the power cables, and then installing the new one and connecting both power cables to the mew one. However, some of the following steps get a little bit tricky, so I would strongly recommend reading through the rest of the steps first, and you will need to be prepared to constantly swap the original and new video cards back and forth until the drivers and patches are woking correctly.

You should try and do as much of the software installation as possible before swapping out the hardware, as all of the software modifications should work even if you haven’t installed the new hardware yet.

One thing it’s worth pointing out, at this point you can run Windows without any problems. The latest nVidia windows drivers will work fine in Boot Camp if you’re using it, and it’s a good way to ensure the hardware is installed correctly. However, you’ll need to have set your startup disk to always boot straight into Boot Camp, as you won’t be able to see the boot screen.

Installing the software

As noted, try to perform these steps before installing the actual hardware, so you can see what you’re doing. Note that these steps are only necessary for OS 10.7.4 or lower (but will be kept here for posterity).

Add support for your card (this is the complicated part): as per these instructions, open Terminal and enter the following command:

sudo nano -wT2 /System/Library/Extensions/NVDAGF100Hal.kext/Contents/Info.plist

Enter your password. Below the line 0x124010de&0xffc0ffff add:

0x108610de&0xffc0ffff

(note: depending on the card you have, you need to enter a different 4 numbers after the first “x”, which is the device ID of the card – refer to this list to find out yours)

Press control-o and control-x

Install the nVidia drivers. If you have a Mac Pro 1,1 or 2,1, you will get an error. To work around this, copy the “270.00.00f06 Web Release.mpkg” to your desktop, right-click and “Show Package Contents” on it. Then edit the Distribution.dist with a text editor. Add your model to the line:

var system_models = [“MacPro3,1”, “MacPro4,1”, “MacPro5,1”];

Note: As of OS 10.7.4, this should be all you need to do. However, early indications are that this will not be sufficient for future updates to the operating system, so we recommend performing the following steps too if you upgrade.

1. Open kext helper.
2. Drag the extracted ATY_Init.kext bundle onto the kext helper window. Enter your password, and click “EASY INSTALL”. (Ocassionally when doing this, we’ve found the system will hang. Reset the power and you should be good to go).
3. Quit kext helper (it will sometimes crash when exiting, don’t worry if it does).
4. Reboot

Note: For 10.7.4, the card will only show as “NVIDIA Graphics Card 512 MB” but should work fine. Luxmark (see below) and presumably other OpenCL-supported applications will recognise the card correctly.

Boosting performance

There are a few things you can do to maximise performance.

First of all, you can follow these instructions to improve the power management settings. (Not required with 10.7.4)

See steps 2 and 3 of this thread to enable OpenCL (OS 10.7.4 or lower only).

You can verify improvements by using the OS X version of the LuxMark testing suite.

Tips & troubleshooting

In many cases, if nothing shows on the screen, the system will still boot. If you are comfortable doing so (and you have previously enabled it) you will be able to SSH into your Mac Pro from another system and verify many of the steps through the command-line. Otherwise, we recommend constantly swapping the new and original cards, and going through the software installation steps again. We also had success using remote desktop to “see” what was going on in some situations.

You can also try the following:

Rebuild the caches. To do so, type the following commands into a terminal window (be careful not to mis-spell these):

sudo rm -rf /System/Library/Caches/com.apple.kext.caches/Startup/kernelcache
sudo kextcache -system-prelinked-kernel
sudo kextcache -system-caches

Open disk utility and repair permissions. This advice tends to be given in every Mac troubleshooting thread ever, but in this case it will actually help. To repair permissions from the terminal, use:

diskutil repairPermissions /

Reboot the system. To reboot via the terminal, use:

sudo shutdown -r now

You can check the netkas.org and macrumors forums for advice (if you’re a beginner, you’ll probably find that the folks at macrumors are more patient with you).

Boot Camp

Because you can’t see the boot screen any more, it becomes a bit trickier to start Boot Camp. However, you can use the free BootChamp utility to boot into Windows directly from OS X.

Upgrading the OS

You may find that the display stops working after upgrading the OS. For example, after updating to OS 10.7.4, our display was blank. We had to open an ssh session into the system from another computer, and perform the “add support for your card” steps above (and the enabling OpenCL steps if desired), then rebuild the caches and repair permissions. After rebooting the system, everything was back to normal. If that doesn’t work for you, try repeating the other steps too.

Upgrading to OS 10.8 Mountain Lion required no additional work.

In closing

This method has been tested and works for OS 10.7.2 and higher. 10.7.5 has not yet been tested, but indications are that it should work. However, it is important to note that there are no guarantees that Apple won’t make changes to the OS that prevent these hacks from working. That said, if you want the best performance out of your Mac (and if you use Boot Camp), this will really give your system a boost in the graphics department. We’ll try to keep this post up-to-date with the latest information. None of this would be possible without the work of the people over at netkas.org.

If you are able to follow these instructions successfully, consider donating some money to netkas so that he can continue with his excellent work. You just spent a few hundred dollars on a graphics card after all!

Note: These instructions were last tested with OS 10.8.2 and updated accordingly.

Artefacts

You may occasionally see small black squares in the corner of the screen when running a graphically intensive game. This has been found to be caused by the driver package from Nvidia’s website, so for now just use the standard 10.7.4 drivers in conjunction with ATY_Init.

Update – 10.7.5: It looks like the forthcoming 10.7.5 update will add somewhat native support for Nvidia fermi cards. The included Nvidia drivers (which are also available in the 2012 MacBook Pro update package, and are at driver version 7.28.1 295.10.15f03) improves performance (by around 30% judging by LuxMark scores), resolves the artefacts issues in previous drivers, and doesn’t require any editing or installation of third-party extensions. We’ll update this post fully when it’s readily available.

Update – 10.8: Adds native support for Fermi cards. You still won’t see the boot screen, but all you need to do is install the hardware now, OpenCL works out of the box (and rather well, with a score of 1200 on LuxMark). Driver version is 8.0.51.295.30.00f01