The OS X port found here
http://subdimensions.com/2013/03/25/hdsdr-osx-2-60-port/ seems like it will work, but since I only have an RTL dongle I haven't been able to get it to recognize the dongle. From the "CAT EXTIO Config ENG" file:
Choose your radio
1. Put the folder "HDSDR OSX" on your desktop
2. Go to http://sourceforge.net/apps/mediawiki/hamlib/index.php?title=Supported_Radios
3. Find the manufacturer of your radio and click on models
4. Find the model of your radio and take note of the "Rig#"
5. Right click on "HDSDR OSX" and chose "Show package contents"
6. Double click on "drive_c", "Program Files", "HDSDR"
7. Open with TextEdit "ExtIO_HamLib.cfg"
8. Find "# Rig Number" text
9. On the next row delete "1" and insert your "Rig#"
10. Save
11. Run the application "Terminal" located in "Applications", "Utility"
12. Type: ls /dev/tty.* and press enter
13. Identify the USB to RS232 conversione interface e.g. "/dev/tty.usbserial"
Setup serial port - Method A
1. Copy and paste in terminal: cd ~/"desktop/hdsdr osx/hdsdr osx.app/contents/resources/dosdevices" then press enter
2. Type: ln -s YOUR-USB-RS232-DEVICE com1 e.g. "ln /dev/tty.usbserial com1"
I can't find a rig number for an rtlsdr.
The HDSDR - Hardware page
http://www.hdsdr.de/hardware.html lists a .dll file for RTLSDR (DVB-T/DAB with RTL2832) which I downloaded and replaced the the ExtIO_HamLib.cfg file with that one. Still no luck.
When showing the package contents of HDSR OSX I noticed that the ExtIO_HamLib.cfg located in 'drive_C/Program Files/HDSDR/' is Highlighted in red. I assume this indicated some error in the file but I'm not sure what it is.
Also, running "ls /dev/tty.*" in terminal only brings up "/dev/tty.Bluetooth-Modem and /dev/tty.Bluetooth-PDA-Sync.