The first means to id a sig is the ear/brain. After you've heard them a few times you begin to know what mode is what and understand the main differences between the most commonly found types. These are fsk, mfsk, psk, ofdm, fax, etc.
You can also look at the logs submitted by members of groups like udxf, hfunderpants, etc to see what, if anything, has been noted previously on the frequency in question.
If you have a lot of money (or are familiar with torrents, cough) you might obtain software decoders such as wcode, k500, hoka, etc, and these often have a routine for determining signal type, rate, bw, etc. You can help these routines by pointing them in the right direction, such as using an fsk analyser on fsk sigs, a psk analyser on a psk signal etc. For example, wcode and hoka have routines for determining rate, sense, and language of a given S4285 signal, and will automatically start decoding once the parameters have been determined - that is, for a non encrypted signal. I suppose most people start out today with sorcerer, multipsk, and sigmira, and advance from there.
You can use an am demodulator to do some signal id work;
find a psk signal, form a bandpass filter around it to exclude noise, and look for a peak on the audio frequency scale, this will be the bitrate, ie 2400 in the case of a S4285 signal.
There are online resources to help, one is udxf, another is the sigidwiki, and there are others. This is one of my favs but be warned it can be math heavy;
http://i56578-swl.blogspot.com/There are all sorts of books online and at libraries on signals analysis and so on and so forth, employ your google fu.
http://www.kd0cq.com/2013/07/sorcerer-decoder-download/http://f6cte.free.fr/index_anglais.htmhttp://www.saharlow.com/technology/sigmira/