Pardon the stupid question but before you excluded the masked areas form the SNR calculation, were they showing as SNR = 0 dB (which is my first thought) or are they showing as SNR = "huge dB", because the signal level is still available to the SNR calculation (despite the mask) and the noise in the masked area is basically zero?
Apologies for the late reply. I don't read here very often.
The answer to your question is "it depends". The SNR ranking algorithm takes a sample of all the waterfall bins (in dB), sorts them, uses the value of the 50% bin (median) as the "noise" level, and the 95% bin as the "signal" level. The difference (95% value - 50% value) is taken as the SNR. Prior to eliminating the masked bins this means that if there were enough of them to effect the 50% noise level value (after sorting) the SNR would become huge (noise level = -200 dB or something).