In attempting to categorize threats to privacy, one of the more salient categories is certainly surveillance. Of course, some surveillance (warranted searches, cameras in banks, etc.) may not offend our sensibilities, while others (unwarranted wiretapping, use of the PATRIOT Act) may. But, regardless of whether we consider any given instance of surveillance to be just, one fact is inherent to all: surveillance or the threat of surveillance alters human behavior.
For example, when we have reason to believe that the police monitor a certain stretch of road for speeders, we may alter our speeds while traversing that area - regardless of whether there is, in fact, a police officer hiding around the bend. We don't know if we are being watched, but we could be. Of course, this is an innocuous example. One doesn't have to think too hard to come across examples of the threat of surveillance stifling human creativity or attempting to normalize social eccentricities (McCarthy-ist America, Soviet Russia, etc.).
One of the great metaphors used for the threat of surveillance is Jeremy Bentham's Panopticon. The Panopticon is Bentham's design for a prison. What's interesting about this prison is that it uses the threat of surveillance to keep prisoners in check. The Panopticon is a round structure with glass-walled cells lining the circumference of the circle - all facing toward the circle's center. In the center of the circle is the guard tower, an all-seeing monolith (think Barad Dur) that could be filled with guards that could be looking at any particular cell. The prisoners cannot see inside the guard tower, and so they have no way to know whether they are being watched. As such, the most logical thing to do is to behave, just as we slow down when we pass by the potential speed trap. Bentham argues that this device of using the threat of surveillance to keep order reduces operating costs for the prison, as it would not need to retain a full staff of guards.
While this is certainly not an original thought on my behalf (Foucault and others have written about the Panopticon as a metaphor for surveillance in society), I think it's a very useful way of weighing situations where surveillance should be allowed. We should always question whether the x that we are attempting to protect against (in cases where security conflicts with privacy) with surveillance is worth the cost.