I mean in operating systems or their applications. The only way I can think of is examine binaries for the use of dangerous functions like strcpy(), and then try to exploit those. Though with compiler improvements like Visual Studio's /GS switch this possibility should mostly be a thing of the past. Or am I mistaken?

What other ways do people use to find vulnerabilities? Just load your target in a debugger, then send unexpected input and see what happens? This seems like a long and tedious process.

Could anyone recommend some good books or websites on this subject?

Thanks in advance.