My server running magento is 4gb ram and 4 core cpu. But still i am struggling with the high CPU usage. I only have 10 visitors at any given point of time. I am not sure if the PHP has to take this high % CPU usage. Attached is the TOP result.

top - 09:18:32 up 2 days, 15:44,  1 user,  load average: 1.16, 2.02, 1.99
Tasks: 179 total,   2 running, 177 sleeping,   0 stopped,   0 zombie
Cpu(s): 46.7%us,  3.9%sy,  0.1%ni, 46.9%id,  1.0%wa,  0.0%hi,  0.0%si,  1.4%st
Mem:   3919972k total,  3164968k used,   755004k free,   530820k buffers
Swap:  1048568k total,   379352k used,   669216k free,  1536388k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15897 vpsadmin  20   0  431m 168m  54m R 91.7  4.4   2:16.16 php-cgi
12308 vpsadmin  20   0  404m 163m  73m S 29.3  4.3  15:15.90 php-cgi
 3644 mysql     20   0 1528m  80m 4944 S  9.8  2.1   1899:58 mysqld
 4969 apache    20   0  471m 6228 2824 S  2.0  0.2   0:18.53 httpd
16148 root      20   0 15024 1220  864 R  2.0  0.0   0:00.01 top
    1 root      20   0 19364 1064  844 S  0.0  0.0   0:02.50 init