How to resolve high CPU + excessive stat("/etc/localtime") and clock_gettime(CLOCK_REALTIME) calls

Posted by Yemster on Server Fault See other posts from Server Fault or by Yemster
Published on 2012-11-28T01:15:31Z Indexed on 2012/11/28 11:05 UTC
Read the original article Hit count: 293


I've been experiencing really high CPU on a ruby on rails app (see stack below) and have been trying to diagnose the possible causes to no avail.

Stack:

  • ruby 1.9.3
  • rails 3.2.6
  • Apache/2.2.21 (Debian)
  • Phusion Passenger 3.0.11

Whenever I run strace against the spiking Rack process PID (see Top excerpt below), I am seeing a tonne of stat("/etc/localtime") and clock_gettime(CLOCK_REALTIME) calls and have no idea how to stop these.


Excerpt from Top showin running PID:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
11674 www-user  20   0  313m 182m 5076 R   99  2.3  63:04.60 Rack: /var/www/my_rails_app/current
11634 www-user  20   0  411m 216m 5144 S   10  2.7 197:55.63 Rack: /var/www/my_rails_app/current


Strace snippet below:

[pid 11674] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 11674] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 11674] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 11674] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 11674] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354058955, 141474018}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354058955, 141577456}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354058955, 143073982}) = 0
[pid 11674] poll([{fd=15, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
[pid 11674] write(15, "b\0\0\0\3SELECT `images`.* FROM `ima"..., 102) = 102
[pid 11674] read(15, "\1\0\0\1\0229\0\0\2\3def\23myappy_productio"..., 16384) = 2063
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354058955, 144138035}) = 0
[pid 11674] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 11674] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 11674] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 11674] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
...
[pid 11674] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 11674] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 11674] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 11674] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354058955, 154076443}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354058955, 154189429}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354058955, 157185700}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354058955, 157298770}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354058955, 165076003}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354058955, 165212572}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354058955, 167542679}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354058955, 167683436}) = 0
....
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354060036, 62052248}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354060036, 62182486}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354060036, 62919948}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354060036, 63057266}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354060036, 63751707}) = 0
 [pid 11674] clock_gettime(CLOCK_REALTIME, {1354060036, 73730686}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354060036, 75874687}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354060036, 76077133}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354060036, 78205019}) = 0
...
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354060036, 89370879}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354060036, 89583247}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354060036, 91637614}) = 0
[pid 11674] clock_gettime(CLOCK_REALTIME, {1354060036, 91782149}) = 0


Have Google'd around and came across a number of suggestions which I've tried with no success.


Things tried so far:

  1. Have tried setting time zone as recommended here
    Made no difference and issue still persists.
    Content of my /etc/localtime:

    TZif2UTCTZif2UTC
    UTC0

  2. Have tried the recommended fix for the leapsecond bug:

    date -s 'date'


No joy so far.


I'm fresh out of ideas so any help/advice on how to diagnose or resolve would be greatly appreciated.


© Server Fault or respective owner

Related posts about debian

Related posts about ruby-on-rails