cherrypy fails to stop when puppet tries to ensure running and refresh it at the same time
- by ento
I am trying to manage a cherrypy service with puppet. However, when the configuration is applied, cherryd ends up with no PID file although the process is up and running. Since the PID file is lost I can no longer stop the process with /etc/init.d/mycherryd stop (unless I modify the handmade init script to lookup the PID with ps or something.)
$ /etc/init.d/mycherryd status
cherryd dead but subsys locked
The problem seems to be that puppet is trying to refresh/restart cherryd (triggered by changes in configuration files) immediately after ensuring it's running (as specified in the manifest), and cherrypy fails to stop and start (restart) itself while still executing its startup process.
Is there a clear cut solution to this problem? Is this a bug on the cherrypy side, or can I write a puppet manifest so refresh is called only after the service is up and running? Any suggestion welcome.
cherrypy log
See how cherrypy catches SIGTERM midway through startup and still starts to listen.
:cherrypy.error[18666] 2010-02-12 13:10:23,551 INFO: ENGINE Listening for SIGHUP.
:cherrypy.error[18666] 2010-02-12 13:10:23,552 INFO: ENGINE Listening for SIGTERM.
:cherrypy.error[18666] 2010-02-12 13:10:23,552 INFO: ENGINE Listening for SIGUSR1.
:cherrypy.error[18666] 2010-02-12 13:10:23,552 INFO: ENGINE Bus STARTING
:cherrypy.error[18671] 2010-02-12 13:10:23,554 INFO: ENGINE Daemonized to PID: 18671
:cherrypy.error[18671] 2010-02-12 13:10:23,554 INFO: ENGINE PID 18671 written to '/var/mycherryd/cherry.pid'.
:cherrypy.error[18671] 2010-02-12 13:10:23,555 INFO: ENGINE Started monitor thread '_TimeoutMonitor'.
:cherrypy.error[18670] 2010-02-12 13:10:23,556 INFO: ENGINE Forking twice.
:cherrypy.error[18666] 2010-02-12 13:10:23,557 INFO: ENGINE Forking once.
:cherrypy.error[18671] 2010-02-12 13:10:23,716 INFO: ENGINE Caught signal SIGTERM.
:cherrypy.error[18671] 2010-02-12 13:10:23,716 INFO: ENGINE Bus STOPPING
:cherrypy.error[18671] 2010-02-12 13:10:23,716 INFO: ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 12380)) already shut down
:cherrypy.error[18671] 2010-02-12 13:10:23,717 INFO: ENGINE Stopped thread '_TimeoutMonitor'.
:cherrypy.error[18671] 2010-02-12 13:10:23,717 INFO: ENGINE Bus STOPPED
:cherrypy.error[18671] 2010-02-12 13:10:23,732 INFO: ENGINE Bus EXITING
:cherrypy.error[18671] 2010-02-12 13:10:23,759 INFO: ENGINE PID file removed: '/var/mycherryd/cherry.pid'.
:cherrypy.error[18671] 2010-02-12 13:10:23,782 INFO: ENGINE Bus EXITED
:cherrypy.error[18671] 2010-02-12 13:10:23,792 INFO: ENGINE Serving on 0.0.0.0:12380
:cherrypy.error[18671] 2010-02-12 13:10:23,826 INFO: ENGINE Bus STARTED
puppet log
puppet tries to refresh the service immediately after ensuring it to be 'running'.
Feb 12 13:10:22 localhost puppetd[8159]: (//mycherrypy/File[conffiles]) Scheduling refresh of Service[cherryd]
Feb 12 13:10:22 localhost last message repeated 12 times
Feb 12 13:10:23 localhost puppetd[8159]: (//mycherrypy/Service[mycherryd]/ensure) ensure changed 'stopped' to 'running'
Feb 12 13:10:23 localhost puppetd[8159]: (//mycherrypy/Service[mycherryd]) Triggering 'refresh' from 13 dependencies
Feb 12 13:11:23 localhost puppetd[8159]: (//mycherrypy/Service[mycherryd]) Failed to call refresh on Service[mycherryd]: Could not stop Service[mycherryd]: Execution of '/sbin/service mycherryd stop' returned 1: at /etc/puppet/manifests/mycherrypy.pp:161
Feb 12 13:11:24 localhost puppetd[8159]: Value of 'preferred_serialization_format' (pson) is invalid for report, using default (marshal)
Feb 12 13:11:24 localhost puppetd[8159]: Finished catalog run in 99.25 seconds
puppet manifest excerpt
class mycherrypy {
file {
'rpm':
path => "/tmp/${apiserver}.i386.rpm",
source => "${fileserver}/${apiserver}.i386.rpm";
'conffiles':
require => Package["${apiserver}"],
path => "${service_home}/config/",
ensure => present,
source => "${fileserver}/config/",
notify => Service["mycherryd"];
}
package {
"$apiserver":
provider => 'rpm',
source => "/tmp/${apiserver}.i386.rpm",
ensure => latest;
}
service {
"mycherryd":
require => [File["conffiles"], Package["${apiserver}"]],
ensure => running,
provider => redhat,
hasstatus => true;
}
}
