solved: puppet master REST API returns 403 when running under passenger works when master runs from command line

Posted by Anadi Misra on Server Fault See other posts from Server Fault or by Anadi Misra
Published on 2012-12-10T09:43:24Z Indexed on 2012/12/10 17:06 UTC
Read the original article Hit count: 242

Filed under:
|
|
|

I am using the standard auth.conf provided in puppet install for the puppet master which is running through passenger under Nginx. However for most of the catalog, files and certitifcate request I get a 403 response.

### Authenticated paths - these apply only when the client
### has a valid certificate and is thus authenticated

# allow nodes to retrieve their own catalog
path ~ ^/catalog/([^/]+)$
method find
allow $1

# allow nodes to retrieve their own node definition
path ~ ^/node/([^/]+)$
method find
allow $1

# allow all nodes to access the certificates services
path ~ ^/certificate_revocation_list/ca
method find 
allow *

# allow all nodes to store their reports
path /report
method save
allow *

# unconditionally allow access to all file services
# which means in practice that fileserver.conf will
# still be used
path /file
allow *

### Unauthenticated ACL, for clients for which the current master doesn't
### have a valid certificate; we allow authenticated users, too, because
### there isn't a great harm in letting that request through.

# allow access to the master CA
path /certificate/ca
auth any
method find
allow *

path /certificate/
auth any
method find
allow *

path /certificate_request
auth any
method find, save
allow *

path /facts
auth any
method find, search
allow *

# this one is not stricly necessary, but it has the merit
# of showing the default policy, which is deny everything else
path /
auth any

Puppet master however does not seems to be following this as I get this error on client

[[email protected] ~]$ sudo puppet agent --no-daemonize --verbose --server bangvmpllda02.XXXXX.com
[sudo] password for amisr1: 
Starting Puppet client version 3.0.1
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /certificate_revocation_list/ca [find] at :110
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /file_metadata/plugins [search] at :110
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /file_metadata/plugins [find] at :110 Could not retrieve file metadata for puppet://devops.XXXXX.com/plugins: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /file_metadata/plugins [find] at :110
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /catalog/blramisr195602.XXXXX.com [find] at :110
Using cached catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /report/blramisr195602.XXXXX.com [save] at :110

and the server logs show

XX.XXX.XX.XX - - [10/Dec/2012:14:46:52 +0530] "GET /production/certificate_revocation_list/ca? HTTP/1.1" 403 102 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:52 +0530] "GET /production/file_metadatas/plugins?links=manage&recurse=true&&ignore=---+%0A++-+%22.svn%22%0A++-+CVS%0A++-+%22.git%22&checksum_type=md5 HTTP/1.1" 403 95 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:52 +0530] "GET /production/file_metadata/plugins? HTTP/1.1" 403 93 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:53 +0530] "POST /production/catalog/blramisr195602.XXXXX.com HTTP/1.1" 403 106 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:53 +0530] "PUT /production/report/blramisr195602.XXXXX.com HTTP/1.1" 403 105 "-" "Ruby"

thefile server conf file is as follows (and goin by what they say on puppet site, It is better to regulate access in auth.conf for reaching file server and then allow file server to server all)

[files]
  path /apps/puppet/files
  allow *
[private]
  path /apps/puppet/private/%H
  allow *
[modules]
  allow *

I am using server and client version 3

Nginx has been compiled using the following options

nginx version: nginx/1.3.9
built by gcc 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) 
TLS SNI support enabled
configure arguments: --prefix=/apps/nginx --conf-path=/apps/nginx/nginx.conf --pid-path=/apps/nginx/run/nginx.pid --error-log-path=/apps/nginx/logs/error.log --http-log-path=/apps/nginx/logs/access.log --with-http_ssl_module --with-http_gzip_static_module --add-module=/usr/lib/ruby/gems/1.8/gems/passenger-3.0.18/ext/nginx --add-module=/apps/Downloads/nginx/nginx-auth-ldap-master/

and the standard nginx puppet master conf

server {
ssl                on;
listen                     8140 ssl;
server_name        _;

passenger_enabled          on;
passenger_set_cgi_param    HTTP_X_CLIENT_DN $ssl_client_s_dn; 
passenger_set_cgi_param    HTTP_X_CLIENT_VERIFY $ssl_client_verify; 
passenger_min_instances    5;

access_log                 logs/puppet_access.log;
error_log                  logs/puppet_error.log;

root                       /apps/nginx/html/rack/public;

ssl_certificate            /var/lib/puppet/ssl/certs/bangvmpllda02.XXXXXX.com.pem;
ssl_certificate_key        /var/lib/puppet/ssl/private_keys/bangvmpllda02.XXXXXX.com.pem;
ssl_crl                    /var/lib/puppet/ssl/ca/ca_crl.pem;
ssl_client_certificate     /var/lib/puppet/ssl/certs/ca.pem;
ssl_ciphers                SSLv2:-LOW:-EXPORT:RC4+RSA;
ssl_prefer_server_ciphers  on;
ssl_verify_client          optional;
ssl_verify_depth           1;
ssl_session_cache          shared:SSL:128m;
ssl_session_timeout        5m;
}

Puppet is picking up the correct settings from the files mentioned because config print command points to /etc/puppet

[[email protected] puppet]$ sudo puppet config print | grep conf
async_storeconfigs = false
authconfig = /etc/puppet/namespaceauth.conf
autosign = /etc/puppet/autosign.conf
catalog_cache_terminus = store_configs
confdir = /etc/puppet
config = /etc/puppet/puppet.conf
config_file_name = puppet.conf
config_version = ""
configprint = all
configtimeout = 120
dblocation = /var/lib/puppet/state/clientconfigs.sqlite3
deviceconfig = /etc/puppet/device.conf
fileserverconfig = /etc/puppet/fileserver.conf
genconfig = false
hiera_config = /etc/puppet/hiera.yaml
localconfig = /var/lib/puppet/state/localconfig
name = config
rest_authconfig = /etc/puppet/auth.conf
storeconfigs = true
storeconfigs_backend = puppetdb
tagmap = /etc/puppet/tagmail.conf
thin_storeconfigs = false

I checked the firewall rules on this VM; 80, 443, 8140, 3000 are allowed. Do I still have to tweak any specifics to auth.conf for getting this to work?

Update

I added verbose logging to the puppet master and restarted nginx; here's the additional info I see in logs

Mon Dec 10 18:19:15 +0530 2012 Puppet (err): Could not resolve 10.209.47.31: no name for 10.209.47.31
Mon Dec 10 18:19:15 +0530 2012 access[/] (info): defaulting to no access for 10.209.47.31
Mon Dec 10 18:19:15 +0530 2012 Puppet (warning): Denying access: Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find] at :111
Mon Dec 10 18:19:15 +0530 2012 Puppet (err): Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find] at :111
10.209.47.31 - - [10/Dec/2012:18:19:15 +0530] "GET /production/file_metadata/plugins? HTTP/1.1" 403 93 "-" "Ruby"

On the agent machine facter fqdn and hostname both return a fully qualified host name

[[email protected] ~]$ sudo facter fqdn
blramisr195602.XXXXXXX.com

I then updated the agent configuration to add

dns_alt_names = 10.209.47.31

cleaned all certificates on master and agent and regenerated the certificates and signed them on master using the option --allow-dns-alt-names

[[email protected] ~]$ sudo puppet cert sign blramisr195602.XXXXXX.com

Error: CSR 'blramisr195602.XXXXXX.com' contains subject alternative names (DNS:10.209.47.31, DNS:blramisr195602.XXXXXX.com), which are 
disallowed. Use `puppet cert --allow-dns-alt-names sign blramisr195602.XXXXXX.com` to sign this request.

[[email protected] ~]$ sudo puppet cert --allow-dns-alt-names sign blramisr195602.XXXXXX.com

Signed certificate request for blramisr195602.XXXXXX.com
Removing file Puppet::SSL::CertificateRequest blramisr195602.XXXXXX.com at '/var/lib/puppet/ssl/ca/requests/blramisr195602.XXXXXX.com.pem'

however, that doesn't help either; I get same errors as before. Not sure why in the logs it shows comparing access rules by IP and not hostname. Is there any Nginx configuration to change this behavior?

© Server Fault or respective owner

Related posts about linux

Related posts about nginx