EC2 instance suddenly refusing SSH connections and won't respond to ping
- by Chris
My instance was running fine and this morning I was able to access a Ruby on Rails app hosted on it. An hour later I suddenly wasn't able to access my site, my SSH connection attempts were refused and the server wasn't even responding to ping. I didn't change anything on my system during that hour and reboots aren't fixing it.
I've never had any problems connecting or pinging the system before.
Can someone please help? This is on my production system!
OS: CentOS 5
AMI ID: ami-10b55379
Type: m1.small
[] ~% ssh -v *****@meeteor.com
OpenSSH_5.2p1, OpenSSL 0.9.8l 5 Nov 2009
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to meeteor.com [184.73.235.191] port 22.
debug1: connect to address 184.73.235.191 port 22: Connection refused
ssh: connect to host meeteor.com port 22: Connection refused
[] ~% ping meeteor.com
PING meeteor.com (184.73.235.191): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
^C
--- meeteor.com ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
[] ~%
========= System Log =========
Restarting system.
Linux version 2.6.16-xenU ([email protected]) (gcc version 4.0.1 20050727 (Red Hat 4.0.1-5)) #1 SMP Mon May 28 03:41:49 SAST 2007
BIOS-provided physical RAM map:
Xen: 0000000000000000 - 000000006a400000 (usable)
980MB HIGHMEM available.
727MB LOWMEM available.
NX (Execute Disable) protection: active
IRQ lockup detection disabled
Built 1 zonelists
Kernel command line: root=/dev/sda1 ro 4
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 65536 bytes)
Xen reported: 2599.998 MHz processor.
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Software IO TLB disabled
vmalloc area: ee000000-f53fe000, maxmem 2d7fe000
Memory: 1718700k/1748992k available (1958k kernel code, 20948k reserved, 620k data, 144k init, 1003528k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 5202.30 BogoMIPS (lpj=26011526)
Mount-cache hash table entries: 512
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
Checking 'hlt' instruction... OK.
Brought up 1 CPUs
migration_cost=0
Grant table initialized
NET: Registered protocol family 16
Brought up 1 CPUs
xen_mem: Initialising balloon driver.
highmem bounce pool size: 64 pages
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
i8042.c: No controller found.
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
Xen virtual console successfully installed as tty1
Event-channel device installed.
netfront: Initialising virtual ethernet driver.
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
NET: Registered protocol family 2
Registering block device major 8
IP route cache hash table entries: 65536 (order: 6, 262144 bytes)
TCP established hash table entries: 262144 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
TCP bic registered
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 15
Using IPI No-Shortcut mode
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 144k freed
***************************************************************
***************************************************************
** WARNING: Currently emulating unsupported memory accesses **
** in /lib/tls glibc libraries. The emulation is **
** slow. To ensure full performance you should **
** install a 'xen-friendly' (nosegneg) version of **
** the library, or disable tls support by executing **
** the following as root: **
** mv /lib/tls /lib/tls.disabled **
** Offending process: init (pid=1) **
***************************************************************
***************************************************************
Pausing... 5Pausing... 4Pausing... 3Pausing... 2Pausing... 1Continuing...
INIT: version 2.86 booting
Welcome to CentOS release 5.4 (Final)
Press 'I' to enter interactive startup.
Setting clock : Fri Oct 1 14:35:26 EDT 2010 [ OK ]
Starting udev: [ OK ]
Setting hostname localhost.localdomain: [ OK ]
No devices found
Setting up Logical Volume Management: [ OK ]
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1
/dev/sda1: clean, 275424/1310720 files, 1161123/2621440 blocks
[ OK ]
Remounting root filesystem in read-write mode: [ OK ]
Mounting local filesystems: [ OK ]
Enabling local filesystem quotas: [ OK ]
Enabling /etc/fstab swaps: [ OK ]
INIT: Entering runlevel: 4
Entering non-interactive startup
Starting background readahead: [ OK ]
Applying ip6tables firewall rules: modprobe: FATAL: Module ip6_tables not found.
ip6tables-restore v1.3.5: ip6tables-restore: unable to initializetable 'filter'
Error occurred at line: 3
Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.
[FAILED]
Applying iptables firewall rules: [ OK ]
Loading additional iptables modules: ip_conntrack_netbios_ns [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0:
Determining IP information for eth0... done.
[ OK ]
Starting auditd: [FAILED]
Starting irqbalance: [ OK ]
Starting portmap: [ OK ]
FATAL: Module lockd not found.
Starting NFS statd: [ OK ]
Starting RPC idmapd: FATAL: Module sunrpc not found.
FATAL: Error running install command for sunrpc
Error: RPC MTAB does not exist.
Starting system message bus: [ OK ]
Starting Bluetooth services:[ OK ]
[ OK ]
Can't open RFCOMM control socket: Address family not supported by protocol
Mounting other filesystems: [ OK ]
Starting PC/SC smart card daemon (pcscd): [ OK ]
Starting hidd: Can't open HIDP control socket: Address family not supported by protocol
[FAILED]
Starting autofs: Starting automount: automount: test mount forbidden or incorrect kernel protocol version, kernel protocol version 5.00 or above required.
[FAILED]
[FAILED]
Starting sshd: [ OK ]
Starting cups: [ OK ]
Starting sendmail: [ OK ]
Starting sm-client: [ OK ]
Starting console mouse services: no console device found[FAILED]
Starting crond: [ OK ]
Starting xfs: [ OK ]
Starting anacron: [ OK ]
Starting atd: [ OK ]
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 390 100 390 0 0 58130 0 --:--:-- --:--:-- --:--:-- 58130
100 390 100 390 0 0 56984 0 --:--:-- --:--:-- --:--:-- 0
Starting yum-updatesd: [ OK ]
Starting Avahi daemon... [ OK ]
Starting HAL daemon: [ OK ]
Starting OSSEC: [ OK ]
Starting smartd: [ OK ]
c
CentOS release 5.4 (Final)
Kernel 2.6.16-xenU on an i686
domU-12-31-39-00-C4-97 login:
INIT: Id "2" respawning too fast: disabled for 5 minutes
INIT: Id "3" respawning too fast: disabled for 5 minutes
INIT: Id "4" respawning too fast: disabled for 5 minutes
INIT: Id "5" respawning too fast: disabled for 5 minutes
INIT: Id "6" respawning too fast: disabled for 5 minutes
