We had a recent election in our replica set (2 read nodes; 1 write node) that changed the primary node. Curious as to why this occurred, I started looking through the logs to find out what happened.
It appears that mongoNode2 could not communicate with mongoNode3. When both nodes could not communicate, it appears that this caused the services on mongoNode2 and mongoNode3 to restart, eventually resulting in a new primary after the services had been started again.
Thu Jun 23 08:27:28 [ReplSetHealthPollTask] DBClientCursor::init call() failed
Thu Jun 23 08:27:28 [ReplSetHealthPollTask] replSet info mongoNode3:27017 is down (or \
slow to respond): DBClientBase::findOne: transport error: mongoNode3:27017 query: { \
replSetHeartbeat: "myReplSet", v: 3, pv: 1, checkEmpty: false, from: \
"mongoNode2:27017" }
Thu Jun 23 08:27:29 got kill or ctrl c or hup signal 15 (Terminated), will \
terminate after current cmd ends
Thu Jun 23 08:27:29 [interruptThread] now exiting
Thu Jun 23 08:27:29 dbexit:
Is there any reason that the mongo service would restart due to a DBClientCursor::init call() failure? Is this a known bug?
It should be noted that mongoNode2 and mongoNode3 are VMs on the same VMware host. MongoNode1 is not on the same host, and it did not have any issues with the service. However, I did not have any other reports of issues with other VMs on the VMware host.
