Resyncing a very stale MongoDB Replica

08/22/2012

I logged into the primary member of the replica set and ran rs.status() which showed me that the replica was too stale (“error RS102 too stale to catch up”):

{
	"_id" : 4,
	"name" : "55.55.55.55:27017",
	"health" : 1,
	"state" : 3,
	"stateStr" : "RECOVERING",
	"uptime" : 502511,
	"optime" : {
		"t" : 1340841938000,
		"i" : 5028
	},
	"optimeDate" : ISODate("2012-06-28T00:05:38Z"),
	"lastHeartbeat" : ISODate("2012-08-22T22:47:00Z"),
	"pingMs" : 0,
	"errmsg" : "error RS102 too stale to catch up"
},

The MongoDB wiki has some instructions on resyncing a very stale replica. I chose to go the simplest route of doing a full resync. To do this I had to figure out where the data was stored, so I looked in /etc/mongodb.conf to see that the dbpath was set to /var/lib/mongodb. Stopping the node, deleting the data directory, and then restarting the node solved the problem. You’ll need the key file (if you’re using auth) and the data directory to both exist with the proper ownership and permissions to bring the node back up.