Rick could have shut down any argument to them going back by the group, if he'd really wanted to be a dick, by reminding them of the Governor.
Rick: "Nope, never saw HIM again. Hey, Herschel, what do you think ever happened to that guy? Herschel? ...oh, yeeeeeaaah."