Allow "remove from recent" only for top-level