Fix corner-case in handling of remaining retry time