'For the individual girl who does not make the team or doesn't get on the stand for the medal ... there's a harm there, and I think we can't sweep that aside,' said Justice Kavanaugh.