A senior councillor who blamed “natural churn” for the decline of Huddersfield’s high street earlier this year says he does not regret his comments despite some backlash.