Concerns have been raised that a new scheme to convert a Victorian farmhouse and build new homes could have a detrimental impact on a village and Grade-II listed hall.