Don't Do It !
It will hurt the value of your property as it limits what people can do to add value. In addition satisfying the "Heretics" Committee is time consuming and will cost you extra money. The people on these committees mean well and my hat goes off to them but they don't care about your investment value so don't partner with them.
I live in a Heritage district and enjoy the environment but am fortunate my home is not a Heritage Home. I pick my own paint colors, don't need their approval for renovations, new front door etc.
There is only one small benefit that I have seen. Some Heritage committees/ towns have a bit of money and will contribute when you make some changes that they like.
Dave