Only Orange smarties are flavored.
For those who don't get Smarties in their countries, they are sugar coated chocolates. The coating comes in many different colors, much like M&Ms. However, a little known secret behind these delicious candies is that the orange colored Smartie has orange flavoring! If you have any smarties from your Halloween candy, taste the coating of each one.
They all have the same flavor except for the orange ones which tastes like oranges! This flavoring is added in a separate process during manufacturing
(Source)