Saffron is from a flower and is considered a spice tho. It might be the way it’s processed that makes a difference. Regardless, when talking of the East India Company, the spice trade, and how that led to conquests, tea is always an important part of that story.
Poppy seeds are definetely also a spice tho. And coca is an herb, which I guess can also be used as a spice… Use of coca by native populations seems to have been mostly medicinal… But then again, that’s also how many spices were used until the 19th century.