My guess is you would have to connect it to touch, taste and smell.
For instance, I'd associate red with heat. A campfire, the outside of a saucepan which has been cooking on the stovetop, the taste of a fresh chili pepper; all of those things can be felt and are commonly associated with the color red.
I'd also associate red with strong tastes and smells, such as strawberries, tomatoes, watermelons and cranberries. Provide each food in question and let the person taste (and smell) the differences.
You can do the same thing for most colors, I suspect.
Yellow in the warmth of the sun on your skin or the peculiar taste of an egg yolk.
Green in the soft, but prickly, feel of new grass or the smell of a lime or taste of a zucchini.
And so on.
