but I suppose there were all sorts of cultural associations that went along with the job. Not sure what exactly is implied here--that they slept around and were also prostitutes? But often they were married to soldiers, as I recall.

Yeah, I'm pretty sure the Duc is using "vivandière" as a synonym for "camp follower", which gets used (not just by him, by many people) as a synonym for "prostitute". In fact, that's the first meaning in which I encountered "camp follower" when I started reading history as a teenager, and it's still my primary association. Actual demographics means nothing when faced with the prejudices and stereotypes of someone like the Duc.