Oh, you're right! I remember Boswell had heard the debauched youth and STD stories in 1764, but had forgotten the impotence rumors were making the rounds already. Need not have been Voltaire, then!

I'm increasingly inclined to think Voltaire picked up on existing rumors rather than started them.