Conversely, and also possible: he might have been afraid Madame Denis would remarry and leave him, otherwise, at least early on.

Davidson quotes numerous letters early on where she keeps talking about getting married to someone else, and Voltaire keeps freaking out, so yes, there is that.