He does sound quite sincere in this, yes, and I had a look at both Pleschinski and the original French and see that my perception might have been influenced a tiny bit by free translation - for example, he translated "je me suis vu engagé à soutenir sa vérité" as "an einer Klärung interessiert", which is a bit more open and ambiguous. If I remember my first reading correctly, I wasn't quite sure what to make of Fritz' thoughts at that point myself, but! - this isn't the end of the topic in their correspondence. Voltaire responds with a whole ode to Remusberg, says that as always he has a different opinion than the monks, but also that "Remus probably would have been as astonished to find himself in paradise as in Prussia". And then, Fritz' final words on the topic, which led me to my interpretation that he's fully aware of the questionableness of the anecdote but loved it nonetheless:

I only gave you Remus' story for what it is worth. The origins of nations are for the most part fabled; they only prove the antiquity of the foundations. Put Remus' anecdote next to the story of the holy ampulla and Merlin's magical deeds.

[Je ne vous ai donné l'histoire de Rémus que pour ce qu'elle vaut. Les origines des nations sont pour la plupart fabuleuses; elles ne prouvent que l'antiquité des établissements. Mettez l'anecdote de Rémus à côté de l'histoire de la sainte ampoule et des opérations magiques de Merlin.]