That's what got her, of course. That everyone thought it so unbelievable that she could possibly attract a man like him. It shouldn't upset her because it was true. She couldn't. Not in this world, in this lifetime. Yet she didn't appreciated everyone else acting as if they were the most improbable twosome since Quasimodo hit on Esmeralda.
