What's Bred in the Bone by Grant Allen

Elma Clifford wasn't the only person who passed a terrible night and suffered a painful awakening on the morning after the Holkers' garden-party. Colonel Kelmscott, too, had his bad half-hour or so before he finally fell asleep; and he woke up next day to a sense of shame and remorse far more definite, and, therefore, more poignant and more real than Elma's.

Hour after hour, indeed, he lay there on his bed, afraid to toss or turn lest he should wake Lady Emily, but with his limbs all fevered and his throat all parched, thinking over the strange chance that had thus brought him face to face, on the threshold of his honoured age, with the two lads he had wronged so long and so cruelly.