James Garfield had a brief affair with a Mrs. Calhoun in New York City in 1862. Garfield's wife caught them.