Thomas Ryder Pepper married Anne Bloomfield, daughter of John Bloomfield and Anne Charlotte Waller. He lived at Loughton House which built in 1777 on lands owned by the Pepper family. The Pepper family lived at Loughton House until Thomas Pepper died as a result of a hunting accident. Thomas Pepper requested in his will that his brother-in-law, the 1st Lord Bloomfield, Benjamin Bloomfield, acquire Loughton House.