Then his two young children die three months apart:
His son, William Comstock Sprague (born October 26, 1857) dies of "bilious diarrhea" on July 27, 1860, while visiting in Astoria , Long Island, New York, .
His daughter, Mary Waterman Sprague (born July 24, 1850) dies of diphtheria on October 21, 1860, in Providence.
In 1864, the Spragues must contend with the "Texas Affair." In this scandal, Byron's famous cousin, William Sprague, is implicated in a scheme of smuggling guns to Texas in exchange for cotton for the Spragues' cotton mill back in Cranston, Rhode Island. The act, if true, would have been treason.
Byron and another chap named Reynolds were arrested and locked up for a time. But little evidence and lots of political clout add up to freedom for Byron. A short-lived victory to be sure, for Byron Sprague dies two years later, in 1866.