James Evan Duffy was born in New York City on July 1, 1932. He attended public schools in White Plains, New York and went on to Clemson College in South Carolina. Duffy graduated from Clemson with a Bachelor of Science in Textile Engineering in 1954. That same year, he married Carol Letitia Brugh. They have two children, Jacquelyn Carol and James Evan. Duffy has worked as a manager in the textile industry since 1957. He has been active in civic and church activities and has served as class agent for the Clemson Alumni Association.

Beginning in the early 1960s, Duffy became active in South Carolina Republican politics. In 1962, he supported the Workman for Senate campaign. He served as delegate to the Anderson County Republican Convention in 1961, 1962, and 1964; State Committeeman for Anderson County, 1961-1965; Third District Ballot Security Officer, 1962; Chairman of the Republican State Convention Steering Committee, 1964; member of the Credentials Committee at the 1964 National Convention; Presidential Elector From South Carolina for Barry Goldwater, 1965; State Ballot Security Officer, 1964-1968; member of the National Republican Finance Committee, 1966; Chairman and Secretary of the Second District Republican Party, 1966; State GOP Organizational Director, 1968; Chairman of the Ravenel for Governor Campaign, 1969-1970; member of the Re-Elect the President Committee, 1970; member of the Workman for Governor Finance Committee; and Chairman of the Third Congressional District Baker for President Committee, 1980. Duffy was active in many Republican campaigns at the local, state, and national level during the 1960s and early 1970s.