He has appeared in films including The Madness of King George (1994), Gosford Park (2001), Vera Drake (2004), The History Boys (2006), The Kings Speech (2010), Les Misérables (2012) and 1917 (2019). He is also known for his roles in television such as Cranford (2007–2009), Upstairs Downstairs (2010–2012), The Paradise (2013), Crashing (2016), A Very English Scandal (2018), and Killing Eve (2019).

Scarborough is also an accomplished theatre actor and won the Laurence Olivier Award for Best Actor in a Supporting Role in 2011 for his role in the Terence Rattigan play After the Dance. He will appear in Tom Stoppard's new play Leopoldstadt from January 2020 at Wyndham's Theatre in London.