The high-flying Forest, currently positioned third in the Premier League, triumphed 4-2 in the shootout at St James Park after the match concluded in a 2-2 draw.
Exeter took an early lead through Josh Magennis in the fifth minute, but Forest responded with goals from Ramon Sosa and Taiwo Awoniyi to take the lead by halftime.
Magennis equalized just five minutes into the second half, and Exeter was reduced to ten men when Ed Turns received a red card in the 87th minute.
Despite their numerical advantage, Forest was unable to secure a victory in extra time, leading to a penalty shootout where Neco Williams converted the decisive penalty.
Forest will face Ipswich in the fifth round.