During the years away from James Brown, Sweets worked with: Al Green, Take 6, Ice-cube, Bootsy (for Snoop Dogg) and many others (also producing gospel groups around Nashville).
In 1976 Fred Wesley quitted James Brown and Mr. Brown made Sweets his musical director and bandleader instead. Charles stayed on till October 1996, then he too quitted because he got enough of Mr. Brown's games and ugly remarks (Mr. Brown made up in 2004).
A few months later, Maceo Parker asked Charles if he wanted to join his band.
(Charles and Maceo used to room together during their James Brown years.)
And so from 1996 till 2004 Sweet Charles toured with Maceo Parker.