Improvising over changes takes many years of dedicated practice to master. It is a highly intellectually demanding act that requires knowledge of music theory as well as an excellent memory and quick thinking. Once players get a basic grasp of Jazz improvisation it is often hard for them to let go and play by ear again, as they did when they first started to improvise. Long before they were thinking about symmetrical scales or tri-tone subs they just followed their ears and let their fingers do the walking. Once they are blowing bebop lines through changes with some proficiency they find it hard to trust only their ears to navigate for them. I often try to get my students to move outside by disengaging their rational minds for just a moment at a time. It's almost harder to get an intellectual player to play by ear than it is to teach an ear player to learn Jazz harmony.
I like hearing bebop players who are able to step outside without using set harmonic formulas. There are many post-bop players who use harmonic devices in order to take it outside, fewer who are able to play freely by ear and then drop right back inside. It can be hard to convince a student who has worked hard to play over changes to ignore them, even if it is just for short periods of time. I may cross out a few chord changes and tell them to just blow whatever they hear over those bars. I tell them not to play anything that is directly harmonically related to the changes that the rhythm section is playing. They should land on the next written chord change with a strong chord tone. I try to get them to feel comfortable with playing totally outside by ear for just a bar at a time. This is like popping the clutch in a stick shift vehicle. The rational mind is forced to disengage from its calculations and computations while the ears and the fingers momentarily take the reigns. When the 'clutch' is re-engaged the rational mind takes over again without losing it's place in the tune. After the student is comfortable with one bar of cosmic freak out I'll have them try for a few more bars at a time.
It's also nice to work your way outside and then work your way back inside using chord substitutions. For example let's take the first five bars of the bridge of 'What is this thing called love?'
The written changes are:
C-7 /F7 /Bbmaj7 / /Ab7 (b9) /
Let's try playing the first chord of the bridge and then work our way out using strong resolutions, then right before the Ab7(b9) we'll play a few changes to get us back inside. So.....
C-7 B7 /E7 A7/FREAK /OUT! E-7Eb7/Ab7 (b9) /
By beat three we are starting to head outside, culminating in a six beat cosmic freak out in the third bar and the first half of the fourth bar. Beats three and four steps us back inside where we land on terra firma in bar five. Unscathed!
This example shows how we can gradually move outside using standard diatonic harmony, play free for a few moments and then step back inside without anyone knowing what hit them. This way the psychedelic freak out is woven into the tonal harmony seamlessly. It doesn't come as so much of a shock (which isn't always bad) to the listener and the transition outside and back will be much smoother. You'll be able to play like Archie Shepp even at a Bar Mitzvah or your hotel lobby gig!
Listen to George Garzone or Ellery Eskelin for their ability to step across the line between inside and outside playing with ease. Free playing doesn't always have to drive the grandmothers out of the room (my grandmother used to ask me when she came to my gigs if I was going to play any of that 'drive the grandmothers out of the room music'). Grandma won't even know that anything's wrong before you're back from your full fledged FREAK OUT.