Check and clean the MAF and the ESS(Eccestric Shaft Position Sensor), are there any CEL? Since you have a 2004, I wouldn't rule out the engine mount(s).
Mazda TSB of testing the motor mounts:
- On manual transmission vehicles, when decelerating in 3rd gear with engine speed less than 3,000 RPM at approximately 30 MPH (50km/h), then quickly depressing the accelerator pedal again, a hesitation may occur. This symptom can only be observed in this engine speed and vehicle speed range.
2004 SS GT 6Spd MT @177,850 (reman' engine after 167,305 original factory) miles
RAZO RA66 400 gm Shift ****-2006 Switchblade Key-JDM corners
Upgraded Starter-Mishimoto Radiator-ViS CF/FRP Hood-YAKIMA roofrack(STOLEN)
18 x 9.5 +38 RAYS Engineering G-L 57F DUNLOP DIREZZA DZ 101 (245/40-18)