Teamrx8 says:

A 10" wide wheel is the max width you can use that sticks close to the factory offset and takes advantage of all the inboard wheel space @ 50 - 52mm offset. With a fat tire (285 or 295) it will fill out the fender and also look mean as hell with all that rubber under there. You also aren't going to experience handling and driveability issues that result from wheel offsets lower than factory.

If you want to run wider than 10" then you have no choice but to start lowering the offset because there is no more inboard space available without extreme mods like tubbing and major suspension relocation/redesign. With every width increment above a 10" width @ +50-52mm offset you'd have to move the offset out by half that amount.

So for 1/2" increase above 10" width (10.5") half that amount is 1/4" = 6.3mm or 7mm for practical purposes. At each 1.5" increase you could subtract 1mm to account for the fractions adding up to a whole number, but there aren't too many RX8s using 11.5" wide wheels let alone 13" wide, lol. At anything past 10.5" wide using this basis the fender will need pulling or flaring for an equally wide tire not to rub because it will be sticking too far outboard of the factory fender lip to clear. So the resulting size/offsets would be:

10.0" @ +50 to +52mm
10.5" @ +43 to +45mm
11.0" @ +36 to +38mm
11.5" @ +30 to +32mm
12.0" @ +23 to +25mm

Otherwise running undersize tires to accomplish this fails on several accounts. To make the tire sidewall lean for fender clearance means that it has to lean both on the inside and the outside. A ridiculously narrow tire is required the further outboard the wheel lip sits outside the fender lip. Way back when tire sidewalls lacked good support but todays performance tires are designed such that a wheel wider than the tread width/contact patch width accomplishes no performance benefit. So a wide wheel with a high positive offset just to put a skiny tire on it is pointless and silly. Some wheel width/tire size reference points are:

9" wheel @ 245 to 255 tire
10" wheel @ 275/285 tire
11" wheel @ 305/315 tire
12" wheel @ 335 tire

Some change in offset from the original +50mm (S1 models) is not going to have much impact, but as you continue to push it out the resulting scrub radius moment arm is creating a force that makes the tires want to turn out, putting the steering components in constant tension while driving in a straight line. As such a setup encounters uneven pavement, bumps, water puddles etc. that moment arm is going to try and rip the steering wheel out of your hand position. Even more so for these conditions under braking because this same scrub radius moment arm is going to magnify the braking deceleration forces against the steering wheel as well.