Moving the speaker closer to a boundary generally affects the baffle loading, not interference with the port. By nominally increasing the width of the baffle by placing it close to the wall, the frequency where the baffle step begins to rise is lowered, which will amplify the lower frequencies. Speakers can be baffle step compensated, so if the xover has already been designed to lower output relative to the baffle step, it would be bad to place the speaker too close to the wall.
Gerg, depending on the cabinet depth, can't the rear port be every bit as in phase as the front port? As for the drywall acting as a bass trap, wouldn't that be in effect no matter which direction the port fired? I guess it may be exacerbated if it fired straight at the drywall. I certainly haven't run any tests on this myself.