Need some help with PF rule letting two machines access each other

[Pat Maddox]

12.34.56.78 runs a server on port 1234
87.65.43.21 should connect to this

Both of them have PF rulesets that block off most traffic, keeping
open the publically available ports I need open.  In this case though,
any traffic over this port should only be between these two machines.
I've tried to set this up, but I keep getting operation not permitted,
connection refused, and connection reset by peer errors.  Thanks for
any info.

Pat