pf starts blocking all traffic after a short while

I thought I understood pf pretty well, but this one puzzles me.
I have a very simple setup here - a machine I omnly want to allow
public IPv6 in from one place, allow private Ipv4 from its local
neighbours, and be able to connect out to anywhere.

Seems to work, I boot it up, I can ssh in. After about five
minutes it just starts blocking all traffic.  I have serial
console access, so I can still examine the machine, and if, when
it is stuck, I load a pf config files which allows everything, then
traffic resumesd again, which is what makes me think pf is doing this.

Heres the rules, all eleven of them...

root@joanna-may:~ # pfctl -s rules
scrub all max-mss 1200 fragment reassemble
block return all
pass quick proto icmp all keep state
pass quick proto ipv6-icmp all keep state
pass in inet from 127.0.0.0/8 to any flags S/SA keep state
pass in inet from 192.168.0.0/16 to any flags S/SA keep state
pass in inet from 172.16.0.0/12 to any flags S/SA keep state
pass in inet from 10.0.0.0/8 to any flags S/SA keep state
pass in inet6 from 2001:470:6cc4::/48 to any flags S/SA keep state
pass in inet6 from 2001:470:1f08:1771::2 to any flags S/SA keep state
pass out all flags S/SA keep state

Nothing particularly controversial there I think!

I've checked the states table, theres a handful in there, and they look fine.
If I ssh in and run top, then that connection eventually drops when the packet
flow ceases. The pf table is left with a state of TIME_WAIT in it.

Any ideas ? This is a mchine inside AWS, so not real hardware, but that
should not make a difference I think... Its also the only time I have used pf
without using NAT, so maybe I have issed something, but really, this was
supposd to be a very simple ruleset to do a very simple job.

-pete.