Given this evidence, should I be worried that I may have been hacked

[Jim Stapleton]

Once I opened up SSH to the outside world, my machine has been
hammered once or twice a day most days, with username failures. None
of the  usernames would fit a username on my system (except root), and
I have ssh set to deny root logins, and only use SSH2. Additionally, I
have the following in my login.access (only active entry, the name
have been changed on this, but the three names would appear as 3 and
four character random alphabetical strings):
-:ALL EXCEPT wrbc crr aqp:ALL EXCEPT local

As of the 9th, I've only seen one set of blatant/brute-force attempt
at my ssh server. It's interesting, but the major drop in attempts has
me more worried than the attempts (could this drop off be because they
 no longer need to hack me? Could they have hacked me an that be the
reason why?)

How worried should I be, and what's the best recourse for this?

Thanks,
-Jim Stapleton