FreeBSD 5.1; max amount of memory

[anton menshutin]

I have wrote a program that needs a lot of memory. But the reason is that it doesn't need a lot of physical RAM simultaneously, and system swapping is a good solution and it should work well. 
Previously i was using 4.7-RELEASE, but it seems that it can't work with big enought  address space. Init don't want to work with such options in kernel config :
options         MAXDSIZ="(2048*1024*1024)"
options         MAXSSIZ="(256*1024*1024)"
options         DFLDSIZ="(2048*1024*1024)".
Application could not allocate more than 1G of memory.
That's why I try to use the 5.1. 

But under this version of system my program works even worse.
Here what the program says after I have set additional options to malloc:
dla in malloc(): error: out of memory
And this is a vmstat listing at the end of execution:
procs      memory      page                    disks     faults      cpu
 r b w     avm    fre  flt  re  pi  po  fr  sr ad0 ad2   in   sy  cs us sy id
 0 1 2  609840 108284   10   0   0   0   0   0   0   0  341    0 328 49  1 50
 0 1 1  609888 108156    8   0   0   0   0   0   0   1  340    0 321 49  1 50
 0 1 1  609936 108024   11   0   0   0   1   0   0   0  340    0 322 49  1 50
 0 2 0  609968  86620    7   0   0   0  31   0   0  69  406    0 726 38  3 59
 0 2 0  609968  87732    8   0   0   0 2192   0   0 203  545    0 1424  0  6 94
At this stage application is using only 526 Mb of memory. The system is a 2 Athlon SMP with 1G of RAM and 2G swap.

Does anybody know what options and where one should set to solve such a problem?
Should I try another version of FreeBSD?