Slow Download Speeds from AWS S3

[Sydney Meyer]

Hi Steve,

> On 16 Feb 2017, at 18:18, Steven Hartland <killing at multiplay.co.uk> wrote:
> 
> It does seem to be related to TCP Receive Window Size.
> 
> When I tested on a 11.0-RELEASE box I got 30MB/s out of the box and the increase of recvspace was only needed on the original 10.2 however the 11 box is only 1.2ms from AWS where as the 10.2 was 17ms away, so likely explains the difference.

Yes, i saw this too.. when i did some tests with a VM from within the same AWS region, i did not see the troughput issues as with connections with a higher latency (probably should have mentioned this in the first post, retrospectively).. this would also explain the vmware guest bridged vs nat'ed difference then..

> 
> It seems that when talking to hosts that don't support Timestamps FreeBSD's performance is going to be poor due to lack of auto scaling :(

I did some tests with Debian 8.x (Linux Kernel 3.16.x) and there the default for /proc/sys/net/ipv4/tcp_timestamps appears to be enabled, at least with Debian's 3.16 Linux Kernel. I can see the TS val field now in tcpdump which is lacking in traffic from AWS S3, regardless from region.

> 
> Reading the thread you linked is seems like we need to implement auto scaling based on RTT estimations similar to linux.

Well, i guess the issue described in the related thread is indeed the case here.
I will adjust this buffer size manually on this machine, as it has no other major purpose than pulling archives from S3 anyway.

But thanks again, learned something new (to me) about FreeBSDs networking stack.

Updated the Forums Thread accordingly here:

https://forums.freebsd.org/threads/59756/

Cheers,

Sydney.

> 
>    Regards
>    Steve
> 
> On 16/02/2017 16:55, Sydney Meyer via freebsd-net wrote:
>> Hi Steve,
>> 
>> increasing the buffer size did indeed enhance throughput.
>> 
>> I am obviously not an expert in this field, but i don't understand why or if the TCP Receive Window Size shouldn't increase automatically.
>> 
>> I found this thread on the ML and i'm reading myself a bit more into the topic right now.
>> 
>> https://lists.freebsd.org/pipermail/freebsd-net/2014-August/039495.html
>> 
>> Thank you for your tip, however..
>> 
>> Sydney
>> 
>>> On 16 Feb 2017, at 16:35, Steven Hartland <killing at multiplay.co.uk> wrote:
>>> 
>>> Window scaling and receive buffer scaling is the most likely cause.
>>> 
>>> Check what the sysctl net.inet.tcp.recvspace is set to, then try increasing it e.g.
>>> sysctl net.inet.tcp.recvspace=655360
>>> 
>>> This jumped the transfer rate with a wget and your test URL from 3MB/s to 30MB/s here.
>>> 
>>>    Regards
>>>    Steve
>>> 
>>> On 16/02/2017 14:34, Sydney Meyer via freebsd-net wrote:
>>>> Hello,
>>>> 
>>>> I'm seeing continuous slow download speeds from Amazon S3, but only on FreeBSD. Other OSes saturate the connection without problems.
>>>> 
>>>> This happens with 10.3-RELEASE and 11.0-RELEASE and only with AWS S3 in different regions (Ireland, London, Frankfurt, US Standard have been tested) whilst using fetch, curl, et. al.
>>>> 
>>>> I have tested this on multiple machines, bare metal, bhyve, Xen and VMware VMs, routed setups and direct pppoe links.. all the same.
>>>> 
>>>> Anyone seeing similar issues?
>>>> 
>>>> Here's a url to try:
>>>> 
>>>> http://s3.eu-central-1.amazonaws.com/4f48caf1d8bcbef8/c5b38f8b3625d2b6/zerofile.raw
>>>> 
>>>> Also, when doing double NAT, i.e. VMware Fusion FreeBSD Guest with "Share with my Mac"-Interface, the machine is doing completely fine, as in saturates the link, 90Mbps otherwise between 12-15Mbps..
>>>> 
>>>> I also switched the FreeBSD Routers with Linux-based ones and with the isp-provided router box, with the same result.
>>>> 
>>>> I have launched VMs with Digital Ocean to "rule out" my ISP and there seems to be the same issue. Downloading from S3 is multiple times slower than any other connection to services outside of S3 or on any other OS.
>>>> 
>>>> It seems like other people are seeing the same issue:
>>>> 
>>>> https://forums.freebsd.org/threads/59756/#post-343064
>>>> _______________________________________________
>>>> freebsd-net at freebsd.org mailing list
>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>> _______________________________________________
>>> freebsd-net at freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>> _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
> 
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"