net/http: add Transport.ReadBufferSize and WriteBufferSize
Previously transport was using the hardcoded bufio.defaultBufSize
(4096), limiting throughput and increasing cpu usage when uploading or
downloading large files.
Add options to allow users to configure the buffer sizes as needed.
I tested the maximum benefit of this change by uploading data from
/dev/zero to a server discarding the bytes. Here is an example upload
using the default buffer size:
$ time ./upload 10 https://localhost:8000/
Uploaded 10.00g in 25.13 seconds (407.49m/s)
real 0m25.135s
user 0m5.167s
sys 0m11.643s
With this change, using 128k buffer size:
$ time ./upload 10 https://localhost:8000/
Uploaded 10.00g in 7.93 seconds (1291.51m/s)
real 0m7.935s
user 0m4.517s
sys 0m2.603s
In real world usage the difference will be smaller, depending on the
local and remote storage and the network.
See https://github.com/nirs/http-bench for more info.