Stream an upload and read a download in chunks

You upload a file straight from disk without reading it into memory first, then read a download piece by piece as it arrives.

fibre_http = "0"

Set up the project#

Start a new binary crate.

cargo new streaming
cd streaming

Put this in the [dependencies] section of Cargo.toml:

[dependencies]
fibre_http = { version = "0", features = ["json"] }
serde_json = "1"

The json feature is only there to read httpbin's answer to the upload.

The program#

The program writes a small file to upload, sends it to httpbin.org's /put and then downloads 100,000 bytes from /stream-bytes. Replace src/main.rs with this:

use std::fs::File;
use std::io::Write;

use fibre_http::SyncClient;
use serde_json::Value;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut file = File::create("upload.txt")?;
    for line in 0..2000 {
        writeln!(file, "line {line} of the upload")?;
    }
    drop(file);

    let client = SyncClient::new();

    let file = File::open("upload.txt")?;
    let length = file.metadata()?.len();
    let echo: Value = client.put("https://httpbin.org/put").body_reader(file, Some(length)).send()?.error_for_status()?.json()?;
    println!("sent {length} bytes, the server counted {}", echo["headers"]["Content-Length"]);

    let mut body = client.get("https://httpbin.org/stream-bytes/100000?chunk_size=10000").send()?.error_for_status()?.into_body();
    let mut chunks = 0;
    let mut total = 0;
    while let Some(chunk) = body.chunk()? {
        chunks += 1;
        total += chunk.len();
    }
    println!("read {total} bytes in {chunks} chunks");
    Ok(())
}

Run it:

$ cargo run
sent 46890 bytes, the server counted "46890"
read 100000 bytes in 12 chunks

The number of chunks changes from run to run, since it depends on how the bytes arrive over the network. The total is always 100000.

What each part does#

body_reader takes anything that implements std::io::Read and sends it as the request body while reading it. Passing the length lets the client send a Content-Length header, which some servers require. Pass None when you do not know the length and the body goes out in chunks instead.

A streamed body can only be read once. So a request with one is never retried. A 307 or 308 redirect comes back to you as the response instead of being followed.

into_body() hands you the response body without reading it. Each call to chunk() returns the next piece as it arrives and None at the end. A blocking body also implements std::io::Read, so std::io::copy works on it too.

If you drop a body before its end, the client closes that exchange so the connection is not left holding unread bytes.

Where to go next#

The async client has the same calls: body_stream sends a stream of byte chunks and chunk().await reads a body piece by piece. The usage guide has both: https://github.com/excsn/fibre/blob/main/http/README.USAGE.md