Choosing an Upload Path
File size, privacy, scanning, processing and serving decide the architecture. There is no default that is right for all five.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
Given this specific file feature, should the bytes go through my backend, straight to storage, or somewhere else entirely?
Three features shipping this quarter: profile avatars, support-ticket attachments, and raw video for a media product. Someone asks whether they should all use the same upload mechanism.
Pick one approach and apply it everywhere. Either everything goes through the API because that is simplest, or everything uses presigned URLs because that is what scales.
Presigned URLs for a 40 KB avatar adds an intent row, an event pipeline, a reconciliation job and a completion state machine to replace a fifteen-line handler (Presigned URLs).
- Presigned URLs for a 40 KB avatar adds an intent row, an event pipeline, a reconciliation job and a completion state machine to replace a fifteen-line handler (Presigned URLs).
- Proxying multi-gigabyte video through your API instances puts a hard ceiling on file size and makes fifty concurrent uploads an outage (File Uploads Through the Backend).
- A rule that ignores scanning gets it wrong in one direction: a feature where users download each other's files needs a scanning stage regardless of which path the bytes took.
- A rule that ignores serving misses the larger cost. For a media product, egress dominates the bill and the upload path is close to irrelevant (Serving Files).
- Uniformity is genuinely valuable for maintenance, and it is worth choosing deliberately rather than by defaulting — one path for the common case, an explicit exception for the outlier.
What is actually happening
- Five inputs decide this, and they are close to independent: size distribution (including the tail, not the median), privacy (may anyone read the object, or only specific principals), inspection (must you validate or scan before the file is usable), processing (does something have to happen to the bytes afterwards), and serving (how often and how widely it is read).
- Size decides whether the request path is viable at all. Small files through the backend cost a little memory and no architecture; large files through the backend cost a request slot for the duration of the client's upload, which is a concurrency budget you do not control (Resource Limits).
- Inspection decides whether "the bytes never touch my process" is acceptable. If a file must be validated before it exists anywhere, the request path is the only place that guarantee lives; if scanning can happen after, direct upload plus a processing state machine works (What Happens After the Bytes Land).
- Privacy decides serving, and serving decides cost. A private object needs a signed read URL or a proxy on every access; a public object can be cached at a CDN edge and stops costing you per read (CDN as Infrastructure).
- Processing decides whether "uploaded" and "usable" are the same state. If they are not, the file has a lifecycle and the API must expose it — which is a contract decision, not an implementation detail (The Async Job Pattern).
- Direct browser upload adds constraints that native clients do not have: CORS on the bucket, a preflight for typed PUTs, and progress reporting that must come from the browser's own upload events.
The decision
Answer for one feature at a time. The inputs are close to independent, so a feature can sit anywhere in the space — small and private and scanned, or large and public and untouched.
How big are the files, who may read them, and what must happen to the bytes before they are usable?
when Small bounded files (avatars, CSV imports, logos) that must be validated or transformed before they exist, and where an immediate result is part of the UX.
cost Memory and a request slot per upload; a hard size ceiling; you pay ingress and egress for every byte (File Uploads Through the Backend).
when Medium files where you must see the bytes — content validation, format checks — but cannot afford to buffer them.
cost Harder handler code, awkward error handling mid-stream, and the request slot is still occupied for the transfer.
when Large files, high concurrency, or mobile clients on unreliable networks. Anything where the transfer time is dominated by the client's uplink.
cost No pre-storage inspection; a completion signal, an intent lifecycle and reconciliation to build (Presigned URLs).
when Large files that still need scanning, transcoding or thumbnails — the common case for media products.
cost A visible state machine: uploaded is not ready, and the API contract must say so (What Happens After the Bytes Land).
when Very large files, or any file uploaded over a mobile network where a dropped connection is likely.
cost Per-part signing, client-side complexity, and abandoned parts that need a lifecycle rule to clean up.
when Images or video that are always transformed and always served publicly, and transformation is the product requirement rather than storage.
cost A vendor in the hot path of a core feature, its own pricing model, and migration cost if it stops fitting.
Three features, three answers
The same team, the same quarter, the same codebase — and no shared answer. This table is the argument for deciding per feature, and for writing the reason next to the decision.
| Avatar | Ticket attachment | Raw video | |
|---|---|---|---|
| Typical size | ~50 KB | 100 KB – 25 MB | 200 MB – 5 GB |
| Who may read it | Anyone who sees the profile | Only ticket participants | Only the owner, until published |
| Must inspect before storing? | Yes — it is resized immediately | No, but must be scanned before download | No; scanned and transcoded after |
| Processing after upload | Synchronous resize | Virus scan, thumbnail for images | Scan, transcode, multiple renditions |
| Upload path | Through the backend | Through the backend, streamed | Direct, resumable, signed |
| Serving | Public object behind a CDN | Signed read URL per request | CDN for renditions, signed for originals |
| Dominant cost | Negligible | Storage | Egress, then transcoding compute |
| If you got it wrong | Nobody notices | Malware reaches other customers | Instances OOM under normal use |
The threshold has to be enforced where it matters
A two-path design is only as good as the boundary between them. If the client decides which path to use, the boundary is a suggestion — and the failure it produces (a huge body arriving at the API path) is exactly what the split was meant to prevent.
- 1Client declares intent
Sends filename, declared type and declared size to the API
fails by Being believed — all three are claims (The Trust Boundary)
- 2Authorize
May this principal attach to this object, and is it within quota?
fails by Only being present on one of the two paths
- 3Choose the path
Declared size below the threshold → proxied; above → signed URL
fails by Letting the client choose, so a declared 1 MB arrives as 1 GB
- 4Bind the limit
Proxied: enforce during body read. Direct: bind size in the signature where the provider supports it
fails by Trusting the declared size as the enforcement
- 5Record intent
A row with status
pending, owner, tenant, key and expected typefails by Only existing on the direct path, so the two paths need different completion handling
- 6Converge
Both paths end at the same "object stored, mark ready-for-processing" step
fails by Two divergent code paths whose validation drifts apart
- 7Process and publish
Scan, transform, then flip status to
readyfails by Serving the file while it is still
pending(What Happens After the Bytes Land)
The convergence step is the one worth insisting on. Two upload paths are acceptable; two validation implementations are not, because the weaker one becomes the one an attacker uses.
How to build it
Most important first.
- Decide per feature, not per codebase, and write down the reason. "Avatars go through the API because they are small and we resize synchronously" is a decision; "we use presigned URLs" is a habit.
- Set a size threshold and make it explicit. Below it, through the API; above it, direct. Enforce the threshold at the edge so a client cannot choose the wrong path (Payload Size: 20KB, 200KB, 5MB).
- If anything must be checked before the file is usable, add a status field from the start —
pending,scanning,ready,rejected— regardless of upload path. Retrofitting a lifecycle onto a boolean is painful. - Keep buckets private by default and decide serving separately: signed read URLs for private, CDN in front for public and hot (Serving Files).
- For direct browser uploads, configure and test CORS before you build the feature. It is the failure that costs an afternoon and produces no useful error message.
- Consider a third option honestly: for small images that are always transformed and always public, a specialized media service handles upload, processing, storage and delivery as one product, and replaces most of this decision (Mapping Services Across Cloud Providers).
What can go wrong
- A threshold that exists in the client and not the server, so a client that ignores it sends a 2 GB body to the API path.
- Two upload paths that diverge in their validation, so the direct path accepts files the proxied path would reject.
- A "ready" state assumed by the frontend before processing has finished, producing broken thumbnails and empty previews (Eventual Consistency in Practice).
- CORS misconfiguration discovered in production because staging used a native client (CORS Without the Myths).
- A cost model built on storage that omits egress, so the bill arrives shaped completely differently from the estimate (Egress: Moving Data Costs Money, Not Just Storing It).
- A scanning stage added later to a feature where files were already downloadable, leaving a backlog of unscanned objects nobody wants to own.
- A user requesting a file between upload completion and processing completion — the object exists and is not yet usable, which is why a status field beats inferring readiness from existence.
- If large uploads go through the API with no enforced limit, an attacker gets a memory-exhaustion denial of service on every instance, cheaply (File Uploads Through the Backend).
- If direct upload is chosen for a feature that needed pre-storage inspection, an attacker gets to place arbitrary content in your bucket, and whether it is ever inspected depends on a pipeline that may lag or fail (File Upload Security).
- If the bucket is made public to simplify serving, an attacker gets read access to every object in it, and key unguessability is the only thing standing between them and your users' files (Public Exposure, Read With Context).
- If two upload paths have different validation, an attacker gets to choose the weaker one. Shared validation code, or one path, is the control.
- If a file is downloadable while still in
pending, an attacker gets to distribute unscanned content through your product to your other users.
- "Direct upload is the modern way." It is the right way for large files and a needless state machine for small ones.
- "We can decide later." The API contract differs: one returns the finished attachment, the other returns a pending resource that becomes ready. That is a client-visible difference (The Async Job Pattern).
- "Scanning can be added afterwards." Technically yes; the backlog of already-downloadable unscanned files is the part that is not free.
- "Upload cost is the cost." For anything read more than a few times, serving dominates (Serving Files).
Operating it
- Track the size distribution of real uploads, not the expected one. The p99 is what decides whether the chosen path holds (Percentiles: Which One, and How Many Users Is That?).
- Count uploads by path (proxied vs direct) and by outcome. A direct path with a low completion rate is a client integration problem you cannot see any other way.
- Watch memory and in-flight request counts on the API path specifically. That is where the proxied path fails (The Metrics a Backend Must Emit).
- Track time from upload to
ready. It is the number users experience, and it lives entirely in the processing pipeline (Queue Backlog). - Attribute storage and egress cost per feature. Avatars and raw video have nothing in common on a bill (Cost Engineering).
- At 10x, the proxied path fails first for large files and is fine for small ones. The threshold you set is what determines whether you notice.
- At 100x, the interesting cost is serving, not uploading. Egress and CDN hit rate dominate; the upload mechanism is a rounding error (CDN Architecture).
- For the avatar case, nothing changes at any scale, and that is the point: the simple path is correct forever for small, bounded files (Choosing an Upload Path exists to let you stop worrying about them).
- Two upload paths mean two sets of failure modes, two sets of tests and two places validation can drift. One path means one of your features is on the wrong architecture.
- A size threshold is a sharp edge in a smooth distribution — a file just over the line behaves completely differently, and users notice.
- Choosing direct upload buys scale and costs synchronous validation, a completion state machine and reconciliation.
- A specialized media service removes most of this work and adds a vendor in the hot path of a core product feature.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe five inputs — size, privacy, inspection, processing, serving — apply regardless of stack or provider.
- CLOUD-SPECIFICWhat the direct path can enforce differs by provider: some signing schemes can bind a maximum content length, others cannot, so "the size limit is enforced by the signature" is true on one path and false on another. Managed scanning and media-transformation services also differ in whether they exist, whether they are event-driven, and whether they can block access until a scan completes.
- SCALE-SPECIFICBelow a few uploads per minute of files under a few megabytes, the proxied path is correct and the direct path is overhead. The threshold is not a number anyone can give you universally — it depends on your instance memory, your runtime's concurrency model and your upload concurrency.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.